In software testing, open-source tools have existed for quite a while and they will keep existing in the future. New testing frameworks and tools appear every single day, so how do you know what works best for you? Are commercial tools better than open-source alternatives or the other way around? There is no clear answer and “it depends” highly on your needs.
- What will you use the tool for?
- Will it solve problems you really have?
- Can it optimize your work?
- Does it match your team’s skill set?
- Do you need SLAs?
Teams are unique and should use whatever tools they want in order to be more efficient, productive, and happy. The secret is to take advantage of what both commercial and open-source tools have to offer. Then, take the best of each approach, leveraging what makes each tool unique and powerful.
In this article, I’ll examine the pros and cons of commercial vs. open-source tools and how to choose a tool for testing and test automation.
What is open-source software?
Oversimplifying it, open-source allows you access to the source-code and gives you the right, under certain conditions, to modify it. There are many open-source licenses in the wild.
Open-source and free software are not the same thing; sometimes people confuse these two concepts. Freedom goes way beyond open-source and it’s not just about the acquisition cost of the software. There can be some open-source licenses that don’t qualify as “free software.”
Private and profit oriented companies may actively support open-source projects (e.g. RedHat, IBM, Microsoft, etc) though, some of them either sell services around that or make “enterprise” variants of the products, with additional features, access to paid software, etc.
What is commercial software?
Proprietary software means that the owner of the software belongs to an entity or person and that you may require the right to use it under certain conditions defined in the EULA. You can also be granted access to the code of proprietary software or even change it, if you comply with the agreement.
However, most of the time, proprietary software is not free and you also don’t have access to its source-code.
The pros and cons of open-source vs. commercial tools
I’ve worked with proprietary and open-source for many years and I think these are not far from reality. Personally, I’ve used unstable commercial and unstable open-source products. I’ve also found emergent open-source tools to be more unstable than “normal,” but sometimes those open-source products didn’t have direct commercial alternatives/competitors.
I’ve provided commercial support myself and also dealt with commercial support from external suppliers. Being involved in open-source tools and projects as well, I’ve seen projects whose support is weak or takes forever and provides unclear answers. I’ve also seen that kind of support in commercial tools, however I’d say that it’s more frequent in open-source projects.
How do you consider the cost?
The real cost of using a certain tool (that fits your needs) does not depend only on the acquisition cost (i.e. CAPEX or capital expenditure) but also on the “ongoing cost” (i.e. OPEX), which includes maintenance, support, etc.
Hence, open-source is not actually free since you have to consider all costs including training and time to learn to use it. However, in some cases these tools are so simple that you won’t need training at all. On the other hand, some commercial tools are so complex that you have to take training or digest long documents in order to use them thoroughly.
How do you choose a tool for automation?
There are many open-source tools that are used to facilitate testing, test automation and automation in general. We all know Selenium WebDriver, Jenkins, Maven and Docker for sure, among frameworks such as JUnit, RSpec, and Robot Framework. New languages and new frameworks appear every day. However, many die along the way, while others, like Cypress, receive great traction.
From my experience, proprietary commercial tools are often focused on different areas such as test management, as opposed to open-source ones. Though there may be some overlapping, the same initial needs will be addressed differently.
With the rise of ML (Machine Learning) and AI (Artificial Intelligence), we’ll see some related features only being available in commercial products, at least for a while. Why? Because of the expertise and resources that are needed to encompass such topics.
Open-source tools will always be present and complement what proprietary tools don’t offer. In the industry, we should not see them as competitors but as complementary. I’m a great defender of open-source tools; however, they do not provide everything we need and sometimes the support offered is not great.
We see open-source tools filling the space of automation libraries, automation/testing frameworks and many tools related with automated testing in general, including containers handling, programming languages, etc. In my opinion, they will keep living together side by side.
Coexistence is the future of testing tools
We all know test management or ALM solutions that try to sell you a bunch of tools that integrate with one another. These tools work “great” as long as you restrict your tool portfolio to them. The problem arises whenever you want to use open-source tools together with them and let’s be honest: some simply don’t like open-source that much.
The team behind Xray, uses a mix of open-source and proprietary tools for supporting all the development and testing activities. We also use commercial software to support project management/issue tracking and customer care/support.
Xray is a commercial tool yet, it integrates with open-source tools/frameworks including, and not only restricted to Jenkins, JUnit, xUnit, TestNG, Robot Framework, Cucumber, SpecFlow and more. It also integrates with commercial tools such as Bamboo, Team City and the old HP QC/ALM.
Don’t let a tool be chosen and implemented for you based on what others think is best. Together, work as a team to identify improvement opportunities and agree on the action points that can push you forward, whether they involve using a commercial tool or open-source.
Hello,
First of all, thank you for this article.
But when I read Open-Source Software CONS, I think you’re confusing Open-Source and Free Software… Funny after you saying before “Open-source and free software are not the same thing”… Open-Source (OS) is not always Communitary Edition (CE), but yours cons are…
In more, some projects are abandoned, it’s PRO not CONS ! Some proprietary projects are abandoned too and you are totaly locked in this case…
For training and documentation… IMHO, it’s wrong again. See IBM docs on proprietary soft and Google doc on Open-Source soft… There is no match…
Open-Source and commercial licence are not incompatible… I think you can replace OS by CE…
BR,
Guillaume.
Hi Guillaume, thanks for your insights.
Well, first of all this is my perspective. I’m an open-source user and contributor by the way.
Concerning abandoned projects, I came across many that where interesting and useful; however, some time afterwards they became stale or abandoned. Of course this happens with commercial tools but it didn’t happened to me as much.
Regarding training and documentation, there are many courses and resources out there; some payed some free. The biggest problem is that when we start using open-source tools you suddenly use a lot; consider for a moment all those OS dependencies and application-layer dependencies (e.g. all NPM packages). From all of them, we will be using just a subset but even so it’s hard to get trainings for these, from my experience.
With commercial tools we can also have big problems; I’ve seen that: vendor lock-in. This may also happen with open-source tools but I don’t think it happens that much if adopted tools have high-tractio.
Regards,
Sergio