In a world where the rapid delivery of new or modified software is becoming increasingly important, where nearly all of us have transitioned to Agile ways of working, where more and more teams are working with DevOps, and where test automation is a necessity to guarantee continuous quality.
In that world, we sometimes seem to put too much trust in test automation. We tend to think that by automating as many tests as possible, we gain more insight and control over product quality. That we get the right results faster and thus work more efficiently. But is that truly the case? Can we rely almost blindly on these automated tests? Should we even want to automate everything? Just because something can be done, does not mean it should be done, or that the final product will be better because of it.
Perhaps this is a strange position for someone with a strong preference for test automation (me). I would like to elaborate on this in this blog post, where I will discuss the prerequisites for test automation. Prerequisites from the business perspective (Ready). From the teams’ perspective (Set). And subsequently, from the testing perspective… (Automate?)
Ready
Just as a house needs a solid foundation, test automation also requires a good basis. Why is test automation being chosen? Is it to solve existing challenges within the development process? Or is it to support and strengthen the existing testing strategy? There are various reasons to apply test automation within an organization. And these reasons partly determine the best way to implement test automation. A few of these reasons are:
- Fast feedback is one of the most common reasons to automate. By running a specific test set after every build or merge to ‘develop,’ you immediately gain insight into potential findings, and these can be addressed right away. This makes the development process more efficient and enables the step towards Continuous Integration/Continuous Delivery (CI/CD).
- Long-term cost savings is also a possible reason. After all, automated tests only need to be set up once and can then be repeated. Naturally, setting up and maintaining those scripts takes time. But in the long run, it prevents repetitive manual testing that consumes significant time. Furthermore, it also frees up capacity to focus on more complex tests or exploratory testing.
- Manual testing not only costs a lot of time but is also prone to errors. A test step might be skipped. Or the test data is not fully or correctly prepared. Accuracy and increasing test coverage is therefore a frequent reason for automating certain tests.
And there are more reasons to adopt test automation: CI/CD, Performance, scalability, reusability.
It is important for an organization to carefully consider this. Good test automation is a significant investment. Not just in money, but also in time, knowledge, and experience. And if it is not implemented correctly, it can ultimately cost quality, time, and money, instead of yielding benefits. In short, there must be a valid business case for introducing test automation.
Once the business case is clear, the next step is to assess whether the organization is technically ready for it. Is the infrastructure suitable for automating tests? Are there sufficient resources to be able to automate tests? Should multiple test environments and databases be available, or should container hosting be used instead? Here too, careful consideration is crucial. What possibilities does the current infrastructure offer, and can we build upon it? Or is a new infrastructure necessary?
Good test automation is a major investment. Not just in money, but also in time, knowledge, and experience. There must therefore be a valid business case for introducing test automation.
Set
The organization is ready. The reasoning has been considered, and the infrastructure is also in place. Next, it is crucial that the teams are also ready. And that starts with the team members. Is there sufficient knowledge and experience for test automation? If not, can this be achieved by following training courses or hiring/contracting technical testers?
The right tooling is also important. Various tools are available, some paid, but many are open-source as well. Which tool is best to use depends on several factors, including the knowledge and experience within the team, but also the application to be tested. Is it a web application with a focus on the GUI, or do you primarily want to test message traffic between applications?
Many tools are also based on Behaviour Driven Development (BDD), which allows test cases to be written in plain, understandable language, for instance, by including specific test steps in a keyword or by using Gherkin (given… when… and… then…). If the tests need to be built or maintained by non-technical testers, or if readability is important, these types of tools are a good option.
A testing tool that programs in the same language as the application is also recommended. This enables developers to support the testers, and it makes it easier for them to build and maintain tests.
To write good automated tests, good functional documentation is necessary. Documentation with clear acceptance criteria, where test cases can be traced back to the functional need. This makes it easier to maintain the tests when functional changes occur.
Automate(?)
heard it quite recently on an assignment. A colleague suggested that all tests that can be automated, should be automated. And I couldn’t disagree more with that notion.
In that particular team, the years-long tendency had been to automate everything. This resulted in a test set of hundreds of tests that took a few hours to run. Not very efficient, and certainly not providing fast feedback. The tests overlapped significantly because the same steps were performed each time, with the only difference being the final test step. Maintenance was a disaster, as one change in functionality meant modifying dozens of tests in the test set.
It also created a false sense of quality. Automated tests actually only check what you already know: what is stated in the functional documentation, or what you can deduce through experience and knowledge. But what you don’t know will never be found. This can only be achieved by performing exploratory testing. Furthermore, the number of tests says nothing about the quality. Poor tests add no value whatsoever. It’s not about the quantity, but about a well-considered test coverage.
To prevent your team from ending up with an automated test set that is simply too large to be efficient, which is barely or not at all maintainable—leaving little time for exploratory testing—it is essential to weigh whether each test is of value for the automated set.
This can be done, for example, by performing a product risk analysis, where you decide to automate only the test scenarios with a medium or high risk. Alternatively, you can use test automation as a way to quickly prepare test data or a specific pre-condition.
It is sometimes forgotten how important it is to perform exploratory testing alongside automated testing—where we only validate what we know—by focusing specifically on the unknown. Automating tests is necessary in the current market. But the extent to which you do it depends on a great many factors.
Do you want advice on the possibilities that test automation can offer your organization, how it can become a part of your existing test strategy, or which tooling to use? Then do not hesitate to contact us.
