The Future of Software Testing: Machine Learning to the Rescue
The past decade has seen a relentless push to deliver software faster. Automated tests has established itself as one of the most important technologies to scale DevOps, companies are investing a tremendous amount of time and effort in building end-to-end software delivery pipelines, and containers and their ecosystem are delivering on their early promise.
The combination of delivery pipelines and containers has helped top performers deliver software faster than ever. That said, many organizations still struggle to find a balance between speed and quality. Many are stuck trying to make progress with legacy software, large test suites, and fragile pipelines. So where do you go from here?
In an effort to release quickly, end users have become software testers. But they don’t want to be your testers anymore, and companies are taking notice. Companies now want to ensure that quality is not compromised in the pursuit of speed.
Testing is one of the key DevOps controls that organizations can leverage to ensure their customers are engaging with an enjoyable brand experience. Others include access control, activity logging, traceability, and disaster recovery. Our company’s research over the past year indicates that slow feedback cycles, slow development loops, and developer productivity will remain top priorities for years to come.
Quality and access control are preventive controls, while others are reactive. There will be a increasing attention to quality in the future, as it prevents customers from having a bad experience. So delivering value quickly – or better yet, delivering the right value at the right level of quality quickly – is the key trend we’ll see this year and beyond.
Here are five key trends to watch.
1. Test Automation
Test automation efforts will continue to accelerate. A surprising number of companies still have manual testing in their delivery pipeline, but you can’t deliver quickly if you have humans on the critical path of the value chain, which slows things down. (The exception is exploratory testing, where humans are a must.)
Automating manual tests is a time-consuming process that requires dedicated engineering time. While many organizations have at least some test automation, more needs to be done. This is why automated testing will remain one of the main trends in the future.
2. Establish a culture of continuous quality
As teams automate testing and embrace DevOps, quality must be part of the DevOps mindset. This means that quality will become a shared responsibility of all members of the organization.
Figure 2. High performers move testing to create new workflows. They move left for earlier validation and right for faster delivery. Source: launchable
Teams will need to become more intentional about where the tests will land. Should they left shift tests to catch problems much earlier, or should they add more quality checks on the right? On the “shift-right” side of the house, practices such as chaos engineering and canary deployments become essential.
Moving large test suites to the left is difficult because you don’t want to introduce long delays when running tests in an earlier part of your workflow. Many companies mark certain tests of a large suite to run pre-merge, but the downside is that these tests may or may not be relevant to a specific change set. Predictive test selection (see Trend 5 below) provides a compelling solution to run only relevant tests.
3. Data-Driven DevOps
Over the past six to eight years, the industry has focused on connecting various tools by building robust delivery pipelines. Each of these tools generates a large amount of data, but this data is used minimally, if at all. We’ve moved from ‘home-made’ or ‘home-made’ solutions to the ‘scale’ stage in the evolution of tools in delivery pipelines.
The next phase is to bring intelligence to the tooling. Expect to see an increased focus from practitioners on data-driven decision-making.
4. The Rise of AI: Test Generation Tools
There are two key problems in testing: not enough testing and too much testing. Test generation tools try to solve the first problem.
To create a UI test today, you either have to write a lot of code or have a tester manually click through the UI, which is an incredibly painful and slow process. To ease this pain, test generation tools use AI to create and run UI tests on various platforms.
For example, one tool my team explored uses a “trainer” that lets you record actions on a web application to create scriptless tests. While scriptless testing isn’t a new idea, what’s new is that this tool “automatically fixes” tests along with changes to your UI.
Another tool we explored is AI robots that act like humans. They press buttons, scan images, enter text and navigate screens to detect problems. Once they find an issue, they create a ticket in Jira for the developers to take action.
Other testing tools using AI will gain popularity in 2021.
5. Machine learning and selection of predictive tests
AI has other uses for testing aside from test generation. For organizations struggling with runtimes of large test suites, an emerging technology called predictive test selection is gaining traction.
Many companies have thousands of tests running constantly. Testing a small change can take hours or even days to get feedback. While more testing is generally good for quality, it also means feedback comes in slower.
To date, companies such as Google and Facebook have developed machine learning algorithms which process incoming changes and run only the tests most likely to fail. This is predictive test selection.
The amazing thing about this technology is that you can run between 10% and 20% of your tests to achieve 90% confidence that a full scan won’t fail. This allows you to reduce a five-hour test suite that normally runs after the merge to 30 minutes before the merge, running only the tests most relevant to source changes. Another scenario would be to reduce a race from one hour to six minutes.
Wait selection of predictive tests become more common in 2021.
The real problem: test execution time
Automated testing is taking over the world. Even so, many teams are struggling to make the transition. The culture of continuous quality will be part of the DevOps mindset. Tools will continue to get smarter. Test generation tools will help bridge the gap between manual and automated testing.
But as teams add more tests, they face real test execution time issues. Although more testing helps improve quality, it often becomes a hindrance to productivity. Machine learning will come to the rescue as 2021 approaches.
