Ease of use: where software testing tools are insufficient

The tools to support and automate parts of software testing activities are important, even essential in some areas, and yet they can be problematic as well. I’ve seen teams struggle with today’s testing tools and wondered why: is this an issue with the skills of the testers or with the usability of the tools themselves?

I have worked in the software industry for decades, and as part of a research project as a PhD student at the University of Malta, I interviewed testers about their experiences with testing tools and on the concerns and challenges they face in their implementation and use. I have collected data through surveys, workshops and interviews with over 100 testers around the world, amassing a huge number of stories about their experiences, good and bad, with testing tools and automation.

As it turns out, testers claim that usability issues affect the success of testing tools. Here’s what else I learned: a preview before my in-depth presentation at the next EuroSTAR Conference.

Ease of implementation, use are key issues

As a software tester, the information you provide to decision makers and your colleagues must be useful, timely and accurate. Delivering software on time and then failing is not a good result, but neither is delivering software late. Using tools to support testing makes sense, but while testing support tools have been around for decades, so does their acquisition and use.

In the 1980s, I worked on a project where we were using tools to run tests. The tools allowed us to deliver on time and with increased certainty about the quality of the software. The software, the tests and the tools have grown together. But over the decades, I have noticed that the successful use of tools is unusual. While more tools are available today, testers complain that their tools have not been easy to implement and use over time.

As the computer industry has grown and software has become ubiquitous via the Internet and mobile technologies, the work of testers has become increasingly in demand. We do more, with a requirement for faster delivery. The tools to support all aspects of testing have become even more important, both in terms of the scope and speed of testing. However, setting up tools and using them successfully remains problematic.

Problems include lack of management support, misconceptions and over-optimism about what the tools can achieve, and technical issues with the tools. These problems are well documented, but still pervasive. Additionally, there is a push to change testers’ skills to include coding, to support scripting in testing tools.

Most users have issues with their testing tools

The first result is rather depressing. The whole time our industry has been trying to embrace and improve tool support, for all of the presentations, tutorials, webinars, books, and discussions we’ve shared, we’re still making the same old mistakes.

Some 82 of the 111 participants surveyed raised at least one problem with their tools. The table below shows the percentage of comments shared on three main topic groups. Management decision making continues to be important, with challenges, blockers and catalysts identified in hiring errors, lack of training support, unreasonable expectations, and ill-defined goals.

Technical blockers abound, including installation issues, issues with tool testing and maintenance, IT security preventing progress, and tools that are incompatible with each other, blocking a flow of testing activity.

Usability is indeed an issue: Testers said they wanted more usability built into their tool design, and 37% of all issues raised were usability related.

From tool frustration to anger: testers expressed a range of emotions

A second result was that the testers had very emotional responses to many of the survey questions, and this was directly related to their experiences with the tools and automation. Testers said they felt ‘blocked’ because ‘the tools are presented as’ magic solutions’ and they also feel’ scared ‘and’ frustrated ‘but also’ quietly proud ‘of what they are doing. they do.

Two participants indicated that they had to use a particular test tool, but due to organizational security policies and the technical configuration of the test environment, they were unable to access the tool. which resulted in unnecessary frustration, anger and demotivation. Two people used the word magical to describe management’s expectations for implementing the tools (team size, reduced costs and time), while other testers spoke of fears and feelings of isolation and worthlessness.

Usability is a priority for designers and tool vendors, but even so, testers reported issues. Usability issues are not a matter of total neglect, but in some cases the result of a superficial approach to usability. Here are five key issues:

1. Salespeople focus on the interface to emphasize attractiveness rather than utility. This resulted in an interface that “looks cool” but does not support the workflow of testers. True usability focuses on achieving goals and supporting a seamless workflow experience.
2. Some testers reported that their tools were too difficult to use. Others reported insufficient flexibility. Usability should take into account the range of tool users and their specific goals, needs and abilities.
3. Others said their tools did not meet their changing needs. Usability should provide task flow for novice and expert users, supporting movement between these levels.
4. The testers, the software under test, the test environment, and the tests themselves all change over time.. Since usability is necessary but not sufficient, maintainability and portability of tests and tools are also important.
5. Other attributes such as security, performance, and reliability must also be supported.

All in all, this means that when you buy tools, you can fall into an “user-friendliness illusion”. This is where you think a tool will be useful and easy to use and solve your testing issues, only to be disappointed when it won’t help you complete your job due to one or more of these five issues. .

Get more of the tools you have

User-friendliness is an essential aspect of the success of a test tool, but it is not trivial to achieve when designing the tool. Testers come from a wide variety of backgrounds and backgrounds, so it’s no surprise that the tools aren’t for everyone.

But how can this be resolved? To help your own tool implementation be more successful, use previously published materials such as the Test Automation Patterns website to help you overcome managerial and technical barriers to success.

Want to know more? Obtain a copy of D. Graham and M. Fewster “Experiences of Test Automation: Case Studies of Software Test Automation” or K. Wiklund ‘, “Impediments for automatic software test execution: A systematic review of the literature”.

And don’t miss my EuroSTAR conference session, “Who are we? What do we do ? How do we do ? where I will talk more about my results and research, what this means for testers, and how you can contact me to participate in the next phase on what can be done to improve the design of the usability of the tools. The conference runs from September 28 to September 30, 2021. TechBeacon readers can save 10% on individual tickets using the code “TechBeacon10” through September 6.

Keep learning