Why fixing software bugs should be the CEO’s problem
CEOs and other leaders can and should be intimately involved with software quality, just as they are with sales and finance divisions. This involves understanding how technical teams work and setting up a quality management system. You can do this by asking two simple questions about a recently shipped product. First, ask what criteria were used to determine when the product was ready to ship. There should be a clear discussion of the duration of system testing, the type of testing performed, and the specific criteria used in the decision to ship the product. Second, ask what the current state of defects is after the first six months. It’s normal to see an increase in defects after initial delivery, due to increased use in real-world environments, but leaders need to probe how engineering teams prioritize bugs and categorize the most serious defects. With this information, executives can drill down into the key metric of open days for a defect, to ensure that the most serious defects are fixed quickly.
Software platforms permeate the fabric of our lives, but only 27% of CEOs in the Fortune 100 have engineering and science degrees. Join a quarterly earnings call and you’ll hear plenty of talk about revenue, expenses, and geographic trends, but little (if anything) about the quality of the company’s software. The results are obvious: for almost all major disasters caused by software faults, the autopsy usually determines that the fault had existed for some time.
The problem isn’t that business leaders need to have an engineering background and they don’t, but that few outside engineering silos know how to discuss critical software systems. As a result, software bugs usually stay under the CEO’s radar unless a cataclysmic event occurs.
Toyota’s sudden acceleration problem was a textbook case of what can happen without a proactive internal quality system. Between 2004 and 2010, several investigations were opened to determine if Toyota vehicles accelerated properly; various theories have blamed electronics, floor mat entrapment, and “sticky” pedals. Toyota maintains that software bugs were not responsible for the errors. When a lawsuit against Toyota was settled three years later, two expert witnesses retained by the plaintiffs reported that the “system was faulty and dangerous, riddled with bugs and gaps in its safeties that led to the root cause of the failure. ‘accident”. A Toyota programmer described the engine control application as “like spaghetti.” The drawn-out nature and lack of resolution of this case are clear signs that there was an inadequate quality review system in place. With a good system, any bugs would have been visible to leaders month after month, early and often, prompting effective and vital action. Since the complaints, Toyota has become much more proactive in responding to concerns. Unrelated to acceleration issues, in February 2010 Toyota proactively recalled 437,000 cars to fix brake software. Another positive outcome was that in a 2014 DOJ settlement regarding floor mats and sticky pedals, Toyota announced that it had made fundamental changes to its corporate structure and internal safety controls, including a team rapid response to investigate potential defects.
Business leaders can and should be intimately involved in software quality, just as they are in sales and finance departments. This involves understanding how technical teams work and setting up a quality management system.
What does leading the way look like
I worked in IBM’s mainframe division at a time when IBM and Microsoft had quality issues that caused significant headwinds and customer satisfaction issues. By the early 1990s, IBM was experiencing serious quality issues in the field in addition to difficulty meeting deadlines for new products. CEO Louis Gerstner Jr. was frustrated that on the due date of some products he was told they would miss the launch date by a year. In a legendary memo, Gerstner proposed an amnesty program – 30 days to reset dates, and after that, missing a date would be grounds for dismissal.
The task of creating new schedules went to Nicholas Donofrio, the SVP of Technology and Manufacturing. Donofrio’s motto was “Be frank [about your schedule and quality issues] and i will be available [about getting you the needed resources].” He didn’t have a direct reporting line for the systems business (hardware and software), but because he was the former head of the business unit, he had deep knowledge and deep personal relationships. Nearly 80% of product dates have been reset and each product organization has end-to-end quality management systems in place. In 1999, IBM was the world leader in servers, with 23% market share.
Similarly, in the 1990s, Microsoft’s Windows operating systems had a series of bugs that caused computers to freeze frequently. The public largely got used to these failures – the “blue screens of death” – but Microsoft was hit by other bugs later: when Windows systems connected to the Internet, they suffered numerous hacks, embarrassing viruses and security issues. Bill Gates responded to the crisis with a memo calling for arms, which was sent to all 50,000 employees and simultaneously published in Wired. In it, he defines “trusted computing”, a wide range of initiatives to improve security and product design, and outlines the prerequisites for a comprehensive quality management system (QMS), including changes in software design and development processes, new error reporting capabilities, and new update features. In 2000, Microsoft’s revenue topped $20 billion, and the company reported that customers cited the new Windows 2000 server as the most reliable version to date.
These two examples show how leaders can turn around deteriorating engineering divisions by asking simple questions, setting standards, and seeing themselves as part of the process.
When software is critical, so are bugs
Creating an effective quality management system that includes both engineers and top-level leaders should be approached as an evolutionary exercise, starting simply with a focus on high impact from the start. CEOs should ask two simple questions about a recently shipped product:
1. “What criteria were used to determine when the product was ready to ship?” There should be a clear discussion of the duration of system testing, the type of testing performed, and the specific criteria used in the decision to ship the product.
2. “What is the current status of defects after the first six months?” It’s normal to see an increase in defects after initial delivery, due to increased use in real-world environments, but leaders need to probe how engineering teams prioritize bugs and categorize the most serious defects. With this information, executives can drill down into the key metric of open days for a defect, to ensure that the most serious defects are fixed quickly.
The 2×2 grid below shows how to score a quality management system based on the answers to these two questions. Along the y-axis, you measure the degree of compartmentalization of information, based on how quickly questions are answered. Along the x-axis, you measure the organization’s response to your query, based on how proactive the team is in resolving bugs. A “mature” quality management system can be identified by the team sharing defect information promptly and seriously with key leaders; in a “troubled” system, answers come slowly and with great antagonism.
If you find that your business is in the Troubled quadrant, your first step should be to immediately create a proactive focus on defects in order to move into the Learning quadrant. This is what Gerstner and Gates did. You should focus on creating a culture where honesty is rewarded and where employees feel safe discussing methods and techniques for resolving software defects. In this environment, information must flow easily between the required silos, moving your system from the Learning quadrant to the Mature quadrant.
Next level quality management
If you are creating a quality management system from scratch, your first step is to decide how the organization will categorize and prioritize bugs. This should be done by customer-facing teams in collaboration with customers. Typically, teams will prioritize two types of bugs: those that cause a system crash and loss of service (these have been rated the highest severity at IBM) and those that are less serious but could be ubiquitous.
Then, as an organization, decide on your target response time for each severity level. If the quality management system is new, the initial goal should be to fix the most serious bugs within hours or days. As you use your system, you can collect data on two key metrics, inbound bug rates and bugfixer productivity, and adjust your targets as needed. Finally, you need to create an appraisal system that involves yourself and other senior leaders. Reviews of open defects and the time taken to resolve a defect should be done with varying degrees of detail at all levels of the organization.
Once the QMS is established, the CEO is unlikely to see many old bugs just because no one wants to give the same excuse two months in a row. The CEO could also review all high-severity bugs and those that are pervasive. The CEO might ask the simplest question: “How was the software released with this type of bug?” The product development team might respond with software engineering jargon about “breakouts,” and the CEO might then ask an almost rhetorical question: “Do we test those conditions next time before launching the product? ?”
This type of QMS will lead to a better customer experience. To understand how important this is, consider a software flaw I regularly encounter in my bank’s online verification system: an expense code function fails frequently, requiring me to restart my computer. I’ve complained countless times over the last year, and all the bank says is, “We’ve told IT, but we don’t know if they’ll fix it. .” This is a sign that the bank lacks a rigorous quality management system, which leaves its financial advisors helpless and annoys its customers, possibly forcing them to switch banks. I wish I could tell them that they can turn around, like IBM and Microsoft did.
Editor’s Note: We have further clarified the paragraph regarding investigations of Toyota vehicle acceleration issues. We have also clarified that the expert witnesses cited in this paragraph were retained by the plaintiffs.
