Although test report documents are traditionally associated with Waterfall, they can also contribute to the Agile development process.
A test report summary contains full details of the environments where the code was tested, who tested it, when it was tested, and how it was tested. The document serves as a physical log that records exactly what code was tested, what system configuration the code was tested in, and any bugs that appeared during testing.
While some IT organizations create test reports longer than 20 pages, document length is subjective. The overall purpose of the test report summary is to record the actions and results of a test. This allows the team to make informed decisions about procedural improvements that can be made for future testing. Often a development team may use test reports for internal audits following a customer complaint, but these reports should be written regularly during the testing process to create better software with each development cycle. .
In Agile development, the test report summary represents a record of test execution. Compared to its companion Waterfall, this version is less formal and more focused on results.
Let’s look at the basic components of writing a test report and why such a summary can be useful in Agile software development.
What’s in a test report?
Before creating a test report, the author should determine the target audience and why the report is needed. For example, if an application requires an audit trail for regulatory reasons, the test report writer will likely need to include more specific data. If the target audience is senior management and they want to understand what the team tested for each release, the writer can include an executive summary outlining the major features tested. Or, if the report is more of an audit that will not be read by anyone unless a critical bug is found, the writer can structure the report to include only technical information. Download a test summary report template here.
While these summaries should all include the same basic information necessary for the target audience, there is no formula for test reports. Testers can add or subtract data as needed to achieve the purpose of the test report. Here are four main things every tester should include when writing a test report.
Test objective. The objective of the test is what type of test the team and its testers ran, and why. For example, if the test report covers functional testing, regression testing, and performance testing, the test report writer should describe the purpose for each type of testing.
In most cases, regression testing is the main purpose of test execution. The purpose of regression testing can vary, but a team typically performs defect finding after developers have added new feature code to an existing codebase. Regression testing is performed before any new release and varies in duration and depth of testing. If a team’s regression testing includes integration, performance, or other types of testing, the writer should state the purpose of each test specifically in the purpose section of the report.
Test cases, test coverage and execution details. The next thing about writing a test report is to explain the test suite. Specifically, indicate what type of test was run, where it is stored, and when it was run. The test report writer may also add the name of the QA professional who performed the test, but this is not a specific requirement. Instead, it’s more important to include the how/what/why and actual test results.
The writer should indicate how many tests were run, passed, failed, or skipped in this section. Skipped tests represent tests that a team scheduled but missed either due to time constraints or because the tests were blocked by reporting defects. In such cases, teams should also include how much code they have tested. A team can use test management applications and tools to specify the amount of code tested by its QA professionals. If such tools are not available, contact the development team for an estimate of test coverage.
Execution details, which a tester manually records or a test management program tracks, include who tested the code and when and where it was tested. A test report writer has some flexibility in how to display data and details; this often depends on the number of tests performed by a team. The test report can include a general data grid to display information or another data report. Again, there is no set rule on how this information is included in a test report. The authors use their own judgement.
Default matters. Another important aspect is to document the defects found by the testers. This section is vital for post-test analysis, which means that test report writers should not just list bug ID numbers. They should include a brief description of each bug to save you time later. However, the author will not necessarily need to list all the bugs found during testing. The report writer may consider waiting for the product management team to confirm that a bug or defect exists before including it in the test report.
Testers will need to carefully review the list of defects to verify that they are not re-reporting known bugs or bugs already in the repair log. The data in this section should focus on the defects found in the indicated version, organized by their priority. Defects and bugs are always found, but their priority and severity determines whether the release is sent to customers or held for repair.
Platform and test environment setup details. The setup and test environment section is tricky. Details are important, but test report writers should also consider security and compliance when sharing information about an application’s code. The writer should assume that the target audience generally understands the test system; this is not the place to reveal details about an application’s servers and code storage.
Once the author shares the test report, there is no guarantee that it will be stored safely. Include the server name and dates, but keep the basic data.
If the configuration was non-standard, the writer should also include this information. The standard configuration should be documented within the test team for reference.
For example, if during testing the configuration changes and causes defects, a writer should include this information in the test report. Someone can analyze the effect this has on the scope or depth of defects and overall test coverage.
Understand who a test report is intended for
The importance of the test report really depends on the needs of a particular business. It’s a handy document for tracking test results by release so members of an IT organization know what was tested and when. Whether it’s a formal document or a simple summary of what has been done, a test report contributes to better software development.