Assessing Quality Risks and Estimating Test Effort

A typical goal of testing in all projects, agile or traditional, is to reduce the risk of product quality issues to an acceptable level before release. Testers in agile projects can use the same types of techniques used in traditional projects to identify quality (or product) risks, assess the associated risk level, estimate the effort required to sufficiently reduce those risks, and then mitigate those risks through test design, implementation, and execution. However, given the short iterations and rate of change in agile projects, some adaptation of these techniques is required.

Assessing Quality Risks in Agile Projects

One of the many challenges in testing is the proper selection, assignment, and prioritization of test conditions. This includes determining the appropriate effort to cover each condition with tests and arranging the resulting tests in a way that optimizes the effectiveness and efficiency of the test work to be performed. Risk identification, analysis, and mitigation strategies can be used by testers in agile teams to determine an acceptable number of test cases to execute, although many interacting constraints and variables may require tradeoffs.

Risk is the possibility of a negative or undesirable outcome or event. The level of risk is determined by evaluating the probability of the risk occurring and the impact of the risk. If the potential problem primarily affects product quality, potential problems are referred to as quality risks or product risks. If the potential problem primarily impacts project success, potential problems are referred to as project risks or planning risks.

In agile projects, quality risk analysis occurs at two points.

  • Release planning: representatives of the business who understand the release functions provide a high-level overview of the risks, and the entire team, including testers, can help identify and assess the risks.
  • Iteration planning: the entire team identifies and assesses the quality risks.

Examples of quality risks to a system include:

  • Incorrect calculations in reports (a functional risk related to accuracy).
  • Slow response to user input (a non-functional risk related to efficiency and response time)
  • Difficulty understanding screens and fields (a non-functional risk related to usability and understandability)

As mentioned earlier, an iteration begins with iteration planning, which culminates in estimated tasks on a task board. These tasks can be prioritized in part by the degree of quality risk associated with them. Tasks associated with higher risks should start earlier and require more testing effort. Tasks associated with lower risks should start later and require less testing effort.

An example of how to conduct the quality risk analysis process in an agile project during iteration planning is outlined in the following steps:

  1. Gather the agile team members, including the tester(s).
  2. List all backlog items for the current iteration (e.g., on a task board)
  3. Identify the quality risks associated with each element, considering all relevant quality attributes
  4. Assess each identified risk, which involves two activities: Categorizing the risk and determining its risk level based on impact and likelihood of failure
  5. Determining the scope of testing in relation to the risk level.
  6. Selecting the appropriate test technique(s) to mitigate each risk based on the risk, risk level, and relevant quality attribute.

The tester designs, implements, and executes tests to mitigate risks. This includes the set of features, behaviors, quality characteristics, and attributes that impact customer, user, and stakeholder satisfaction.

Throughout the project, the team should be alert to additional information that may change the set of risks and/or the level of risk associated with known quality risks. The quality risk analysis should be adjusted periodically, resulting in adjustments to testing. Adjustments include identifying new risks, reassessing the level of existing risks, and evaluating the effectiveness of risk mitigation measures.

Quality risks can also be mitigated before test execution begins. For example, if issues with user stories are identified during risk identification, the project team can thoroughly review the user stories to mitigate the risk.

EstimatingTesting Effort Based on Content and Risk

During release planning, the agile team estimates the effort required to complete the release. The estimate also refers to the testing effort. A commonly used estimation technique in agile projects is planning poker, a consensus-based technique. The product owner or customer reads a user story to the estimators. Each estimator has a deck of cards with values similar to the Fibonacci sequence (i.e., 0, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, …) or any other progression (e.g., shirt sizes from extra-small to extra-extra-large). The values represent the number of story points, effort days, or other units the team estimates. The Fibonacci sequence is recommended because the numbers in the sequence reflect that uncertainty increases proportionally with story size. A high estimate usually means that the story is not well understood or should be broken into several smaller stories.

Estimators discuss the feature and ask the product owner questions as needed. Aspects such as the development and testing effort, the complexity of the story, and the amount of testing play a role in the estimate. Therefore, it is advisable to consider the risk level of a backlog item in addition to the priority set by the product owner before the planning poker. When the feature has been fully discussed, each estimator privately selects a card to represent their estimate. All cards are then revealed simultaneously. If all estimators have chosen the same value, it is considered the estimate. If not, the estimators discuss the differences in the estimates, whereupon the poker round is repeated until an agreement is reached, either by consensus or by applying rules (e.g., using the median, using the highest score) to limit the number of poker rounds. These discussions ensure a reliable estimate of the effort required to complete the Product Backlog Items required by the Product Owner and help improve the collective knowledge of what needs to be done.

Was this article helpful?

Related Articles