There are certain testing practices that can be followed in every development project (agile or not) to produce quality products. These include writing tests in advance to express proper behavior, focusing on early defect prevention, detection, and removal, and ensuring that the right test types are run at the right time and as part of the right test level. Agile practitioners aim to introduce these practices early. Testers in Agile projects play a key role in guiding the use of these testing practices throughout the lifecycle.
Test-Driven Development, Acceptance Test-Driven Development, and Behavior-Driven Development
Test-driven development, acceptance test-driven development, and behavior-driven development are three complementary techniques in use among Agile teams to carry out testing across the various test levels. Each technique is an example of a fundamental principle of testing, the benefit of early testing and QA activities since the tests are defined before the code is written.
Test-driven development(TDD)is used to develop code guided by automated test cases. The process for test-driven development is:
- Add a test that captures the programmer’s concept of the desired functioning of a small piece of code
- Run the test, which should fail since the code doesn’t exist
- Write the code and run the test in a tight loop until the test passes
- Refactor the code after the test is passed, re-running the test to ensure it continues to pass against the refactored code
- Repeat this process for the next small piece of code, running the previous tests as well as the added tests
The tests written are primarily unit level and are code-focused, though tests may also be written at the integration or system levels. Test-driven development gained its popularity through Extreme Programming [Beck02] but is also used in other Agile methodologies and sometimes in sequential lifecycles. It helps developers focus on clearly-defined expected results. The tests are automated and are used in continuous integration.
Acceptance Test-Driven Development
Acceptance test-driven development[Adzic09]defines acceptance criteria and tests during the creation of user stories. Acceptance test-driven development is a collaborative approach that allows every stakeholder to understand how the software component has to behave and what the developers, testers, and business representatives need to ensure this behavior.
Acceptance test-driven development creates reusable tests for regression testing. Specific tools support the creation and execution of such tests, often within the continuous integration process. These tools can connect to data and service layers of the application, which allows tests to be executed at the system or acceptable level. Acceptance test-driven development allows quick resolution of defects and validation of feature behavior. It helps determine if the acceptance criteria are met for the feature.
Behavior-Driven DevelopmentBehavior-driven development [Chelimsky10]allowsadeveloperto focus on testing the code based on the expected behavior of the software. Because the tests are based on the exhibited behavior of the software, the tests are generally easier for other team members and stakeholders to understand.
Specific behavior-driven development frameworks can be used to define acceptance criteria based on the given/when/then format:
Given some initial context,
When an event occurs,
Then ensure some outcomes.
From these requirements, the behavior-driven development framework generates code that can be used by developers to create test cases. Behavior-driven development helps the developer collaborate with other stakeholders, including testers, to define accurate unit tests focused on business needs.
The Test Pyramid
A software system may be tested at different levels. Typical test levels are, from the base of the pyramid to the top, unit, integration, system, and acceptance. The test pyramid emphasizes having a large number of tests at the lower levels(bottom of the pyramid) and, as development moves to the upper levels, the number of tests decreases (top of the pyramid). Usually, unit and integration level tests are automated and are created using API-based tools. At the system and acceptance levels, automated tests are created using GUI-based tools. The test pyramid concept is based on the testing principle of early QA and testing (i.e., eliminating defects as early as possible in the lifecycle).
Testing Quadrants, Test Levels, and Testing Types
Testing quadrants, defined by Brian Marick [Crispin08], align the test levels with the appropriate test types in the Agile methodology. The testing quadrants model and its variants help to ensure that all-important test types and test levels are included in the development lifecycle. This model also provides a way to differentiate and describe the types of tests to all stakeholders, including developers, testers, and business representatives.
In the testing quadrants, tests can be business (user) or technology (developer) facing. Some tests support the work done by the Agile team and confirm software behavior. Other tests can verify the product. Tests can be fully manual, fully automated, a combination of manual and automated, or manual but supported by tools. The four quadrants are as follows:
- Quadrant Q1 is unit level, technology facing, and supports the developers. This quadrant contains unit tests. These tests should be automated and included in the continuous integration process.
- Quadrant Q2 is system-level, business-facing, and confirms product behavior. This quadrant contains functional tests, examples, story tests, user experience prototypes, and simulations. These tests check the acceptance criteria and can be manual or automated. They are often created during the user story development and thus improve the quality of the stories. They are useful when creating automated regression test suites.
- Quadrant Q3 is a system or user acceptance level, business-facing, and contains tests that critique the product, using realistic scenarios and data. This quadrant contains exploratory testing, scenarios, process flows, usability testing, user acceptance testing, alpha testing, and beta testing. These tests are often manual and are user-oriented.
- Quadrant Q4 is a system or operational acceptance level, technology facing, and contains tests that critique the product. This quadrant contains performance, load, stress, and scalability tests, security tests, maintainability, memory management, compatibility and interoperability, data migration, infrastructure, and recovery testing. These tests are often automated.
During any given iteration, tests from any or all quadrants may be required. The testing quadrants apply to dynamic testing rather than static testing.
The Role of a Tester
Throughout this syllabus, general reference has been made to Agile methods and techniques, and the role of a tester within various Agile lifecycles. This subsection looks specifically at the role of a tester in a project following a Scrum lifecycle [Aalst13].
Teamwork is a fundamental principle in Agile development. Agile emphasizes the whole-team approach consisting of developers, testers, and business representatives working together. The following are organizational and behavioral best practices in Scrum teams:
- Cross-functional: Each team member brings a different set of skills to the team. The team works together on test strategy, test planning, test specification, test execution, test evaluation, and test results reporting.
- Self-organizing: The team may consist only of developers, but, as noted in Section 2.1.5, ideally there would be one or more testers.
- Co-located: Testers sit together with the developers and the product owner.
- Collaborative: Testers collaborate with their team members, other teams, the stakeholders, the product owner, and the Scrum Master.
- Empowered: Technical decisions regarding design and testing are made by the team as a whole (developers, testers, and Scrum Master), in collaboration with the product owner and other teams if needed.
- Committed: The tester is committed to question and evaluate the product’s behavior and characteristics with respect to the expectations and needs of the customers and users.
- Transparent: Development and testing progress is visible on the Agiletask board
- Credible: The tester must ensure the credibility of the strategy for testing, its implementation, and execution, otherwise the stakeholders will not trust the test results. This is often done by providing information to the stakeholders about the testing process.
- Open to feedback: Feedback is an important aspect of being successful in any project, especially in Agile projects. Retrospectives allow teams to learn from successes and failures.
- Resilient: Testing must be able to respond to change, like all other activities in Agile projects.
These best practices maximize the likelihood of successful testing in Scrum projects.
Sprint zero is the first iteration of the project where many preparation activities take place. The tester collaborates with the team on the following activities during this iteration:
- Identify the scope of the project (i.e., the product backlog)
- Create an initial system architecture and high-level prototypes
- Plan, acquire, and install needed tools (e.g., for test management, defect management, test automation, and continuous integration)
- Create an initial test strategy for all test levels, addressing (among other topics) test scope, technical risks, test types, and coverage goals
- Perform an initial quality risk analysis
- Define test metrics to measure the test process, the progress of testing in the project, and product quality
- Specify the definition of “done”
- Create the task board.
- Define when to continue or stop testing before delivering the system to the customer
Sprint zero sets the direction for what testing needs to achieve and how testing needs to achieve it throughout the sprints.
In Agile projects, the objective is to deliver customervalue on a continuous basis (preferably in every sprint). To enable this, the integration strategy should consider both design and testing. To enable a continuous testing strategy for the delivered functionality and characteristics, it is important to identify all dependencies between underlying functions and features.
Since testing is fully integrated into the Agile team, test planning should start during the release planning session and be updated during each sprint. Test planning for the release and each sprint should address the issues discussed in Section ‘Release and Iteration Planning’.
Sprint planning results in a set of tasks to put on the task board, where each task should have a length of one or two days of work. In addition, any testing issues should be tracked to keep a steady flow of testing.
Agile Testing Practices
Many practices may be useful for testers in a scrum team, some of which include:
- Pairing: Two team members (e.g., a tester and a developer, two testers, or a tester and a product owner) sit together at one workstation to perform a testing or other sprint task.
- Incremental test design: Test cases and charters are gradually built from user stories and other test bases, starting with simple tests and moving toward more complex ones.
- Mindmapping: Mind mapping is a useful tool when testing [Crispin08]. For example, testers can use mind mapping to identify which test sessions to perform, to show test strategies, and to describe test data.
These practices are in addition to other practices.