The following success factors apply to test automation projects that are in operation, and therefore the focus is on influences that affect the long-term success of the project. Factors that affect the success of test automation projects in the pilot phase are not considered here.

Key success factors for test automation include the following:

Test Automation Architecture (TAA)

• The test automation architecture (TAA) is very closely aligned with the architecture of a software product. It should be clear which functional and non-functional requirements the architecture should support. Usually, these will be the most important requirements.
• Often the TAA is designed for maintainability, performance, and learnability. (See ISO/IEC 25000:2014 for details on these and other non-functional properties.) It is helpful to involve software engineers who understand the architecture of the SUT.

Testability of the SUT

• The SUT must be designed to be testable and support automated testing. In the case of GUI testing, this might mean that the SUT should decouple GUI interaction and data from the appearance of the graphical interface as much as possible. Data should be decoupled from the appearance of the graphical interface. In the case of API testing, this could mean that more classes, modules, or the command line interface need to be exposed as public so that they can be tested.
• The testable parts of the SUT should be tested first. In general, a key factor in the success of test automation is the ease of implementing automated test scripts. With this goal in mind, and also to provide a successful proof of concept, the test automation engineer (TAE) must identify modules or components of the SUT that can be easily tested in an automated manner and start there.

Test automation strategy

• A practical and consistent test automation strategy that takes into account the maintainability and consistency of the SUT.
• It may not be possible to apply the test automation strategy in the same way to old and new parts of the SUT. When creating the automation strategy, consider the costs, benefits, and risks of applying it to different parts of the code.
• Consider testing both the user interface and the API with automated test cases to verify consistency of results.

Test Automation Framework (TAF).

A user-friendly, well-documented, and maintainable test automation framework (TAF) supports a consistent approach to automating tests.

To establish a easy to use and maintainable TAF, the following steps must be taken:

• Implement reporting capabilities: Test reporting should provide information (passed/failed/failed/not executed/cancelled, statistical, etc.) about the quality of the SUT. Reporting should provide an overview of quality to the testers, test managers, developers, project managers, and other stakeholders involved.
• Enable easy debugging: In addition to test execution and logging, the TAF must provide an easy way to troubleshoot failed tests. The test may fail due to
  • Errors found in the SUT
  • Errors found in the TAS
  • Problems with the tests themselves or the test environment.
• Take care of the test environment appropriately: Test tools depend on the consistency of the test environment. A dedicated test environment is necessary for automated testing. If there is no control over the test environment and test data, the setup for testing may not meet test execution requirements and produce incorrect execution results.
• Document the automated test cases: The objectives for test automation must be clear, such as which parts of the application should be tested and to what extent, and which attributes should be tested (functional and non-functional). This must be clearly described and documented.
• Tracking of the automated test: TAF must support tracing so that the test automator can trace individual steps back to the test cases.
• Enable easy maintenance: Ideally, the automated test cases should be easy to maintain so that maintenance does not consume a significant portion of the test automation effort. In addition, the maintenance effort must be proportionate to the amount of change made to the SUT. To achieve this, cases must be easily analyzable, modifiable, and extensible. In addition, the reuse of automated testware should be high to minimize the number of items that need to be changed.
• Keep automated tests up to date: If new or changed requirements cause tests or entire test suites to fail, don’t disable the failed tests, fix them.
• Plan for deployment: ensure that test scripts can be easily deployed, modified, and redeployed.
• Retire tests as needed: ensure that automated test scripts can be easily retired when they are no longer useful or necessary.
• Monitor and recover the SUT: In practice, the SUT must be monitored to continuously execute a test case or set of test cases. To execute test cases, the SUT must be continuously monitored. If the SUT encounters a fatal error (such as crash), the TAF must be able to recover, skip the current case, and resume testing with the next case.

Maintaining test automation code can be very complex. It is not uncommon for the code for testing to be as much as the code for the SUT. For this reason, it is of utmost importance that the test code is maintainable. This is due to the different testing tools, the different types of verification, and the different test artifacts that are used. Artifacts that need to be maintained (e.g., test input data, test oracles, test reports).

With these maintenance considerations in mind, in addition to the important items that should be taken care of, there are also some things that should not be done, such as:

• Do not create code that is sensitive to the interface (i.e., it would be affected by changes to the graphical interface or in insignificant parts of the API).
• Do not create test automation that is sensitive to data changes or has a high dependency on specific data values (e.g., test input dependent on other test output).
• Do not create an automation environment that is sensitive to context (e.g., operating system date and time, operating system localization parameters, or the content of another application). In this case, it is better to use test stubs when needed so that the environment can be controlled.

The more success factors are met, the more likely the test automation project will be successful. Not all factors are required, and in practice, rarely are all factors met. Before starting a test automation project, it is important to analyze the project’s chances of success by evaluating the existing and the The risks of the chosen approach and the project context must be taken into account. Once the TAA is in place, it is necessary to investigate which items are missing or still need to be addressed.

Source: ISTQB®: CTAL Test Automation Engineer Syllabus Version 1.0