Test Automation Concept and Planning

There are two main activities in implementing and rolling out a TAS: piloting and deployment.
The steps that make up these two activities vary depending on the type of TAS and the specific situation.

At a minimum, the following steps should be considered for piloting:

  • Identifying an appropriate project
  • Planning the pilot project
  • Implementation of the pilot project
  • Evaluation of the pilot project

At a minimum, the following steps should be considered for implementation:

  • Identify initial target project(s).
  • Implement the TAS in the selected projects
  • Monitor and evaluate TAS in the projects after a specified period of time.
  • Extend to the rest of the organization/projects.

Pilot project

The implementation of a tool usually starts with a pilot project. The objective of the pilot project is to ensure that the TAS can be used to achieve the intended benefits. The goals of the pilot include:

  • Obtain more detailed information about the TAS.
  • Examine how the TAS fits with existing processes, procedures, and tools and determine the extent to which they need to be modified. (It is usually preferable to modify the TAS to fit existing processes/procedures. If these need to be modified to “support” the TAS, this should at least be an improvement to the processes themselves.)
  • Design the automation interface to meet the needs of the auditors.
  • Establish standard practices for using, managing, storing, and maintaining the TAS and test assets, including integration with configuration and change management (e.g., establishing file and test naming conventions, creating libraries, and defining test suite modularity).
  • Identify metrics and measurement methods to monitor test automation in use, including usability, maintainability, and extensibility.
  • Evaluate whether benefits can be achieved at a reasonable cost. This is an opportunity to redefine expectations once TAS is deployed.
  • Determine what capabilities are required and which of them are present and which are missing.

Identify an appropriate project
The pilot project should be carefully selected using the following guidelines:

  • Do not select a critical project. If the introduction of TAS causes delays, it should not have a major impact on critical projects. The introduction of TAS will take time at the beginning. The project team should be aware of this.
  • Do not choose a trivial project. A trivial project is not a good candidate because the success of the rollout will not predict the success of non-trivial projects, resulting in less information available for the rollout.
  • Involve the necessary stakeholders (including management) in the selection process.
  • The pilot project’s SUT should be a good reference for the organization’s other projects, e.g., the SUT should include representative GUI components that need to be automated.

Plan the pilot project
The pilot project should be treated like a regular development project: Plan it, reserve budget and resources, report on progress, define milestones, etc. It is especially important to ensure that the people working on the TAS rollout (i.e., a “champion”) can devote sufficient time to the rollout even when other projects are taking up resources for their activities. It is important to have senior management commitment, especially for shared resources. These individuals will likely not be able to work full time on the implementation.
If the TAS was developed in-house rather than provided by a vendor, the appropriate developers will need to be involved in the deployment activities.

Conduct the pilot
Conduct the pilot phase of the deployment, paying attention to the following issues:

  • Does the TAS provide the functionality expected (and promised by the vendor)? If not, this needs to be addressed as soon as possible. If the TAS was developed in-house, the appropriate developers need to support the rollout by providing missing functionality.
  • Do the TAS and the existing process support each other? If not, they need to be aligned.

Evaluate the pilot project
Involve all stakeholders in the evaluation.


Following the evaluation of the pilot, the TAS should be rolled out to the entire department/organization only if the pilot is deemed successful. The rollout should be incremental and well managed. Success factors for the rollout include:

  • An incremental rollout: Phase the rollout to the rest of the organization, in small increments. This way, support for the new users is introduced in “waves” rather than all at once. In this way, TAS usage can be increased incrementally. Potential bottlenecks can be identified and resolved before they become real problems. Licenses can be added as needed.
  • Aligning and improving processes to use the TAS: When different users use the TAS, different processes come into contact with the TAS and need to be aligned with the TAS, or the TAS needs to make (small) adjustments to the processes.
  • Training and coaching/mentoring for new users: new users need training and coaching on how to use the new TAS. Ensure this is in place. Training/workshops should be offered to users before they actually use the TAS.
  • Establish usage guidelines: it is possible to create guidelines, checklists, and FAQs for using the TAS. This can prevent extensive support requests.
  • Implement a process to collect information on actual usage: there should be an automated way to collect information on actual usage of the TAS. Ideally, not only the usage itself, but also which parts of the TAS (specific functionalities) are used. In this way, the use of the TAS can be easily monitored.
  • Monitoring TAS usage, benefits, and costs: monitoring the usage of the TAS over a period of time shows whether the TAS is actually being used. This information can also be used to recalculate the business case (e.g., how much time was saved, how many problems were avoided).
  • Support the test and development teams for a given TAS.
  • Gather lessons learned from all teams: Conduct evaluation/retrospective sessions with the different teams using TAS. Lessons can be learned in this way. Teams will feel that their input is needed and desired to improve the use of TAS.
  • Identify and implement improvements: Based on the team’s feedback and monitoring of the TAS, identify and implement improvement actions. Also communicate this clearly to stakeholders.

Use of the TAS in the software life cycle.

The use of a TAS depends heavily on the development phase of the software project to be tested by the TAS.

Usually, a new TAS or a new version of it is implemented either at the beginning of the project or when a milestone is reached, such as the code freeze or the end of a sprint. The reason for this is that the implementation activities, with all the testing and changes, take time and effort. Also, this is a good way to mitigate the risk of the TAS not working and causing interruptions in the test automation process. However, if there are critical issues that need to be fixed for the TAS, or if a component of the environment it is running in needs to be replaced, then the deployment is done independently of the development phase of the SUT.

Source: ISTQB®: CTAL Test Automation Engineer Syllabus Version 1.0

Was this article helpful?

Related Articles