Technical problems can lead to product or project risks. Typical technical problems include:

• Too much abstraction can lead to difficulty understanding what is really happening (e.g., keywords)
• Data-driven: data tables can become too large/complex/difficult to understand
• Dependence of TAS on the use of certain operating system libraries or other components that may not be available in all target environments of the SUT

Typical risks of a deployment project include:

• Staffing issues: finding the right people to maintain the code base can be difficult.
• New SUT deliveries may cause the TAS to not function correctly
• Delays in introducing automation
• Delays in updating the TAS based on changes to the SUT.
• The TAS cannot track the (non-standard) objects it is supposed to track.

Possible failure points for the TAS project include:

• Migration to a different environment
• Deployment to the target environment
• Redelivery from development

There are a number of risk mitigation strategies that can be employed to deal with these risk areas. These are discussed below.

The TAS has its own software lifecycle, whether it is an internally developed solution or an acquired solution. It should be noted that, like any other software, the TAS is subject to version control and its functions must be documented. Otherwise, it becomes very difficult to deploy different parts of it and make them work together or in specific environments.

There must also be a documented, clear, and easy-to-follow deployment procedure. This procedure is version dependent; therefore, it must also be included in version control.

There are two different cases when deploying a TAS:

1. Initial deployment
2. Maintenance deployment – the TAS already exists and needs to be maintained.

Before beginning the initial deployment of a TAS, it is important to ensure that it can run in its own environment, that it is isolated from random changes, and that test cases can be updated and managed. Both the TAS and its infrastructure must be maintained.

Initial deployment requires the following basic steps:

• Define the infrastructure in which the TAS will run.
• Create the infrastructure for the TAS
• Establish a procedure for maintaining the TAS and its infrastructure.
• Establish a procedure for maintaining the test suite that will run the TAS.

Initial deployment risks include:

• The total execution time of the test suite may be longer than the planned execution time for the test cycle. In this case, it is important to ensure that the test suite is given sufficient time to fully execute before the next scheduled test cycle begins.
• There are issues with the installation and configuration of the test environment (e.g., database setup and initial load, start/stop of services). In general, the TAS needs an effective method to create the necessary conditions for the automated test cases within the test environment.

There are additional considerations for maintenance deployments. The TAS itself needs to evolve, and updates to it need to be deployed to production. Before an updated version of the TAS is deployed to production, it must be tested like any other software. It is therefore necessary to test the new features to ensure that the test suite can run on the updated TAS, that reports can be sent, and that there are no performance issues or other functional regressions. In some cases, the entire test suite may need to be modified to match the new version of the TAS.

When deploying maintenance, the following steps are required:

• Evaluate the changes in the new version of the TAS compared to the old version.
• Test the TAS for both new features and regressions.
• Checking whether the test suite needs to be adapted to the new version of the TAS.

An upgrade also carries the following risks and corresponding mitigations:

• The test-suite must be modified to run on the updated TAS: Make the necessary changes to the test suite and test it before deploying it on the TAS.
• The stubs, drivers, and interfaces used in testing must be modified to match the updated TAS: Make the necessary changes to the test harness and test it before deploying it on the TAS.
• Infrastructure must be adapted to the updated TAS: Make an assessment of the infrastructure components that need to be changed, make the changes, and test them with the updated TAS.
• The updated TAS has additional deficiencies or performance issues: Perform a risk-benefit analysis. If the problems discovered make updating the TAS impossible, it may be best not to update or to wait for a next version of the TAS. If the problems are negligible compared to the benefits, the TAS can still be updated. Be sure to create a release note with known issues to notify test automation engineers and other stakeholders, and try to get an estimate of when the issues will be fixed.

Source: ISTQB®: CTAL Test Automation Engineer Syllabus Version 1.0