The development of test automation solutions is not trivial. They must be modular, scalable, understandable, reliable and testable. To add to the complexity, test automation solutions – like any other software system – need to evolve. Whether due to internal changes or changes in the environment in which they are deployed
maintenance is an important aspect of the architecture of a TAS. Maintaining the TAS by adapting to new types of systems under test, supporting new software environments, or adapting to new laws and regulations helps ensure the reliable and safe operation of the TAS. It also optimizes the service life and performance of the TAS.

Types of Maintenance

Maintenance is performed on an existing, in-service TAS and is triggered by changes, migration, or retirement of the system. This process can be divided into the following categories:

• Preventive Maintenance – Changes are made to allow the TAS to support more test types, test on multiple interfaces, test multiple versions of the SUT, or support test automation for a new SUT.
• Corrective maintenance – Changes are made to correct errors in the TAS. The best way to keep a TAS in operation, and thus reduce risk during use, is to perform regular maintenance testing.
• Perfect maintenance – The TAS is optimized and non-functional problems are corrected. They may affect the performance of the TAS, its usability, robustness or reliability.
• Adaptive Maintenance – As new software systems come to market (operating systems, database managers, web browsers, etc.), the TAS may be required to support them. It may also be the case that the TAS must comply with new laws, regulations, or industry-specific requirements. In this case, changes will be made to the TAS to adapt it accordingly. Note: Typically, compliance with laws and regulations results in mandatory maintenance with specific rules, requirements, and sometimes audit requirements. As integration tools are updated and new versions are created, tool integration endpoints must also be maintained and kept functional.

Scope and Approach

Maintenance is a process that can affect all layers and components of a TAS. The scope depends on:

• The size and complexity of the TAS
• The scope of the change
• The risk of the change

Because maintenance applies to an in-service TAS, an impact analysis is required to determine how the system may be affected by the changes. Depending on the impact, changes may need to be phased in and testing performed after each step to ensure the continued functioning of the TAS. Note: Maintenance of the TAS can be difficult if its specifications and documentation are outdated.

Since time efficiency is the most important factor in the success of test automation, it is critical to have good practices for maintaining the TAS, including:

• Deployment procedures and use of the TAS must be clear and documented.
• Third-party dependencies must be documented, along with drawbacks and known issues.
• The TAS must be modular so that parts of it can be easily replaced.
• The TAS must run in an environment that is interchangeable or consists of interchangeable components.
• The TAS must separate the test scripts from the TAF itself.
• The TAS must run in isolation from the development environment so that changes to the TAS do not adversely affect the test environment.
• The TAS must be under configuration management along with the environment, test suite, and testware artifacts.

There are also considerations for maintaining third-party components and other libraries as follows:

• Very often it is the case that the TAS uses third-party components to run the tests. It may also be the case that the TAS depends on third-party libraries (e.g., the UI automation libraries). All third-party components of the TAS must be documented and under configuration management.
• It is necessary to have a plan in case these external components need to be modified or repaired. The person responsible for TAS maintenance must know who to contact or where to report a problem.
• Documentation must be available on the license under which the third-party components are used so that it is known if, to what extent, and by whom they can be modified.
• Information about updates and new versions must be available for each third-party component. Keeping third-party components and libraries up to date is a preventive measure that pays off in the long run.

Considerations for naming standards and other conventions:

• The idea of naming standards and other conventions is for a simple reason: the test suite and the TAS itself must be easy to read, understand, change, and maintain. This saves time in maintenance and minimizes the risk of regressions or miscorrections that could otherwise be easily avoided.
• It is easier to introduce new staff to the test automation project when standard naming conventions are used.
• Naming standards can refer to variables and files, test scenarios, keywords, and keyword parameters. Other conventions relate to pre- and post-conditions for test execution, the content of test data, the test environment, the status of test execution, and execution logs and reports.
• All standards and conventions must be agreed upon and documented at the beginning of a test automation project.

Documentation considerations include:

• The need for good, up-to-date documentation for both the test scenarios and the TAS is obvious, but there are two problems with this: someone has to write it and someone has to maintain it.
• While the test tool code may be either self-documenting or semi-automatically documented, all designs, components, integrations with third parties, dependencies, and deployment procedures must be documented by someone.
• It is good practice to introduce the creation of documentation as part of the development process.
  A task should not be considered complete until it is documented or the documentation is updated.

Considerations for training materials include:

• If the documentation is well written for the TAS, it can be used as the basis for the TAS training material.
• The training material consists of a combination of functional specifications of the TAS, design and architecture of the TAS, use and maintenance of the TAS, use of the TAS (user manual), practical examples and exercises, and tips and tricks.
• The maintenance of the training material consists of writing it first and then reviewing it at regular intervals. In practice, it is performed by the team members who are designated as trainers for the TAS and most likely takes place towards the end of an iteration of the SUT lifecycle (e.g., at the end of a sprint).

Source: ISTQB®: CTAL Test Automation Engineer Syllabus Version 1.0