A test automation engineer (TAE) is tasked with designing, developing, implementing, and maintaining test automation solutions (TAS). The development of each solution involves completing similar tasks, answering similar questions, and addressing and prioritizing similar problems. These recurring concepts, steps, and approaches to test automation form the basis for the generic test automation architecture, or gTAA.

The gTAA presents the layers, components and interfaces of a gTAA, which are then further defined in the specific TAA for a particular TAS. It enables a structured and modular approach to building a test automation solution by:

• Defining the concept space, layers, services and interfaces of a TAS to enable the realization of TAS by both in-house and externally developed components.
• Supporting simplified components for effective and efficient test automation development.
• Reuse test automation components across different or evolving TASs for software product lines and families, and across software technologies and tools.
• Facilitate the maintenance and evolution of TASs.
• Defining the essential characteristics for the user of a TAS.

A TAS consists of both the test environment (and its artifacts) and the test suites (a set of test cases including test data). A test automation framework (TAF) can be used to realize a TAS. It provides support for the realization of the test environment and provides tools, test harnesses or supporting libraries.

It is recommended that the TAA of a TAS conform to the following principles that support easy development, evolution, and maintenance of the TAS:

• Single responsibility: each TAS component must have a single responsibility, and that responsibility must be fully encapsulated within the component. In other words, each component of a TAS should be responsible for exactly one thing, e.g., generating keywords or data, creating test scenarios, executing test cases, logging results, generating execution reports.
• Extension: Each TAS component must be open to extension, but closed to change. This principle means that it should be possible to change the behavior of the components or to extend the behavior of the components without breaking the backward compatible functionality.
• Substitution: Each TAS component must be substitutable without affecting the overall behavior of the TAS. The component may be replaced by one or more substitute components, but the behavior shown must be the same.
• Component separation: it is better to have more specific components than a generic, general-purpose component. This makes substitution and maintenance easier by avoiding unnecessary dependencies.
• Reverse dependencies: The components of a TAS must depend on abstractions, not depend on details on details. In other words, components should not depend on specific automated test scenarios.

Typically, a gTAA-based TAS is implemented by a set of tools, their plugins and/or components.
It is important to note that the gTAA is vendor neutral: it does not specify a specific method, technology, or tool for implementing a TAS. The gTAA can be implemented using any software engineering approach Software engineering approach can be implemented using any software engineering approach, e.g., structured, object-oriented, service-oriented, model-driven, and any software technologies and tools. In fact, a TAS is often implemented using standard tools, but typically requires additional SUT-specific additions and/or customizations.

Other guidelines and reference models related to TAS include software engineering standards for the chosen SDLC (Software Development Lifecycle), programming technologies, formatting standards, etc. It is not within the scope of this curriculum to teach software engineering in general, however, a TAE is expected to have skills, experience, and expertise in software engineering.

In addition, a TAE must be familiar with industry coding and documentation standards and best practices to use in the development of a TAS. These practices can increase the maintainability, reliability, and security of the TAS. Such standards are usually domain-specific. Popular standards include:

• MISRA for C or C++
• JSF coding standard for C++
• AUTOSAR rules for MathWorks Matlab/Simulink®

Overview of the gTAA

The gTAA is organized into horizontal layers for the following:

• Test generation
• Test definition
• Test execution
• Test adaptation

The gTAA (see Figure 1: The Generic Test Automation Architecture) includes the following:

• The test generation layer that supports the manual or automated design of test cases. It provides the means for designing test cases.
• The test definition layer, which supports the definition and implementation of test suites and/or test cases cases. It separates the test definition from the SUT and/or test system technologies and tools. It contains means for defining high-level and low-level tests that are handled in the test data, test cases, test procedures, and test library components, or combinations thereof.
• The test execution layer, which supports test case execution and test logging. It provides a test execution tool to automatically execute the selected tests, as well as a logging and component.
• The test adaptation layer, which provides the necessary code to adapt the automated tests to the different components or interfaces of the SUT. It provides various adaptors for connecting to the SUT via APIs, protocols, services and others.
• It also has interfaces for project management, configuration management and test management related to test automation. For example, the interface between test management and test adaptation layer governs the selection and the test adaptation layer governs the selection and configuration of the appropriate adaptors with respect to the test configuration chosen for the test.

The interfaces between the gTAA layers and their components are typically specific and therefore are not discussed further here.

It is important to understand that these layers may be present or absent in any given TAS. Example:

• If test execution is to be automated, the test execution and test customization layers must be used. They do not need to be separated and can be implemented together, e.g. in unit test frameworks.
• If test definition is to be automated, the test definition layer is needed.
• If test generation is to be automated, the test generation layer is needed.

The most common way to start implementing a TAS is from the bottom up, but other approaches such as automatic test generation for manual tests can also be useful. In general, it is advisable to implement the TAS in incremental steps (e.g., sprints) to be able to deploy the TAS as quickly as possible and to prove the added value of the TAS. Proofs of concept are also recommended as part of a test automation project. Project recommended.

Any test automation project must be understood, set up, and managed like a software development project Project must be understood, set up, and managed like a software development project and requires dedicated project management. Project management for TAF development (i.e., supporting test automation for an entire company, product families, or product lines) can be separated from project management for TAS (i.e., test automation for a specific product).

Figure 1: The Generic Test Automation Architecture

Test generation layer

The test generation layer consists of tool support for the following:

• Manually designing test cases
• Developing, capturing, or deriving test data
• Automatically generate test cases from models that define the SUT and/or its environment (i.e., automated model-based testing).

The components in this layer are used to:

• Edit and navigate test-suite structures
• Relate test cases to test objectives or SUT requirements
• Document the test design

For automated test generation, the following capabilities may also be included:

• Ability to model the SUT, its environment, and/or the test system.
• Ability to define test policies and configure/parameterize test generation algorithms.
• Ability to trace the generated tests back to the model (elements).

Test Definition Layer

The test definition layer consists of tool support for the following:

• Specification of test cases (high level and/or low level).
• Definition of test data for low-level test cases.
• Specification of test procedures for a test case or a set of test cases.
• Definition of test scripts for the execution of the test cases.
• Providing access to test libraries as needed (e.g., for keyword-based approaches).

The components in this layer are used to:

• Partition/constrain, parameterize, or instantiate test data.
• Specify test sequences or full-fledged test behavior (including control statements and expressions), parameterize and/or group them together.
• Document test data, test cases and/or test sequences.

Test Execution Layer

The test execution layer consists of tool support for the following:

• Automatic execution of test cases
• Logging of test case executions
• Reporting of test results

The test execution layer may consist of components that provide the following capabilities:

• Setup and teardown of the SUT for test execution.
• Set up and tear down test suites (i.e., a set of test cases including test data).
• Configure and parameterize the test setup.
• Interpreting test data and test cases and converting them into executable scripts.
• Instrumenting the test system and/or the SUT for (filtered) logging of test execution and/or for fault injection.
• Analyze SUT responses during test execution to guide subsequent test runs.
• Validation of SUT responses (comparison of expected and actual results) for automated test case execution results.
• Timing control of automated test execution.

Test adaptation layer

The test adaptation layer consists of tool support for the following:

• Control of the test harness
• Interaction with the SUT
• Monitoring of the SUT

Simulation or emulation of the SUT environment The test adaptation layer provides the following functions:

• Mediation between the technology-neutral test definitions and the specific technology requirements of the SUT and the test equipment.
• Applying various technology-specific adaptors to interact with the SUT.
• Distributing test execution across multiple test devices/test interfaces or executing tests locally.

Configuration management of a TAS

Typically, a TAS is developed in different iterations/versions and needs to be compatible with the iterations/versions of the SUT. Configuration management of a TAS may include the following:

• Test models
• Test definitions/specifications including test data, test cases, and libraries
• Test scripts
• Test execution engine and complementary tools and components
• Test adapters for the SUT
• Simulators and emulators for the SUT environment
• Test results and test reports

These elements make up the testware and must be in the correct version to match the version of the SUT. In some situations, it may be necessary to revert to earlier versions of the TAS, such as when field problems need to be reproduced with older SUT versions. Good configuration management enables this capability.

Project Management for a TAS

Since any test automation project is a software project, it requires the same project management as any other software project. A TAE must perform the tasks for all phases of the established SDLC methodology during the development of the TAS. In addition, a TAE must understand that the TAS development environment should be designed so that status information (metrics) can be easily extracted or automatically reported to TAS project management.

TAS Support for Test Management.

A TAS must support test management for the SUT. Test reports, including test logs and test results, shall be easily extracted or automatically reported to the SUT’s test management (people or system).

Source: ISTQB®: CTAL Test Automation Engineer Syllabus Version 1.0