Introduction to TAA Design

There are a number of key activities required to design a TAA that can be arranged according to the needs of the test automation project or organization. These activities are discussed in the following sections. Depending on the complexity of the TAA, more or fewer activities may be required.

Gather the requirements needed to define an appropriate TAA.
The requirements for a test automation approach must consider the following:

• What activity or phase of the test process is to be automated, e.g., test management, test design, test generation, or test execution. Note that test automation refines the basic test process by inserting test generation between test design and test execution.
• What level of testing should be supported, e.g., component level, integration level, system level.
• What type of test should be supported, e.g. functional testing, conformance testing, interoperability testing
• Which test role should be supported, e.g. test executor, test analyst, test architect, test manager
• Which software product, software product line, software product family should be supported, e.g., to define the time period and lifetime of the implemented TAS
• Which SUT technologies should be supported, e.g. to define the TAS in terms of compatibility with SUT technologies

Compare and contrast different design/architectural approaches.
The TAE shall analyze the advantages and disadvantages of different approaches in designing selected layers of the TAA. These include, but are not limited to:

Considerations for the test generation layer:

• Selection of manual or automated test generation.
• Selection of, for example, requirements-based, data-based, scenario-based, or behavior-based test generation
• Choice of test generation strategies (e.g., model coverage such as classification trees for data-based approaches, use case/exception coverage for scenario-based approaches, transition/state/path coverage for behavior-based approaches, etc.)
• Choice of test selection strategy. In practice, fully combinatorial test generation is not feasible as it may lead to an explosion of test cases. Therefore, practical coverage criteria, weightings, risk assessments, etc. should be used to guide test generation and subsequent test selection

Considerations for the test definition layer:

• Selecting a data-driven, keyword-driven, pattern-based, or model-driven test definition
• Selection of test definition notation (e.g., spreadsheets, state-based notation, stochastic notation, data flow notation, business process notation, scenario-based notation, etc., using spreadsheets, domain-specific test languages, Testing and Test Control Notation (TTCN-3), UML Testing Profile (UTP), etc.)
• Selection of style guides and guidelines for defining high quality tests
• Selection of test case repositories (spreadsheets, databases, files, etc.)

Test execution layer considerations:

• Selection of test execution tool
• Choice of interpretation (using a virtual machine) or compilation approach for implementing test procedures – this choice typically depends on the test execution tool chosen
• Choice of implementation technology for test execution (imperative, e.g., C; functional, e.g., Haskell or Erlang; object-oriented, e.g., C++, C#, Java; scripting, e.g., Python or Ruby, or a tool-specific technology) – this choice is usually dependent on the test execution tool chosen
• Selection of auxiliary libraries to facilitate test execution (e.g., test instrument libraries, coding/decoding libraries, etc.)

Considerations for the test adaptation layer:

• Selection of test interfaces to the SUT
• Selection of tools to stimulate and observe the test interfaces
• Selection of tools to monitor the SUT during test execution
• Selection of tools to track test execution (e.g., including timing of test execution)

Identifying areas where abstraction can provide benefits
Abstraction in a TAA enables technology independence, as the same test suite can be used in different test environments and on different target technologies. Portability of test artifacts is increased. In addition, vendor neutrality is ensured, avoiding lock-in effects for a TAS. Abstraction also improves maintainability and adaptability to new or evolving SUT technologies. In addition, abstraction helps make a TAA (and its instantiations by TAS) more accessible to non-technical people, as test suites can be documented (including graphical means) and explained at a higher level, improving readability and understandability.

The TAE needs to discuss with software development, quality assurance, and testing stakeholders which level of abstraction should be used in which area of the TAS. For example, which test adaptation and/or test execution layer interfaces need to be externalized, formally defined, and kept stable throughout the life of the TAA? It must also be discussed whether an abstract test definition will be used or whether the TAA will use a test execution layer with test scripts only. Similarly, it needs to be clarified whether test creation is abstracted through the use of test models and model-based testing approaches. The TAE must be aware that there are tradeoffs between sophisticated and simple implementations of a TAA in terms of overall functionality, maintainability, and extensibility. When deciding which abstraction to use in a TAA, these trade-offs must be considered.

The more abstraction used for a TAA, the more flexible it is in terms of further development or transition to new approaches or technologies. This comes with higher initial investments (e.g., more complex test automation architecture and tools, higher skill requirements, larger learning curves), which delays the initial breakeven but may pay off in the long run. It can also lead to lower TAS performance.

While detailed return on investment (ROI) considerations are the responsibility of the TAM, the TAE must provide inputs to the ROI analysis by providing technical assessments and comparisons of various test automation architectures and approaches in terms of schedule, cost, effort, and benefits.

Understanding SUT technologies and how they relate to TAS
Access to the SUT test interfaces is central to any automated test execution. Access can be provided at the following levels:

• Software level, e.g., SUT and test software are interconnected
• API level, e.g. the TAS calls the functions/operations/methods provided at a (remote) application programming interface
• Protocol level, e.g. the TAS interacts with the SUT via HTTP, TCP, etc.
• Service level, e.g. the TAS interacts with the SUT services via web services, RESTful services, etc.

In addition, the TAE must decide which interaction paradigm of the TAA to use for interaction between the TAS and the SUT when the TAS and SUT are separated by APIs, protocols, or services. These paradigms include the following:

• Event-driven paradigm, which controls interaction via events exchanged over an event bus
• Client-server paradigm, which controls interaction via service invocations from the service requester to the service provider
• Peer-to-peer paradigm, which controls interaction via service invocations from both peers

Often the choice of paradigm depends on the SUT architecture and may have implications for the SUT architecture. The connection between the SUT and the TAA must be carefully analyzed and designed to choose a future-proof architecture between the two systems.

Understanding the SUT environment
A SUT can be stand-alone software or software that functions only in conjunction with other software (e.g., system of systems), hardware (e.g., embedded systems), or environmental components (e.g., cyber-physical systems). A TAS simulates or emulates the SUT environment as part of an automated test setup.
Examples of test environments and use cases include the following:

• A computer with both the SUT and the TAS – useful for testing a software application
• Individual networked computers for a SUT and a TAS – useful for testing server software
• Additional test equipment to stimulate and observe the technical interfaces of a SUT – useful for testing the software e.g. on a set-top box
• Networked test equipment to emulate the operating environment of the SUT – useful for testing the software of a network router
• Simulators to simulate the physical environment of the SUT – useful for testing the software of an embedded control unit

Time and complexity to implement a particular testware architecture
While effort estimation for a TAS project is the responsibility of a TAM, a TAE must assist a TAM by providing good estimates for the time and complexity of a TAA design. Methods for estimates and examples include the following:

• Analogy-based estimation such as function points, three-point estimation, broadband Delphi, and expert estimates
• Estimation using work breakdown structures, such as those found in management software or project templates
• Parametric estimates such as the Constructive Cost Model (COCOMO)
• Size-based estimates such as Function Point Analysis, Story Point Analysis, or Use Case Analysis
• Group estimates such as Planning Poker

Usability for a particular implementation of the testware architecture
In addition to the functionality of the TAS, its compatibility with the SUT, its long-term stability and evolvability, its effort requirements, and ROI considerations, a TAE has a special responsibility to address usability issues of a TAS. This includes, but is not limited to:

• Tester-oriented design
• Usability of the TAS
• TAS support for other software development, quality assurance, and project management roles
• Efficient organization, navigation, and searching in/with the TAS
• Useful documentation, manuals and help texts for the TAS
• Practical reporting by and about TAS
• Iterative designs to incorporate TAS feedback and empirical findings

Approaches to automating test cases

Test cases must be translated into sequences of actions that are executed against a SUT. This sequence of actions can be documented in a test procedure and/or implemented in a test script.
In addition to the actions, the automated test cases should define test data for interaction with the SUT and include verification steps to verify that the expected result was achieved by the SUT. Several approaches can be used to create the action sequence:

1. The TAE implements test cases directly in automated test scripts. This option is the least recommended because it lacks abstraction and increases maintenance overhead
2. The TAE designs test procedures and converts them into automated test scripts. This option provides abstraction, but lacks the automation to create the test scripts
3. The TAE uses a tool to translate test procedures into automated test scripts. This option combines both abstraction and automated scripting
4. The TAE uses a tool that generates automated test procedures and/or translates the test scripts directly from models. This option has the highest level of automation

Note that the options are highly dependent on the context of the project. It can also be efficient to start test automation with one of the less advanced options, as they are usually easier to implement.
This can add value in the short term, although it leads to a less maintainable solution.

Established approaches to test case automation include:

• Capture/playback approach, which can be used for Option 1
• Structured scripting approach, data-driven approach and keyword-driven approach, which can be used for Option 2 or 3
• Model-based testing (including the process-driven approach), which can be used for Option 4

These approaches are discussed below in terms of key concepts and advantages and disadvantages.

Capture/Playback Approach

Pricipal concept
Capture/playback approaches use tools to record interactions with the SUT during execution of the sequence of actions defined by a test procedure. Inputs are recorded; outputs can also be recorded for later review. During event replay, there are several manual and automated ways to verify output:

• Manual: the tester must monitor SUT outputs for anomalies
• Complete: all system outputs recorded during the replay must be reproduced by the SUT
• Exact: all system outputs recorded during the recording must be reproduced by the SUT to the level of detail of the recording
• Checkpoints: only selected system outputs are checked for specified values at specific times

Advantages
The capture/playback approach can be used for SUTs at the GUI and/or API level. First, it is easy to set up and use.

Disadvantages
Record/replay scripts are difficult to maintain and evolve because the recorded SUT execution depends heavily on the SUT version from which the recording originated. For example, when recording at the GUI level, changes in the GUI layout can affect the test script, even if it is only a change in the positioning of a GUI element. Therefore, capture/replay approaches remain vulnerable to changes.
Implementation of test cases (scripts) cannot begin until the SUT is available.

Linear scripting

Principal concept
As with all scripting techniques, linear scripting starts with some manual test procedures. Note, however, that these are not necessarily written documents – the knowledge of which tests to run and how may be “known” to one or more test analysts.
Each test is executed manually while the test tool records the sequence of actions and, in some cases, captures the visible output of the SUT on the screen. This generally results in a (typically large) script for each test operation. Recorded scripts can be edited to improve readability (e.g., by adding comments to explain what happens at key points) or to add more tests using the tool’s scripting language.
The scripts can then be replayed by the tool so that the tool repeats the same actions that the tester performed when the script was recorded. Although this can be used to automate GUI tests, it is not a good technique if a large number of tests are to be automated and they are required for many versions of the software. The reason for this is the high maintenance costs that are typically caused by
changes to the SUT cause (each change to the SUT may require many changes to the recorded scripts).

Advantages
The advantages of linear scripts are primarily that little or no preparatory work is required before you can start automating. Once you learn to use the tool, all you have to do is record and play back a manual test (although the recording part may require additional interaction with the test tool to request comparisons of actual output with expected output to verify that the software is working correctly). Programming skills are not required, but are usually helpful.

Disadvantages
The disadvantages of linear scripts are numerous. The effort required to automate a particular test procedure depends mainly on the size (number of steps or actions) required to perform it. For example, the 1000th test procedure to be automated will require a similar proportional effort as the 100th test procedure. In other words, there is not much room to reduce the cost of creating new automated tests.
If there were a second script that performed a similar test, but with different input values, that script would contain the same sequence of statements as the first script; only the information contained in the statements (called statement arguments or parameters) would differ. If there were multiple tests (and thus scripts), they would all contain the same sequence of instructions, which would have to be maintained whenever changes were made to the software that affected the scripts.
Since the scripts are written in a programming language and not in a natural language, they can be difficult to understand for non-programmers. Some testing tools use proprietary languages (only for the particular tool), so it takes some time to learn the language and master it.
Recorded scripts contain only general statements in the comments, if at all. Long scripts, in particular, should have comments to explain what is going on in each test step.
This makes maintenance easier. Scripts can quickly become very large (with many statements) if the test involves many steps.
Scripts are not modular and are difficult to maintain. Linear scripting does not follow the usual paradigms of software reusability and modularity and is tightly coupled to the tool used.

Structured scripting

Principal concept
The main difference between structured scripting and linear scripting is the introduction of a script library. This contains reusable scripts that execute sequences of statements that are commonly needed in a set of tests. Good examples of such scripts are those that drive the operations of SUT interfaces, for example.

Advantages
The benefits of this approach include a significant reduction in the number of maintenance changes required and a reduction in the cost of automating new tests (since existing scripts can be used instead of having to create them all from scratch).
The benefits of structured scripting are largely achieved through script reuse. More tests can be automated without having to create the amount of scripts that would be required with a linear scripting approach. This has a direct impact on creation and maintenance costs. The second and subsequent tests are less costly to automate because some of the scripts created to implement the first test can be reused.

Disadvantages
The initial effort required to create the common scripts can be seen as a disadvantage, but this initial investment should pay off if approached correctly. Programming skills are required to create all scripts, as simple recording alone is not enough. The script library must be well managed, i.e. the scripts should be documented and it should be easy for technical test analysts to find the scripts they need (a reasonable naming convention will help here).

Data-driven testing

Principal concept
The data-driven scripting technique builds on the structured scripting technique. The main difference is how the test inputs are handled. The inputs are extracted from the scripts and stored in one or more separate files (usually called data files).
This means that the main test script can be reused to implement a number of tests (rather than just a single test). Usually, the “reusable” main test script is called a “control script”. The control script contains the sequence of commands required to run the tests, but reads the input data from a data file. One control script can be used for many tests, but is usually not sufficient to automate a wide range of tests. Therefore, a number of control scripts are required, but they represent only a fraction of the tests to be automated.

Advantages
The cost of adding new automated tests can be significantly reduced using this scripting technique.
This technique is used to automate many variations of a useful test, adding depth to the tests in a particular area and increasing test coverage.
Since the tests are “described” by the data files, test analysts can specify “automated” tests simply by inserting one or more data files. This gives test analysts more freedom to specify automated tests without being so dependent on technical test analysts (who can be a scarce resource).

Disadvantages
The need to manage data files and ensure they can be read by TAS is a disadvantage, but can be properly addressed.
Also, important negative test cases can be overlooked. Negative testing is a combination of test procedures and test data. In an approach that focuses primarily on test data, “negative test procedures” may be overlooked.

Keyword-driven testing

Principal concept
The keyword-driven scripting technique builds on the data-driven scripting technique. There are two main differences: (1) the data files are now called “test specification files” or similar (e.g., action word files); and (2) there is only one control script.
A test specification file contains a description of the tests in a form that should be easier for test analysts to understand (easier than the corresponding data file). It usually contains data like the data files, but keyword files also contain high-level instructions (the keywords or “action words”).
The keywords should be chosen to be meaningful to the test analyst, the tests being described, and the application being tested. They are most commonly (but not exclusively) used to represent high-level business interactions with a system (e.g., “place an order”). Each keyword represents a series of detailed interactions with the system under test. Sequences of keywords (including relevant test data) are used to specify test cases. Special keywords may be used for verification steps, or the keywords may contain both the actions and the verification steps.
The responsibility of the test analysts includes the creation and maintenance of the keyword files.
So, once the supporting scripts are implemented, test analysts can add “automated” tests by simply specifying them in a keyword file (as in data-driven scripting).

Advantages
Once the control script and supporting scripts for the keywords are written, the cost of adding new automated tests is significantly reduced by this scripting technique.
Since the tests are “described” by the keyword files, test analysts can specify “automated” tests simply by describing the tests using the keywords and associated data. This gives test analysts more freedom to specify automated tests without depending on technical test analysts (who can be a scarce resource). The advantage of the keyword-driven approach over the data-driven approach is the use of keywords. Each keyword should represent a sequence of detailed actions that provide a meaningful result. For example, “create account,” “place order,” “check order status” are all possible actions for an online shopping application, each involving a series of detailed steps. When one test analyst describes a system test to another test analyst, they are likely to be talking about these higher-level actions, not the detailed steps.
So the goal of the keyword-driven approach is to implement these higher-level actions and allow tests to be defined in terms of the higher-level actions without referring to the detailed steps.
These test cases are easier to maintain, read, and write because the complexity can be hidden in the keywords (or libraries, in the case of a structured scripting approach). The keywords can provide an abstraction from the complexity of the interfaces of the SUT.

Disadvantages
Implementing the keywords remains a major task for test automators, especially when using a tool that does not provide support for this scripting technique. For small systems, implementation may be too much of a hassle and the cost would outweigh the benefits.
Care must be taken to ensure that the correct keywords are implemented. Good keywords are often used in many different tests, while bad keywords are likely to be used only once or a few times.

Process-driven scripting

Principal concept
The process-driven approach builds on the keyword-driven scripting technique, except that scenarios – representing use cases of the SUT and variants thereof – form the scripts that are parameterized with test data or combined into higher-level test definitions.
Such test definitions are easier to handle because the logical relationship between actions, e.g. “check order status” after “place order” in feature tests or “check order status” without previous “place order” in robustness tests, can be specified.

Advantages
The use of a process-like, scenario-based definition of test cases makes it possible to define test procedures from a workflow perspective. The goal of the process-driven approach is to implement these high-level workflows using test libraries that represent the detailed test steps (see also keyword-driven approach).

Disadvantages
The processes of a SUT may not be easy for a technical test analyst to understand – and this also applies to the implementation of the process-oriented scripts, especially if the tool does not support business process logic.
Care must also be taken to implement the right processes by using the right keywords. Good processes will be referenced by other processes and result in many relevant tests, while poor processes will not pay off in terms of relevance, error detection capability, etc.

Model-based testing

Principle concept
Model-based testing refers to automated generation of test cases – as opposed to automated execution of test cases – using capture/playback, linear scripting, structured scripting, data-driven scripting, or process-driven
scripting. Model-based testing uses (semi-)formal models that abstract from the scripting technologies of TAA. Different test generation methods can be used to derive tests for each of the previously discussed scripting frameworks.

Advantages
Model-based testing, through abstraction, allows you to focus on the essentials when testing (in terms of the business logic, data, scenarios, configurations, etc.) under test. It also enables the generation of tests for different target systems and target technologies, so that the models used for test generation are a future-proof representation of testware that can be reused and maintained as the
technology evolves.
When requirements change, only the test model needs to be modified; a complete set of test cases is automatically generated. Test case design techniques are built into the test case generators.

Disadvantages
Modeling skills are required to effectively run a model-based testing approach. Modeling by abstracting the interfaces, data, and/or behavior of an SUT can be difficult. In addition, tools for modeling and model-based testing are not yet widely available, but they are evolving. Model-based testing approaches require adjustments in testing processes. For example, the role of the test designer must be defined. In addition, the models used for test generation are important artifacts for the quality assurance of an SUT and must also be quality assured and maintained.

Technical Considerations of the SUT

In addition, technical aspects of a SUT should also be considered when designing a TAA. Some of these are discussed below, although this is not an exhaustive list, but is intended to serve as an example of the important aspects.

Interfaces of the SUT
A SUT has internal interfaces (within the system) and external interfaces (to the system environment and its users or to exposed components). A TAA must be able to control and/or observe all interfaces of the SUT that are potentially affected by the test procedures (i.e., the interfaces must be testable). In addition, it may be necessary to log the interactions between the SUT and the TAS at various levels of detail, typically including timestamps.
A test focus (e.g., a test) is required at the beginning of the project (or continuously in agile environments) during architecture definition to verify the availability of the necessary test interfaces or test facilities required to make the SUT testable (design for testability).

SUT data
A SUT uses configuration data to control its instantiation, configuration, management, etc. It also uses user data that it processes. A SUT may also use external data from other systems to perform its tasks. Depending on the test procedures for a SUT, all of these data types must be definable, configurable, and instantiable by the TAA. The specific way in which the SUT data is handled is defined in the TAA design. Depending on the approach, the data may be handled as parameters, test data sheets, test databases, real data, etc.

SUT configurations
A SUT can be deployed in different configurations, for example, on different operating systems, on different target devices, or with different language settings. Depending on the test procedures, different SUT configurations must be considered by the TAA. The test procedures may require different test setups (in a lab) or virtual test setups (in the cloud) by the TAA in combination with a specific SUT configuration. It may also require adding simulators and/or emulators of selected SUT components for selected SUT aspects.

SUT standards and legal settings
In addition to the technical aspects of a SUT, it may be necessary for the TAA design to consider legal and/or standards requirements in order to design the TAA in a compatible manner. Examples include privacy requirements for the test data or confidentiality requirements that impact the logging and reporting capabilities of the TAA.

Tools and tool environments used to develop the SUT
When developing a SUT, various tools may be used for requirements engineering, design and modeling, coding, integration, and deployment of the SUT. The TAA, along with its own tools, should consider the tool landscape of the SUT to enable tool compatibility, traceability, and/or reuse of artifacts.

Test interfaces in the software product
It is strongly recommended that not all test interfaces be removed prior to product release. In most cases, these interfaces can be left in the SUT without causing problems with the final product. If left in place, the interfaces can be used by service and support engineers to diagnose problems as well as test maintenance releases. It is important to verify that the interfaces do not pose security risks. If necessary, developers can usually disable these test interfaces so that they cannot be used outside the development department.

Development/Quality Assurance Process Considerations

Aspects of the development and quality assurance processes of a SUT should be considered when designing a TAA. Some of these are discussed below, although this is not an exhaustive list, but is intended to serve as an example of the important aspects.

Requirements for the control of the test execution
Depending on the level of automation required by the TAA, interactive test execution, batch mode test execution, or fully automated test execution must be supported by the TAA.

Reporting requirements
Depending on the reporting requirements, including the types of reports and their structures, the TAA must be able to support fixed, parameterized, or defined test reports in various formats and layouts.

Role and access rights
Depending on security requirements, the TAA must provide a role and access rights system.

Established tool landscape
SUT project management, test management, code and test repository, defect tracking, incident management, risk analysis, etc. can all be supported by tools that make up the established tool landscape.
The TAA is also supported by a tool or set of tools that must integrate seamlessly with the other tools in the landscape. In addition, the test scripts should be stored and versioned like the SUT code so that revisions for both follow the same process.

Source: ISTQB®: CTAL Test Automation Engineer Syllabus Version 1.0