For software testing, test automation (including automated test execution) includes one or more of the following tasks:

• Using purpose-built software tools to control and define test requirements.
• Execution of tests
• Comparing actual results with predicted results.

A good practice is to separate the software used for testing from the system under test (SUT) itself to minimize interference. There are exceptions, such as embedded systems, where the test software must be deployed on the SUT.

Test automation is expected to help execute many test cases consistently and repeatedly on different versions of the SUT and/or environments. However, test automation is more than a mechanism for executing a test suite without human interaction. It involves a process of developing testware, including:

• Software
• Documentation
• Test cases
• Test environments
• Test data

Testware is required for the following test activities:

• Implementation of automated test cases
• Monitoring and controlling the execution of automated tests
• Interpreting, reporting, and logging automated test results.

Test automation has several approaches to interacting with an SUT:

• Testing through the public interfaces to classes, modules, or libraries of the SUT (API testing).
• Testing through the SUT’s user interface (e.g., GUI tests or CLI tests) Testing through a service or protocol.

Test automation goals include:

• Improve testing efficiency
• Provide broader functional coverage
• Reduce overall testing costs
• Performing tests that manual testers cannot perform
• Reducing test execution time
• Increasing the frequency of testing/decreasing the time required for test cycles

Benefits of test automation include:

• More tests can be run per build
• Ability to create tests that are not manually feasible (real-time, remote, parallel testing)
• Tests can be more complex
• Tests run faster
• Tests are less prone to operator error
• More effective and efficient use of testing resources
• Faster feedback on software quality
• Improved system reliability (e.g., repeatability, consistency)
• Improved consistency of tests

The disadvantages of test automation include:

• Additional costs are incurred
• Initial investment to set up TAS
• Requires additional technologies
• Team must have development and automation skills
• Ongoing TAS maintenance required
• May distract from testing objectives, e.g., by focusing on automating test cases at the expense of test execution
• Tests may become more complex
• Additional defects may result from automation

Limitations of test automation include:

• Not all manual tests can be automated
• Automation can only check machine-interpretable results
• Automation can only check actual results that can be verified by an automated test oracle
• Not a substitute for exploratory testing

Source: ISTQB®: CTAL Test Automation Engineer Syllabus Version 1.0