Evaluation and Selection of Test Automation Tools

The primary responsibility for tool selection and evaluation lies with the Test Automation Manager (TAM). However, the TAE is involved in providing information to the TAM and performs many of the evaluation and selection activities.

The TAE is involved in the overall tool evaluation and selection process, but has particular influence over the following activities:

  • Assess organizational maturity and identify opportunities for test tool support.
  • Evaluating appropriate targets for test tool support.
  • Identifying and gathering information on potentially appropriate tools.
  • Analysis of tool information with respect to objectives and project constraints.
  • Estimating cost-benefit based on a solid business case.
  • Making a recommendation for the appropriate tool.
  • Identifying the compatibility of the tool with the SUT components.

Functional test automation tools often cannot meet all expectations or situations that may arise in an automation project. Below are some examples of these types of problems (but the list is by no means complete):

Finding Examples Possible Solutions
  • The tool’s interface does not work with other tools that are already in place
  • The test management tool has been updated and the connecting interface has changed
  • The information from pre–sales support was wrong and not all data can be transferred to the reporting tool
  • Pay attention to the release notes before any updates, and for big migrations test before migrating to production
  • Try to gain an onsite demonstration of the tool that uses the real SUT
  • Seek support from the vendor and/or user community forums
  • Some SUT dependencies are changed to ones not supported by the test tool
  • The development department has updated to the newest version of Java
  • Synchronize upgrades for development/test environment and the test automation tool
  • Object on GUI could not be captured
  • The object is visible but the test automation tool cannot interact with it
  • Try to use only well–known technologies or objects in development
  • Do a pilot project before buying a test automation tool
  • Have developers define standards for objects
  • Tool looks very complicated
  • The tool has a huge feature set but only part of that will be used
  • Try to find a way to limit the feature set by removing unwanted features from the tool bar
  • Select a license to meet your needs
  • Try to find alternative tools that are more focused on the required functionality
  • Conflict with other systems
  • After installation of other software the test automation tool will not work anymore or vice versa
  • Read the release notes or technical requirements before installing
  • Get confirmation from the supplier that there will be no impact to other tools
  • Question user community forums
  • Impact on the SUT
  • During/after use of the test automation tool the SUT is reacting differently (e.g., longer response time)
  • Use a tool that will not need to change the SUT (e.g., installation of libraries, etc.)
  • Access to code
  • The test automation tool will change parts of the source code
  • Use a tool that will not need to change the source code (e.g., installation of libraries, etc.)
  • Limited resources (mainly in embedded environments)
  • The test environment has limited free resources or runs out of resources (e.g., memory)
  • Read release notes and discuss the environment with the tool provider to get confirmation that this will not lead to problems.
  • Question user community forums.
  • Updates
  • Update will not migrate all data or corrupts existing automated test scripts, data or configurations
  • Upgrade needs a different (better) environment
  • Test upgrade on the test environment and get confirmation from the provider that migration will work
  • Read update prerequisites and decide if the update is worth the effort
  • Seek support from the user community forums
  • Security
  • Test automation tool requires information that is not available to the test automation engineer
  • Test automation engineer needs to be granted access
  • Incompatibility between different environments and platforms
  • Test automation does not work on all environments/platforms
  • Implement automated tests to maximize tool independence thereby minimizing the cost of using multiple tools.

Source: ISTQB®: CTAL Test Automation Engineer Syllabus Version 1.0

Was this article helpful?

Yes No

Related Articles