Test automation

Return on Investment Calculator

Test automation can significantly improve testing process, but it is very important to understand its possible benefits and limitations.

In order to be successful, automatic tests ought to be designed by professional developers with specific experience in this field.

Test automation benefits

• Improves testing efficiency
• Consistent and repeatable testing process
• Supports quality metrics for test optimization
• Improved regression tests - decrease cost of change
• More tests can be run in less time - better coverage and shorter time to market
• 24/7 operation - better use of resources
• Human resources are free to perform advanced manual tests
• Objective and measurable performance and stress tests
• Execution of tests that can't be done manually
• Tests can access system parameters that are not visible to tester
• Simple reproduction of found defects
• Testing costs decrease with reuse

Test automation problems/challenges

Unrealistic expectations may kill test automation project at the start - it will not solve all the testing problems and in many cases will detect less defects than manual tests.

Automation of badly designed/not effective testing procedures will not create effective automatic tests.

Automatic tests cover only specific subset of possible defects - there is always a chance that other defects exist after automatic test passed.

Test automation tools are complex and require significant training and experience.

Maintenance of automatic tests may be expensive.

Significant effort may be necessary for analysis of automatic test results.

Test automation is a software development project and should be correspondingly organized (including testing of testing scripts and tools).

Test automation means change in business processes and should be supported by management - including necessary resource allocation in related development/manufacturing/service teams.

Test automation limitations

Automatic tests compliment and not replace manual tests. Some tests should not be automated:

• rarely used
• complex/expensive to automate
• without objective pass criteria
• requiring manual interaction
• tests of frequently changing software

Automatic tests check the same path multiple times (and in many cases after manual execution during development) - so the probability to find a defect is lower then in a manual test.

Ability of automated tests to detect defects depends on design and implementation quality of the tests and requires qualified software developers.

Automation of specific test case can decrease cost of its execution, but will not improve its ability to detect defects.

Automated tests are very sensitive to software changes.

Automated test does not have common sense - it can fail on a minor change in software behavior or pass when major error is reported in unexpected way.

Automatic tests don't handle unexpected behavior well.

Test automation priorities

Right selection of test procedures for automation is critical.

The test automation project should prove its usefulness to the organization by providing maximum impact for minimum effort from the start.

The following list represents recommended candidates for the first automation stage:

• simplest to automate tests
• frequently running tests
• critical set tests
• system wide subset of tests for fast "smoke test" execution