Testing process assessment and optimization

Return on Investment Calculator

Here is a checklist for testing process assessment:

Testing process assessment objectives

• Diagnose testing process weaknesses
• Recognize a need for new test tools
• Technical evaluation of test practices
• Economical evaluation of test practices

Testing process documentation

Testing process should be consistently documented in order to:

• be communicated to other teams
• be executed in a repeatable way
• be effectively improved

Data collected for assessment

• System complexity
• Complexity of development stage under test
• Testing process goals, objectives and strategies
• Development stage under test cost/budget
• Corresponding testing process cost/budget - by testing phase/test type
• Development and testing schedule
• Testing process inputs/outputs
• Testing resources:
  • Test team - structure and skills, test planning, execution and automation responsibilities
  • Hardware resources
  • Software resources and testing tools
• Total number of tests
• Testing phases and number of tests per phase
• Test techniques used
• Number of automated tests - per phase - and tools used
• Test coverage information:
  • Requirements coverage
  • Acceptance criteria coverage
  • Structural coverage
  • Data path coverage
  • Source code lines coverage
• Number of defects detected by each testing phase - according to module/severity/test type
• Number of defects found on beta sites - according to module/severity
• Number of defects reported by users - according to module/severity
• Costs of critical defects reported by users
• Potential cost of defects found by testing - had them not been detected

Testing process prerequisites

• Goals, objectives, strategies and methodology have been defined
• Testing process has been documented and reviewed
• Budget is adequate
• Testing team size, structure, skills, experience and training are adequate
• Space, hardware and software resources with technical support are sufficient
• All the necessary testing tools are accessible - with training and support
• Work plan is defined, understandable and doable
• Testing metrics are used
• Testing starts from requirements/specification phase and goes in parallel to system development
• Users are involved

Testing metrics

• Coverage metrics
• User involvement
• Relative Testing Costs (testing costs/total costs)
• Cost per defect found
• Testing budget conformity (real/planned)
• Test efficiency (number of defects found/number of tests)
• Not detected start-up failures (easily detectable major defects)
• Not detected critical defects
• Test automation ratio (automated tests/total tests)
• Testing effectiveness (number of defects found/test cost) - per module/test phase/test type
• DDP - Defect Detection Percentage (number of defects found by testing/total known defects)

Testing process optimization

Results of assessment are used for testing process improvements.

Testing metrics provide feedback about effectiveness of correcting actions.

Periodical cycles of assessment/modification maximize testing process business value.