The Testing TCO Approach To Driving Improvements

The concept of Testing Total Cost of Ownership (TTCO) can be defined as the sum of the development, testing and on-going maintenance costs, plus the costs of any enhancements or downtime caused by failures in the production environment. The approach was borne from experience that testers who talk to organisations about improvements in efficiency measures don’t receive as much reception as testers who talk about monetary savings.

TTCO is based on the assumption that IT is a fundamental component of all business organisations and is essential to the survival and wellbeing its host organisation. Its aim is to demonstrate that the TTCO of the organisation’s IT systems is minimised.

Given the importance, significant expenditure and investment in an organisation’s IT, it should be viewed and treated in the same way as any other asset. IT assets – especially non-tangible software – tend to be retained by the organisation for a significant number of years, albeit undergoing enhancements along the way and subject to regular routine maintenance.

By using TTCO, the measurement of improvements in testing effectiveness and testing efficiency changes from criteria – such as the number of defects identified in testing or the number of test scripts executed per man day (useful as these measures may be) – to a bottom-line measure.

For example, imagine a company that spends £100M per annum on IT, of which:

  • £40M is spent on development
  • £8M is spent on testing
  • £3M is spent on application maintenance and support (i.e. fixing defects found in the production environment and developing temporary work-arounds)
  • £1M is the IT overheads (e.g. heating, lighting, accommodation etc.) attributable to testing

In addition:

  • £1M is attributable to the costs of the Support Centre attributable to customer and outlet calls relating to IT issues
  • £5M is the assessed cost of downtime in the company’s outlets due to unplanned IT outages and issues

The TTCO would be calculated as: £40 + £8M + £3M + £1M + £1M + £5M = £58M. Mapped to the previous list, a target TTCO could then be estimated as follows:

  • Assume design and build inefficiencies of 15 per cent can be removed by testing earlier in the lifecycle
  • Assume test inefficiencies of 10 per cent can be removed
  • Assume that 90 per cent of defects in the live environment can be eradicated by improving testing effectiveness
  • Testing overheads are fixed
  • Assume 90 per cent of the costs of the Support Centre attributable to IT issues can be removed through increased testing effectiveness
  • Assume 90 per cent of the assessed cost of downtime in the stores due to unplanned IT outages and issues can be removed through increased test effectiveness

The target TTCO in this instance would be calculated as:

  • (£40M x 0.85) + (£8M x 0.9) + (£3M X 0.1) + (£1M) + (£1M x 0.1) + (£5M x 0.1)
  • £34M + £7.2M + £0.3M + £1M +£0.1M + £0.5M = £43.1M

The shift in focus from ‘investing in test process improvement to achieve a TMMi level 3 rating’ to ‘investing in test process improvement to bring the TTCO down by £XM’, provides a clearer basis for a business case to invest in test process improvements. It also speaks to a more business decision makers, as the focus on the bottom line is a more compelling proposition.

Investment linked to setting the IT organisation and testing services – in particular annual targets for reducing the TTCO – helps an organisation to drive improvements in test effectiveness and efficiency. In fact, once the testing services provider starts to implement the process improvements based on reaching a TTCO target, the effects will force remediation of weaknesses and inefficiencies throughout the IT organisation.

An IT organisation, and its testing services provider, will always seek to improve testing processes. They will address issues arising from poor quality requirements and design documentation and seek to change organisational and cultural norms that detract from a focus on quality. In addition, they will look to ensure new IT systems are easily and cost-effectively maintainable and upgradeable and that the IT estate is managed effectively.

In conclusion, TTCO is by no means a panacea for driving the required behaviours to achieve efficient and effective IT organisations. It does, however, help to focus an organisation’s attention on driving tangible business benefits. TTCO provides an objective measure of quality, and the standard to aspire to by focusing attention on the problem and providing the impetus to drive improvement activities. Finally, TTCO also provides an organisation with an on-going measure of efficiency and effectiveness and contributes significant benefits to the organisation.

John Cox is a Principal Consultant at Xceed Group. With more than 25 years' experience as a Test Consultant, John has worked for the likes of Price Waterhouse Cooper, The National Programme for IT in the Health Service and Cognizant. He is well versed in advising on testing strategy for critical business testing - a role he has performed with distinction for some of the biggest names in industry and some of the biggest programmes, such as the LTSB/HBOS Integration. An accomplished operator, John is an excellent communicator and is one of those people who not only 'gets it' but can translate so that everyone else 'gets it'.

  • J. Unsworth

    Spot on. BCS has recently stated that 3% more spent on testing upfront can reduce faults by as much as 70%