The level of performance proof required of projects can fall on both sides of an optimal “cost – benefit” line for a system. If a project spends too little effort on proving performance then the result will be problems and complaints in production. If a project has overly stringent performance criteria, the result will be high costs and long delivery delays for the project.
The key to deciding is to consider the level of risk that performance problems will pose to the business. There seem to be relatively few organisations that use a blend of testing and future projection based on a risk-driven approach. It is this approach that is going to produce the most cost effective projects, since the degree of testing and proof provided will reflect the business risk that is inherent in the system. Spend sufficient to offset the cost of risk that a system carries if the performance is insufficient in the long term. Remember that it is likely to be much more expensive to solve a performance problem a number of years after initial delivery than when the project is in full flow. However important the initial delivery may be, a project is not complete until it has been proven to be capable of being operational for the long term.
Suggestion:
Use a simple set of standard criteria for a project representing “average” performance risk for the business:
- A soak test to prove that the system will run for an extended period, say 12 hours, at average loading without performance degradation.
- A load test to demonstrate that the system can process peak demand volumes for a predefined period, such as an hour, without degrading.
- A volume test, to prove that the system can operate with production volumes at a defined point in the future – say 2 years hence.
- Within any performance tests use a small set of key business functions to test that the system performs adequately.
If the risk related to poor system performance, or failure, is especially high then these performance criteria would be made more stringent. The degree of performance proof required, however, should be kept proportional to the level of risk in the system failing to perform well. If the system was business critical, for example, it may be considered appropriate to require that the system be run using a much wider set of business functions, for 72 hours continuously, and at 5 times the expected peak volumes for the system for the next 20 years. Unsurprisingly, as the proof requirements become more stringent the cost and time needed to prove them will increase significantly.
Sarquol Ltd 