24 June 2005
Meet the New Tester Part 2
David Evans and Nick Pointon of Cresta
To reduce the high overhead costs of defect management in the system testing phase we must look to methods that allow a much greater proportion of functional errors to be detected and corrected during the development phase. The authors argue that this goal can be achieved by introducing a policy of thorough testing at the development phase, with the help of a new type of tester.
The new tester will be an active and co-located member of the development team. By working alongside developers and helping the team to test code as it is written, the tester provides immediate feedback on quality and catches defects at a stage when they can be fixed quickly and cheaply.
Skills and Qualities of the New Tester
The role of the embedded tester is no small one. A Test Manager will do well to choose carefully the right candidate to be successful within a development team. For starters the tester must be personable, communicative and able to state their case dispassionately yet confidently to possibly defensive developers.
He/she must also be technically confident, and ideally should be able to read and understand program code, even if not required to write it. They must be steadfast and not bow to pressure from the development manager to ‘let through’ sub-standard code.
An active interest in the tools of the trade is a must; developers will be willing to use tools to support quality initiatives if they can be convinced of their worth and effectiveness. The more au fait the tester is with these tools, the more acceptance they are likely to win from the developers.
Automation skills are a definite bonus: the ability to code unit tests, write batch files or debug Perl scripts will encourage creative solutions to embedding quality processes, as well as earning hefty credibility with the developers.
The Management Structure
So where does our new tester sit within the management structure of the project? The key point to remember is that these people are dedicated testers. They may sit within the development team and they may be executing something between Unit Tests and System Tests, but they must be answerable to the Test Manager. In this way the tester has a responsibility to uphold the testing ethos and to ensure that these aims are not subverted by any conflicting demands of the development team, such as meeting a delivery deadline.
As with all other phases of testing, development testing must adhere to a set of project standards. It is likely that these standards will differ from those for System or User Acceptance Testing (a sort of ‘System Test Lite’), but the basic principles that apply to all testing are still relevant: the testing must be visible, accountable and documented in sufficient detail to provide confidence in the testing carried out, and if necessary to allow handover to another experienced tester.
But this is not some special pre-System Test team, sitting apart from the developers; these are embedded testers. The testers must be co-located with the rest of the development team and get involved in the day-to-day activities of the developers, attending the developers’ planning, design and progress meetings. Even more valuable, by sitting with the developers the new testers will hear all the informal information that passes around the team, much of which will contain vital information about the system behaviour and pointers about where to direct testing efforts.
Although answerable to the Test Manager, the new tester will take direction on a day-to-day basis from the development team leader. Only the development team leader knows which packages are actively being worked on and which are ready to be tested, and can steer the work of the tester appropriately (Fig. 1)
The delivery out of the development team is now dependent on the tester successfully passing her tests. The tests to be executed are directed by the test strategy or methodology, which is the responsibility of the Test Manager. This means that the development team leader appears to be dependent on something that is outside his control. For example he cannot affect the tests to ensure that they meet his delivery deadline. On occasions this can be frustrating for development team leaders and often they will exert undue influence to get tests changed or removed. For this reason the new tester needs to be a senior tester with a strong personality, who is comfortable defending her testing principles; and of course she needs to be backed up and supported by the Test Manager.
Of course, it is not really true to say that the development team leader is responsible for something that is outside his control and he shouldn’t consider the presence of an embedded tester to be at variance with his objectives. His goal is to delivery high quality software in a timely way, and the new tester is helping him to do this by providing accurate and unbiased feedback. The development team leader should welcome the new tester as a valuable resource in the team.
Peripheral Advantages of the ‘New Tester’
We talked initially about the cost savings that can be made by identifying defects earlier in the development life cycle. Having embedded testers is a key means of realising these cost savings, however there are a number of other benefits that will be enjoyed.
In some organisations there is an apparent conflict of interests between developers and testers. A divide, both physically and emotionally, can spring up between the two parties and a degree of ‘finger pointing’ on both sides can foster resentment between the two parties. This effect can be particularly pronounced where the development is outsourced to a third party, who is keen to maintain its independence from the rest of the organisation.
Embedding testers in the development team breaks down barriers between the two teams. The natural interactions that happen as a result of sitting in amongst the developers will foster a spirit of understanding and create personal relationships between the two teams’ members. This, in turn, promotes co-operation between the teams and can help the project work as a coherent unit.
The embedded testers also serve to raise the profile of testing within development. By making each development team responsible for the quality of their software, where that quality is independently verified, means that each team has to ensure that the software it produces functions correctly. It prevents a ‘that’ll do’ mentality as developers cannot throw the code ‘over the wall’ until it has correctly passed its tests. In essence, it keeps the developers honest.
The new testers’ tests are, by their very nature, the exit criteria from development into System Testing. In many organisations, exit criteria for each phase are elaborately set out during the planning phase only for them to be ignored when the project gets underway. With the independent stamp of approval required from the embedded testers to exit development, the exit criteria are a visible and integral part of the process. The project management can choose to disregard them, but it cannot pay them lip service and just forget about them.
By embedding testers within the development team, the develop-test-fix cycle becomes a much tighter loop. The testers are able to provide very swift feedback to the developers, and the developers can provide fixes very quickly, without the need for extensive change control procedures. Normally only unit tests enjoy this level of near instantaneous response, but unit tests are not produced by professional testers and are not designed with the same attention to detail and the business-oriented mindset that professional testers’ tests are.
The result is cleaner and more robust builds, which give System Test a much greater chance of success. Given that the System Test phase is the one that is most commonly cut short by fixed delivery deadlines, any improvement in quality going in to System Test is will reduce the total cost of testing and result in fewer production problems.
Defect Migration
Defects are introduced into a system because a mistake is made at a certain phase of the system’s design or development. There are various types of mistake, and phases of development in which they are made. Examples of mistakes that introduce defects would include:
- an ambiguously specified requirement
- a flawed technical design decision
- an incorrect coding algorithm
- a fix for one defect found in testing that caused another elsewhere in the system
When a defect enters the system in one phase but is not removed until a later phase, if at all, it is known as a ‘phase escapement.’
While it is better to find and remove defects in the Testing phase, rather than letting them be discovered in production by users, it is best to remove defects in the same phase in which they are introduced. It is therefore best to apply quality control to the products of all phases, not just developed code. This is because the cost of defect removal increases dramatically for each phase that the defect remains.
The quality of the products from one phase directly affects the quality of those in the ‘downstream’ phases. The impact increases as a defect escapes into later phases, because the size of the products of each phase increases dramatically. For example, a system may have hundreds of documented requirements, which leads to thousands of items in a technical specification, which leads to hundreds of thousands of lines of developed code, which leads to millions of lines of installed code.
If a mistake is made that leads to a defect in a single requirement, this can typically have a 10-fold impact on the design and a 100-fold impact on the code. If such a defect is not discovered until the Test phase, it could therefore require changes to hundreds of lines of code. If, however, the same defect were found and corrected during the Requirements Specification phase, phase escapement would be prevented . As a result, the cost of correcting the requirement will be negligible, because it is before design and coding has been affected.
