Software Project Convergence

People who manage software development projects become sensitive to code convergence, the changes that software goes through as its defects are found and removed to get it ready to ship. This takes time because

• testing all the code takes time
• fixing defects takes time
• removing defects requires changing code which introduces more defects. 

Software project managers need to estimate when code will converge into so they can predict ship dates. Some of the methods they use are

1. using the product to get a feel for the overall level of maturity
2. plotting number of open (found but not yet fixed) defects vs. date
3. plotting code churn vs. date

If a metric (or method) can be used to predict the ship date of a real software project then that metric should do a great job of predicting the ship date of a simple mathematical model of a software project. I am going to test the 3 methods above on a simple model of a software project. If these methods give reasonable predictions then I will test them against some more realistic models.

Software Project Model
This model starts when the code in the hypothetical project is functioning then it runs for one year of one week development cycles. In each week the development organization fixes as many bugs as it can and a separate QA organization tests the development organization's code from the previous week. The model's input parameters are

• Initial bugs: The number of bugs in the code when the model starts.
• Average number of lines of code (LOC)  change to fix a bug.
• Average number of bugs introduced per LOC changed.
• Maximum number of bugs that the development organization can fix each week.
• QA test cycle.
• The fastest and slowest bug find rates in the QA test cycle.

The model outputs

• Remaining bugs in code. This is the model's actual maturity. It cannot be observed directly so it needs to be estimated from the other output parameters.
• Open bugs (bugs found but not yet fixed)
• Code churn.
• Fraction of bugs that were introduced in the previous week.

This is a very simple model and does not take into account many of the things that happen in real-world software development. In particular it does not take into account the lags between introducing a bug, finding it and fixing it, which often have a major impact on project convergence. The spreadsheet that implements the model is at the end of this post.

The following graphs show the model outcomes for 4 sets of input parameters.

initial bugs: 1000
bugs introduced per loc:  0.007
loc to fix a bug: 100
max bugs fixed per week: 120
max bugs found per week: 200
min bugs found per week: 50
QA test cycle(weeks): 8

initial bugs: 500
bugs introduced per loc:  0.009
loc to fix a bug: 100
max bugs fixed per week: 120
max bugs found per week: 200
min bugs found per week: 50
QA test cycle(weeks): 8

initial bugs: 500
bugs introduced per loc: 0.009
loc to fix a bug: 100
max bugs fixed per week: 200
max bugs found per week: 200
min bugs found per week: 200
QA test cycle(weeks): 8

initial bugs: 500
bugs introduced per loc: 0.009
loc to fix a bug: 100
max bugs fixed per week: 100
max bugs found per week: 200
min bugs found per week: 100
QA test cycle(weeks): 8


What the Metrics Say About the Models

1. Using the product to get a feel for the overall level of maturity. This was necessary in most cases because the long term trends were different to the short term trends. The gradients in the number of open (found but not yet fixed) bugs and code churn graphs did not predict code stabilization directly.  The exception to this was when there was no pattern in the bug find rate. Bug find rates cannot be guaranteed to not have patterns, so using the product is necessary, as common sense would have suggested.
2. Plotting number of open defects vs. date. This graph trended in the same direction as the number of remaining underlying defects in code so it was a good metric. Some work is required to filter out the effects of the bug find rate.
3. Plotting code churn vs. date.  This graph trended in the same direction as the number of remaining underlying defects in code so it was a good metric. Some work is required to filter out the effects of the bug find rate.

Conclusion
The metrics work for the simple model. Schedule predictions require that code churn and number of open defects need to be corrected for bug find rate, or patterns need to be removed from the bug find rate. One way to remove bug find patterns is to measure bugs with an automated test system where tests are selected by a random number generator. See test automation for complex systems.

The Model