Metrics: Lunches, Looks, and Loops
© Don Gray, 2004
Michelle works as a software engineer. Recently we discussed the “utility” concept (Measuring and Managing Performance in Organizations, Austin)1 and she asked, “Can’t we just measure for the sake of learning?”
Measuring for learning is a wonderful idea. Project managers, developers, QA staff, and their clients and customers can all benefit from knowing what is happening (no matter how painful it might seem at first). However, I’ve never seen a metrics program that was designed “learning’s sake”. Even if there was such a program, the very act of measuring changes the system being measured. Here are several reasons why.
The Lunch Law
“There is no such thing as a free lunch.” Robert Kiyosaki
The Lunch Law masquerades in several guises. Mom taught me “You can’t get something for nothing.” A physicist calls this law the conservation of energy. Both my mother and the physicist are pointing out that to gain something (knowledge) you have to give up something. What are we giving up in software development in order to measure?
Time
The first thing we give up is time. Taking measurements takes time. Collecting the measurements takes time. Processing the measurements takes time. And it takes time to report the measurements. In the “internet time”, lean & mean, bare bones, full speed ahead corporate mentality, from where (or more correctly whom) does this time come?
Energy
Energy that could be used to develop is now being used to measure. Knowledge workers rarely have a singular job, with a single attribute. More energy will put on the item being measured, moving energy away from the other job mix components2. Software developers moving too much energy to the measured attribute change the customer’s total benefit. A measurement program becomes dysfunctional when enough energy shifts and the total benefit is less than originally existed.3
Focus
The greatest forfeiture that occurs is focus. Once a measurement program starts, focus shifts from development, to development and measurement and wondering what does this new measurement system mean. Will this affect my performance review? How do my measurements compare with my teammates? This dilutes the focus that had been solely on development. The informal communication network will focus on “what does this mean”.
The Look Law
“Believe only half of what you see and nothing that you hear.” - HYPERLINK "http://www.brainyquote.com/quotes/authors/n/normancous130503.html" Norman Cousins
The Look Law cautions against casually accepting measurements as “the truth”. Consider the following project status report: “The project is 50% complete.”
50% complete can mean the following:
We’ve spent 50% of the money.
Half of the project time line has elapsed.
50 % of the functionality has been developed and tested.
And which 50% is complete? It could be the easiest half or the hardest half; the most valuable half or the least valuable half, and so on.
This demonstrates the need to ask the following questions:
Are they sending what they’re seeing?
Are you receiving what they’re sending?
Is what you’re receiving a surrogate value from which you must infer the information you really want?
Pertinent Positive
As you look at your metrics program’s data, what do you expect to see? Are things going smoothly? The next iteration of the company flagship software product should (hopefully) display fairly consistent metrics with previous iterations.
Pertinent Negative
Should they be going smoothly? Occasionally “all thing going awry” should be expected. New product development using new technology should have different metrics than an incremental release of an existing product that used existing technology.
Pertinent Possible
What aren’t you seeing that you think you should see? On a recent project my client mentioned the software wasn’t working because the value he was watching wasn’t changing. I pointed out that the difference between the systems was two orders of magnitude, and that one system SHOULD take significantly longer to show a change.
Pertinent Plausible
Finally, do the measurements agree with other information about the project? “Hard numbers” tend to make us feel like we know the project status. The meaning of the “hard numbers” directly relates to our diligence in applying the Look Law. Do the numbers agree with more important (but less concrete) information you collect during walk-abouts and status meetings?
The Looping Law
“Round and round she goes. Where she stops, nobody knows” – A Carny Cry.
The Looping Law points out features of measurement programs:
Measurements from a single development cycle can help.
Measurements from multiple cycles reveal patterns.
Patterns reveal structure that can be modified to improve software development.
Software impinges on our reality. From microwave ovens to antilock brakes to magnetic resonance imaging to software on our computers, software (and its quality) constantly impacts our quality of living. Better software benefits society.
Measuring the analogous values over multiple projects can reveal behavior over time characteristics. One ongoing project I’m involved with has measurement data for four releases. We compare current measurements to the previous cycles to track our status.
You need to pay the Lunch Law costs, and practice the Look Law while paying. The benefit of doing so leads us to the
The Last Law
“Without visibility, control is not possible.” The Cybernetic Rule4
Measurement programs should answer the three big questions:
How good it is? (Bug counts and severity)
Is it done yet? (Ship and functional criteria)
How did we do? (Time and cost)
We can better estimate future projects of similar scope and difficulty if we have measurement data from previous projects. Historical data provides the base line for observing the effects of software development process change. Managers monitor measurements evaluating the change’s nature.
Software metrics is a double-edged sword. The societal and financial reward can be enormous. The overall cost in terms of developer satisfaction and output can be devastating. If you decide to implement a measurement / metric program, carefully consider the following.
Communicate Early and Often
Clearly define the measurement program in advance. Know and share:
What will be measured.
Why it is being measured.
How it is going to be measured.
Who the measurement will affect.
You need to anticipate the Lunch Law will bias the results if the measurement affects performance reviews. People systems are self-directing and goal seeking. Tell me what you want, and I’ll do my best to deliver. Even measuring for measuring’s sake skews effort and could become dysfunctional.
Collecting Data
Consider ways to collect data without burdening the developers. Mine the data from existing documents and reports. Automate the collection, rollup and reporting processes as much as possible. This reduces the Lunch Law’s impact. Consider getting information from:
Status reports
Bug Counts
Client input
Measurement data has many attributes. These include:
Value. Some number, hopefully between the expected minimum and maximum.
Quality. How much do you trust the value? Use the Look Law.
Timing. Values should be “here and now”, not “then and there”. The more current the value, the more useful it can be.
Relevance. What does knowing this value accomplish for me? If nothing, the Lunch Law suggests quit collecting it.
Recognizing the difference between the words we use when talking about measuring software helps keep information straight.5
Measurement is a number, created form observation that allows ranking and comparison. 5 granks is both different from and bigger than 3 granks.
Metric is a ratio of a measure to some other number, placing the measure in a context. “granks” is a measure. “granks per module” is a metric.
Indicator is the result of comparing one metric to another. “granks per module this build” versus “granks per module last build”.
Consider Value
Project managers, developers, QA staff, and the clients and customers benefit from knowing the answers to the three big questions. Managers need to estimate the quantitative and qualitative costs due to the Lunch and Look laws. Determining how the new information benefits the company and the clients provides the go/no go decision for implementing a measurement program.
1 Measuring and Managing Performance in Organizations, Robert Austin, ©1996 Dorset House Publishing
2 DeMarco’s Law as cited in Quality Software Management, Volume 2, First Order Measurement © Gerald M. Weinberg 1993, Dorset House Publishing pg 29
3 Austin, op. cit. pg 15
4 Weinberg, op. cit. pg 48
5 adapted from Jim Bullock’s post at HYPERLINK "/wiki/scribble.cgi?read=MetricVsMeasurement" /wiki/scribble.cgi?read=MetricVsMeasurement
© Don Gray 2004
Email this article to a friend.
If you enjoyed this article and want to learn more about the conference,
sign up for our email newsletter. We'll let you
know about new developments, conference discounts, and other news.
And we'll never, ever give away or sell your name or email address.
