SessId033

Reporting Progress On Ambiguous Tasks (Like Testing)

My brother, Jonathan, and I, will explore the interesting problem of how to express an estimate of progress on an ambiguous task. We face this problem frequently in our practice of exploratory software testing.

By its nature, exploratory testing is an intuitive and multi-dimensional activity. There's no one right way to answer the questions "How much progress have you made? Are you done yet?"

Our session will include an exercise where we experience and explore this problem and evaluate the various solutions.

The core of the exercise is this question: "On a scale of 0 to 100, how completely have you tested that product?" Variations of this question have plagued testers since the time of Aristotle.

-- JamesBach

James, I like the ideas you've put forth. Irreverantly, I ask, "Why is every estimate '90%'? Why do we so misunderstand and miss on our estimates?"

Warmly, Sharon

I have many answers for that, all of which seem good to me, but any of which may be quite wrong-- just like my estimates.

We don't know how to estimate.
We don't know how to negotiate.
We don't know that we don't know, but rather believe that DisHonesty and carelessness are solely to blame.
We're dishonest and careless about estimates.
We do estimation in project environments that punish honesty and care.
Innaccurate estimates are better than accurate ones.
Estimates feed back into the control system of the thing being estimated, which changes the system, which in turn invalidates the basis of the estimate.
Our ability to perform projects requires that we fool ourselves, because we cannot accept what must be done to run a project. That may be why the pain of childbirth accrues many months after the pleasure of conception.
We don't know what to do about this state of affairs.
We don't want to do anything about it. In ambiguity, there is freedom.

On a scale of 0 to 100, I'm 85 sure I'm right about this.

-- JamesBach

My "estimate" is that you are about 20 per cent right on this, given that I generally agree with two of the ten bullet items you listed. JamesWard

I like that you supplied the percentage and also supplied your derivation of it. Offhand, 20% right sounds like a low number. Is it? JamesBach

James,

Your list is very provocative, which is 100% engaging.
When you state that an inaccurate estimate is "better" than an accurate one, I'm not sure what you mean. Useful? Better because it will be discounted? Count me as 1% confident in interpreting your meaning there.
I concur totally with 8 out of 10 for 80% confidence.
Don't know that I agree with your perspective on estimating feedback as stated. Analogy: if I'm feeling good about my work today and that makes someone I work with more optimistic about the overall project...does it invalidate my original feedback? Call it a potential 100% error in feedback due to ambiguity in the reader's understanding.

- BeckyWinant

When ignorance is bliss, knowledge is a curse. If my innaccurate estimate is more helpful to me and the project and the interests of the company than one that's accurate... Actually that item on the list is redundant with the ones immediately before and after it. -- JamesBach

After reading the subsequent comments by yourself and Becky, I probably need to raise my estimate somewhat from 20 per cent. However, I strongly believe that we do know how to estimate, and that belief calls your additional points into question. That we don't always estimate accurately (or at all on some projects) is not necessarily an indication that we don't know (as we certainly should know) how. There are ample sources that tell us how to estimate accurately, and tools are constantly getting better. In large part I think that one of our gurus (Tom De Marco ??) said it best when he stated that in no other field is there a larger discrepancy between common practice and best practices.

I also think that feedback is a good thing, and is as likely to re-validate the estimate as to invalidate it. If our estimates become a self-fulfilling prophecy is this a bad thing? Or is this just another case of work expanding to fill the time allotted to it. Of course, we've all been involved in projects where work refuses to contract to fill the time allotted, so I guess that is a one way street.

When I teach project management, I emphasize how important a tool estimating is to the project manager. However, as soon as estimation becomes a "commitment" in someone's eyes, or we are evaluated by someone based on the accuracy of our estimates, this ceases to be a tool and becomes a millstone (or is it an albatross?) around the neck of the project manager. I think that this is the real source of many of our negative feelings about estimating.

I am very much looking forward to this session at the conference and think it will very nicely complement your SessId032 with Becky. I hope they are presented back-to-back. JamesWard

So far, all I've seen on this thread are point estimates. Point estimates are dangerous, in that they subtly imply more knowledge than they justify. Giving the basis for your reasoning, yes, is one way to overcome that, but bureaucrats don't remember words, only numbers. At least I'm 50-70% sure of that. - JerryWeinberg

I think, or at least hope, that we were being facetious about the percentage estimates, as I know you are. The only way to truly judge progress on a task or project is at the binary level, it's done or it isn't. Percentage of completion estimates not only imply more knowledge than they justify, they are always overly optimistic and mask the true status, especially when a project is in trouble. JamesWard

James W., I agree with you that an estimate of the completion of a task often masks its true status, especially when the project is in trouble. I disagree, however, if you are implying that a estimated percentage range of task completion isn't useful for helping someone understand the state of a project.

I find these kind of ranges extremely helpful in understanding the state of a project. If I distrust the estimates because of all the wacky social dynamics that can happen during a project, I need to probe further with questions like, "What data supports your estimated range of completion? What are the risks to completing the task on time. How are you mitigating those risks?"

One of the perils with an estimate of completion expressed as a percentage range is that the receiver will only hear the upper value and discard the notion of a range. Another peril is that the estimator will feel that it's dangerous to his or her career to tell others what is really thought or felt. In my experience, almost all cultures reward people for providing reports that further the agenda of their leaders, rather than for accurately showing the state of a project. This political problem is something that has plagued the project world long before the time of software and even Aristotle. --SteveSmith

You're right James W., I don't use percent complete, ever. My preference for reporting things like "estimated completion date" is not a day or a range of days, but a probability distribution, expressed on a graph. It's harder to extract one number, as Steve warns, from a graph (though of course poor managers can extract numbers even from the air). I plot date vs. estimated probability of completion by that date, and am prepared to back it up with the reasoning behind it. - JerryWeinberg

Jerry, yes, a probability distribution makes even more sense to me than a range. Unfortunately, I've never had the priviledge of being on a project that someone displayed information that way. I now know how I'll be showing estimates of completion in the future. Way cool! Thanks. --SteveSmith

I'm replying to everyone in this discussion. I have been working to get clients to look at trends instead of actually numbers (or percents, or dates). What I find difficult is providing these in corporate cultures where absolute numbers are expected as the reporting mechanism to show progress and to predict completeion dates.

Since I am primarily involved in the Quality Assurance end of things, questions from management outside of QA like the following haunt me. I find that I have to be a change agent to be understood. Its an uphill if not constant battle.

"When will testing be done?"
"How much testing has been done?"
"We expect testing to be 100% complete by the time we ship; it's in the contract. Can you prove that for our customers?"

The posers of these questions don't often know what they are asking, or rather, they have a hard-coded understanding of what testing can accomplish and how it can be measured. James (Bach), I've heard you speak about this topic before with regard to Exploratory Testing. Will this session move beyond Exploratory testing to some more traditional methodologies? The same problems exist with them as well. I am looking forward to this session! AnnaAllison

Anna, Welcome to the forum. You raise some great questions, looking forward to meeting you next week. The answer I give, only somewhat facetiously, is that testing will be done when no more errors are found or on the ship date, whichever comes first. Actually, I believe testing should be completed when the system can be demonstrated to perform as stated in the requirements document, per the criteria that were established for each requirement at the time the requirements were specified. JamesWard

Updated: Friday, November 3, 2000