IterativePlanning

I've been using iterative planning since the mid-80's. I don't have a crystal ball, and I can't predict the future. I use a running 3-6 week detailed plan, with major and some minor milestones past that. I've been very successful.

Have you use iterative planning for projects? What did you find when you did? -- JohannaRothman 2003.04.01

I am not sure if this fits under iterative planning or not, but one technique that has begun to stick here is the meeting from SCRUM. On development projects, we have 30 minute meetings every other day, starting after we have figured out what the pieces of a solution are. We prime the process by laying out what actions we think we need to do for each piece on flip chart paper. Those mini-plans are hung up on the walls of our work room. One person is responsible for each piece of work and participates in the scrum. We follow the rules of SCRUM, more or less. Three questions are asked for each piece of work:

what progress was made? (usually indicated by putting an asterisk by a task to signify that work has begun there or a line through the task to indicate that it is done)
What is planned?
What is in your way?

No (or very little) content discussions are allowed. These have worked very well - people feel that they can see what is going on - even sometimes having an impact on what is being done. And, we connect up progress meetings with what is really going on.

BobKing 2003.04.03

Putting plans (and tracking) on the wall is using an InformationRadiator. I hear it's very effective in some situations.

KeithRay 2003.04.03

Having a short, regular interval when planning will happen again balances between two failure modes:

Constant redirection, when you discover something new. It's knowledge work. We're always supposed to be discovering something new. (Duh!) So we've got to get work done in the face of new information.
Getting stuck with the old, grand plan, in the face of new information. It's knowledge work. We're always supposed to be discovering something new. (Duh!) So we've got to be able to adjust our plans and commitments in the face of new information.

I call the constant redirection problem "Brownian Motion." I'll explain that if anyone hasn't heard it already (Laurent asked about "forcing functions" a couple months ago. Got a 7 page paper out of the deal, which I may end up publishing somewhere. He hasn't yet told me it sucked.) Thing is, with constant redirection, you don't get anywhere. Personally, I really like small, fixed delivery iterations as a way to work with the uncertainty of planning.

We'll do what it takes to hit this small increment as promised.
Anything after that is subject to renegotiation as we learn more.

This iteration is a commitment, while everything after that is a notion. Similar to the stack of build & delivery processes I suggested in ShortIterations there is a stack of planning & replanning processes as Johanna has suggested. " . . . a running 3-6 week detailed plan, with major and some minor milestones past that." I think some of these processes involve commitments to deliver at larger granularities than one software iteration. Planning for more staff, or involving external teams often has to have a time horizons of bigger than one week, or several weeks, and it also has to be a commitment sometimes. "We're going to ship on this date - 9 weeks from now - because the Navy won't pay us $ 55 million unless we hit this date." So I usually maintain a stack of plans, with different types of things in them, and with different (usually bigger) time horizons and scope of commitments.

Tuning your control system to the dynamics of the process being controlled is a theft from the domain of process control, and more generally the study of dynamic linear systems, just like forcing functions.

-- JimBullock, 2003.04.03

Trolling off on Bookshelved, I was reminded that there's a useful distinction between "iterative" vs. "incremental", essentially:

Incremental means work done a chunk at a time.
Iterative means the same work refined a bit at a time.

The planning processes described so far are both. Interesting to me, "iterative" and "incremental" have a scoping relationship. Looking at a project planned and managed as Johanna describes, the creation of the detailed plan is "incremental". One could also view this as "iteratively" creating the comprehensive detailed plan - it's refined a bit at a time. Switching frames is a useful check on what we're committed to or not, I think, which is clearly my big point of view here. A lot of stuff gets clearer in a hurry from the "commitment vs. estimate" frame.

-- JimBullock, 2003.04.03

When I teach project management, I teach about lifecycles. I use four major categories:

Linear: Waterfall and waterfall with feedback
Iterative: Spiral, where the whole product is up for grabs each time
Incremental: Where you add to the product in pieces
Agile: cycles (iterations) of chunks (increments): Add to the product in pieces, where each iteration you can deal with the whole product.
Random: code and fix

It turns out that a lot of PMs haven't thought about lifecycles. They're comfortable with one kind, and not another kind, and they have trouble adapting their work to different lifecycles. Yikes. And that means they don't take advantage of their lifecycle's time to replan. Double-yikes. JohannaRothman 2004.04.04

Interesting. How did you come up with that taxinomy? I ask in part because iterative vs. incremental sparks some contention sometimes, and of course Agile"(tm) vs. everything else is trolling around looking for controversey any more. I have seen far too many practitioners who adhere to the one true way, usually the one true way that they learned first, or even the one true way that they did things unconsciously and by accident, the first time. Seems independent of job title; includes programmers, "architects", PM's, department managers, etc.

I've had occasional success talking about life cycles by separating the sequence of activities from the artifacts produced and their relationships. They really are two sets of concerns. How does a requirement relate to a design, to some code, to a deployed system, statically? What's goodness in a requirement? Then how you get there and what you do in what order are another set of questions.

But that's just me. I'm methodology agnostic, technology agnostic, and even indifferent to big-endian vs. little-endian implementation of numbers. "I am without conviction, if that's what you mean." said the marvelously corrupt captain in the movie Casablanca. Of course he wasn't. He was just quite focused on some other things. In our case, I think the focus ought to be on shipping. The rest is a means to an end, or noise. -- JimBullock, 2003.04.04

I used Chapter 7 of Steve McConnell's "Rapid Development" and the iteration/incremental thing just came to me. Maybe Steve actually uses those words, but I couldn't find them on my brief look just now.

When I describe them, I talk about iterating through the product on each cycle (spiral), or incrementally building a product (or creating chunks of small projects that come together at a release.

My first project as a PM was a spiral lifecycle. My first large project was a staged delivery (chunking) lifecycle. I was unconsciously competent, because I had no idea there were different lifecycles or that I was planning differently.

If you'd asked me, I would have vehemently said I planned everything the same way: start with what you know and only plan for what you know; re-evaluate; and keep planning. The planning process works no matter what the lifecycle is. -- JohannaRothman 2004.04.05

start with what you know and only plan for what you know; re-evaluate; and keep planning.

That's good. I'll be borrowing that. I've been having some converstaions with plan a lot / plan a little folks (I do this sometimes - engage with two or more "opposing" points of view. It appeals to my syncretist tendencies.) My concern has been with prescribing how much planning and how much detail. That doesn't seem to be consistent for all cases. I had tentative conclusions:

You get different amounts of knowledge in different circumstances, so the degree of detail that you can plan is different.
You have different costs & payoffs for repeatability and adaptability in different circumstances, so the balance point between planning and experimenting is also different.
People often have experience in a narrow range of repeatability, or available knowledge, then generalize this to: "All systems are plannable with this precision."
The emotion around planning "too much" or "too little" (and there's a lot) really comes from someone being forced into a situation where they were held accountable for plans more precise than the information warranted, or suffered consequences from not planning with information easily available.

It's all about tuning to how much you know, can know, and should know (both possible and cost). -- JimBullock, 2003.04.0

In addition to short duration iterations, to me the agile approach also includes delivering working something (software, etc.). So agile iterative development is both timeboxed AND feature-driven. --JimHighsmith

I like the "working something" part. But I think it's more than that. I think it's also "something to work with" for the customer. That means it has to be "working," but also visible and "workable with" by the people you want feedback from. Getting something working is not just an exercise for the programmers. - JerryWeinberg 2003.04.08

I'd like to understand how iterations that are "time-boxed", "feature driven" and "something to work with" are always possible under three kinds of challenges.

Technical deficit for an "existing" system. Examples from my direct experience:
- Brokerage trading "system" that nobody has all the parts or could even name them. 100s of parts, multiple languages, COTS stuff & more.
- Billing & customer care system for cellular industry, in "development" for 3 years, in "deployment project" for 6 months - nothing "deployed".
- Manufacturing tooling model used to maintain glass molds. This one was pure screen-driven software in one language. "Nobody" knew how it worked.
No development environment. Starting from a blank room, it takes a while to get an environment, infrastructue and team up and going even when you decide to just use whatever you used the last place you worked.
Target infrastructure is hard. For an analytical system doing year over year comparisons of auto sales by demographic and brand, it takes a while to get the piece parts in place to make it go.

Is there a scope thing to the prescription for increments that are: "time boxed", "feature driven" and "something to work with?" Like maybe you've got to be able to deliver anything at all, before how you decide on the chunks matters? Or am I not getting it? -- JimBullock, 2003.04.08

Jim, you have put your finger on some prerequisites for successful time-boxed iterative development. Without sufficient infrastructure, you can't "just do it!" The Agile premise is that there are important advances in the last 20-40 years, and Agile builds upon them as prerequisites to effectiveness.

Some other show stoppers:

Grotesque programming language. (Assembler, RPG, FORTRAN)
Home-grown (or missing!) SCM.
No shared access environment.
Code via keypunch or paper tape (Refactor this!!!)
Memory constrained development environment.
Time reporting to the 5-minute granularity, with categories dictated by phase by the accounting folks.

Being Agile with one hand tied behind your back is probably PrettyAdventuresomeProgramming!

--BobLee 2003.04.09

OK. I'm feeling better now. Most of the time I've been in situations where the infrastructural stuff wasn't present, yet folks were off doing coding and wondering why it wasn't working. So it's been about getting enough infrastructure in place to let the work happen.

I'll take issue with some of your showstoppers though. You can, in fact, do rapid iterations and refactoring in both assembler and C. And depending on your target environment, it may make economic sense to use those languages, say 100,000s of units of very cost sensitive consumer products. "Home Grown" SCM makes little sense these days, but "back in the day" rolling your own was better than not having. Etc.

I do get a little jarred when I hear difinitive statements out of self -identified "agile" folks. "Ya can't use assembler." -- JimBullock, 2002.04.09

You can, in fact, do rapid iterations and refactoring in both assembler and C

I meant grotesque languages like Assembler-F or RPG or (old) FORTRAN:

6- or 3- or 8-character names, all upper case.
Little support for structured macro coding (or merely unaware of it!)
Globals and dirty side-effects.

All of these make most refactoring MUCH more difficult, so short iterations start growing longer. I've done higher-level assembler with a macro library and opsys interfaces that acted much more like C than assembler, and it did raise the productivity bar. Using Assembler-H allowed us long names with underscores. Almost tolerable!

Recognize when the mechanical technical debt is too deep, and counter it.

--BobLee 2003.04.10

I always use an Iteration 0. It is defined as not generating anything useful for the customer, but can be used for training, initial requirements gathering, putting infrastructure in place, etc. Some project can get away with a week in Iteration 0, others need more time. By defining it as not generating anything useful for a customer (ie. no visibile features available), encourages the team to keep I0 as short as possible. --JimHighsmith 2003.04.11

I don't worry about "slips" until after the 2nd iteration, particularly if the team hasn't worked together before. It takes at least 2 iterations to work the kinks out of the development and build environment, adjust some bad habits, and otherwise get the team into a groove. Until then, any "velocity" calculations are wild stabs at best. (They're still worth charting, if only to demonstrate the silliness of initial guesses.)

One difficulty of this approach is that demands for "when is this going to be DONE!?!" don't abate. --DaveSmith 2003.04.10

I'm feeling much better now. Thanks to BobLee, JimHighsmith, JohannaRothman, and DaveSmith. Whew.

Some project can get away with a week in Iteration 0, others need more time.

OK, better. I usually force myself (and anybody else, if I'm in charge) to deliver something on cycles of a week or so in Iteration 0 - however long that is. This seems to herd the cats a bit better, when they've been stuck in Brownian motion for a while. My experience fits with DaveSmith's - the third cycle is the first one with anything like stable behavior. If they've been in the non-delivery zone for a while, it takes at least three repetitions for folks to decide that you are serious, aside from flusing out the technical issues.

One difficulty of this approach is that demands for "when is this going to be DONE!?!" don't abate.

Sometimes a defined investment offer works: "I don't know when it will be done. Right now, I don't even know if it can be done in a way that will work for you. Are you willing to invest a week, after which I ought to be able to tell you what it will take to start delivering functionality at all?" That can start a real interesting conversation. Be rested for that one. -- JimBullock, 2003.04.10

I work with a number of product companies. The length of iteration 0 also depends on how much work has been done in a product/project feasibility or discovery phase. Many of the agile approaches, XP for sure, assume (implicitly at least) that we have a valid, approved project and we can just get started--which isn't always so. But we can also get hung up in endless feasibility, discovery work. Rather than endless paper shuffling, I also recommend (when needed) iterations -1 and -2. These are throw-away prototypes, etc. that are used to access feasibility. I worked with a company a couple years ago that had spent 12 months putting plans on paper for a new product. We did two short iterations in about 2 months that gave them more info than at least 10 of those 12 months did. --JimHighsmith 2003.04.11

Jim, your watch is a day fast. Nope, a year behind--new battery :).

I can think of several project where just turning the crank one or two times would have surfaced important issues that instead had to wait for 6 or more months to be uncovered. Often it's simply a matter of smoke-testing new technology.

Looking back on a particular project, I'm still puzzled over what impact using an Agile methodology might have had. The stakeholders (executive management) wanted to explore a new business direction. They had a vision, and wanted an answer to "What's it going to cost us to realize this vision?" Using a traditional Big Design Up Front (BDUF) approach, they eventually got an answer: "More than you want to spend," but only after we'd churned through a lot of analysis and design for the whole system.

Had we instead gone Agile, starting with a handful of stories and fixed iterations, I'm not sure they would have gotten that answer much sooner, though the stakeholders certainly would have had an early chance to get feedback on their original vision. The problem was one of complexity, and a foreign problem domain that we had little in-house experience with when we started. I think it would have taken us several, possibly many, iterations to even be at the point of giving ballpark estimates for all of the stories. The difference would have been less throwaway design, in exchange for something that the stakeholders could see running.

I suspect they would still have cancelled the project (the right choice, in this case). --DaveSmith 2003.04.10

When I was first reading Kent Beck's writings about XP, I was pretty sure that XP only applied to teams doing the kind of software they already knew how to do. I thought this in spite of the XP practice of SpikeSolution.

However, I have read of more than a few teams using XP for research projects...

This paper, describes a research group studying software agents... http://www.agilealliance.org/articles/articles/KnowledgeTransfer.pdf

I attended a presentation at Parc, Inc. (formerly Xerox Parc), where they researched a new paradigm for e-mail. The irony here is that the on-site customers were Human Interface Designers -- this was the first time they had to trade cost versus features / (gui quality), and since it was for research, they choose cheaper features that required some user training rather than more expensive features that would be more intuitive.

Also someone (doing XP for One), researching CPU efficiency, was on the XP mailing list.

KeithRay 200304.10

Jerry mentioned the "Hudson's Bay Start" technique here. The ensuing thread has been moved to HudsonsBayStart.

Kent Beck wrote this the Lean Development mailing list...

We start with 1-day timeboxes for the first 2-3 weeks, so people get a lot of practice estimating, writing stories, integrating, planning, etc. After 10-15 cycles, we're ready to move to 1-week iterations.

KeithRay 2003.04.18

Interesting. I've seen something similar in the project concept / scoping kind of activity (not BigDesignUpFront.) You have to cycle the "what we're trying to do here" about daily, until people have enough common definitions.

JimBullock 2003.04.21

Updated: Monday, April 21, 2003