|
Home | Login | Recent Changes | Search | All Pages | Help
WaterfallFolkloreFrom WaterfallIsSilly Other Discussions of waterfall:
Here is the reference: Winston W. Royce. "Managing the Development of Large Software Systems: Concepts and Techniques", 1970 WESCON Technical Papers, v.14, Western Electronic Show and Convention, Los Angeles, Aug. 25-28, 1970; Los Angeles: WESCON, 1970, pp.A/1-1 -- A/1-9; Reprinted in Proceedings of the Ninth International Conference on Software Engineering, Pittsburgh, PA, USA, ACM Press, 1989, pp.328--338. A link: http://facweb.cti.depaul.edu/jhuang/is553/Royce.pdf -- CharlesAdams 2003.08.16 I believe that Waterfall as predictable schedule delivery has a range of workability. I also believe that most of our systems developed naturally tend to violate that envelope of workability. Factors that allow waterfall scheduling to work are:
My experience in software is:
Outside of the academic/training world, there's not much demand for "Program to copy cards to tape" anymore. Playground sized programs like that don't represent the essence of useful project work. I'd do something like that in a 1-2 hour exploration in some new language or new operating system. One essence of modern software projects seems to be managing the interfaces of components, libraries, cooperating systems that all move at different paces. The compiler system, the runtime library, the toolkits, and target cooperators over the web all keep upgrading without consulting your project schedule. Delivering value with less control of essential variations requires dealing with rapid change more effectively, less on up front planning. I also think that there are variations in our branches of software development that have differing amounts of these effects. Traditionally, embeded products were more insular, but web and WIFI and other effects are dragging the network effects into many of those formerly closed domains. The same thing happened to Data Processing / MIS / IT with first private networks and now the web. Holding requirements stable seems quantitatively more difficult now than 10-20 years ago. BobLee 2003.08.17 On re-reading the Royce paper last night, three things jumped out at me:
I do think when we talk "waterfall" we end up talking about three things, conflated:
The point is to get any given step "right enough" to allow the next step to proceed successfully. Where I get bent about this is when folks start insisting that there is god's one true amount of "right enough" usually whatever was right enough for that step in the most successful project of their acquaintence. Worse is insisting that there is god's one true way of doing any given step which universally and unequivocably produces the one true amount of "right enough." Sometimes card planning is good enough and sometimes it's overkill. Sometimes a great, honking pile of UML is overkill and sometimes, believe it or not, it's not good enough. It's as dumb to try to predict and project things you can't know yet as it is to decline to analyze when experiments are damn expensive. You don't check whether a gun is loaded by pulling the trigger, because that's a damn expensive way to find out. - JimBullock, 2003.08.19 Charles, thanks for the link! On my rereading Royce's oiginal waterfall presentation, I was grabbed by the nature of the 1970 environment Royce was prescribing for:
Given those constraints, Royce's appeal to Big Design Up Front makes sense. What has happened in the 33 years since this paper is that all of the environment constraints have been attacked in development.
BobLee 2003.08.20 Bob, Jim, you've got a lot of it right, but I was there, and there's a lot more to the story. The big emphasis of "waterfall" was that you had to have something visible (and, we emphasized, reviewable) at the end of each phase. This was a result of things growing bigger, not a result of things being small. Around the late 1960s, invisible programming (where the programmer went away and came back some time later with your product) wasn't working - for exactly the same reason it still doesn't work. At some point it becomes impossible to know the future or even understand the present. So, Royce and many others were attempting to convert an invisible process into a visible one. That's the big lesson, and it's still totally valid today. We still have developers who want to say "trust me," take your money, and not be bothered until they give you (or don't) what they think you (should) want. Any process that doesn't have visible checkpoints (reviewable checkpoints) is going to be an unreliable process. How far apart should these checkpoints be - that varies with the context of the customer's problem and the development organization's capabilities. There is no one right answer, as Jim points out. How far ahead should you project the sequence, again, that depends on the same factors. In short, no visible method with meaningful checkpoints is "silly." Calling such an approach "silly" is silly, because it doesn't take context into account. A process can be a better or worse fit for a certain context, and most organizations have at least enough different contexts to make four or five distinct approaches necessary if they want to do a good job. Of course, then they have to actually think about what approach they're going to use before they start - and that's apparently too strenuous for some people. - JerryWeinberg 2003.08.20
I am noticing two papers related to the contents of this thread. One is a pretty informed discussion of methodologies. The comments from one contribution to another tie out and tie together nicely. The other paper is about having several methodologies in one organization. I have gotten good results recently proposing a "stack" of methodologies, one for content, another for software, and a third for platform / architecture. The content / software / platform categories work pretty well for web-ish systems, and the "stack" word helps because it's familiar to them. I don't have the distinctions and implications completely worked out yet, but it's actually in draft and out for a couple of sympathetic reviews. As usual, Jerry's formulated the fundamental model that goes with the example - there are a bunch of methodologies in any large organization. Of course, if picking among several processes is strenuous, tuning your process to the problem at hand is going to be impossible. I wrote in IEEE Computer, in 1999 I think, that the good-as-in-consistently-successful project managers tune their processes to the problem at hand, especially when they're doing something unusual. So if we know this, why can't we seem to tune our processes, especially in the face of feedback that what we're doing could work better? What's the sticking point? (Make that TheStickingPoint.) - JimBullock, 2003.08.20 (It's not a methodlogy. Just do things exactly my way, damnit.) Thanks, Jerry -- I'd forgotten invisibility. About 1970, I was a FORTRAN programmer at Boeing, and I taught myself COBOL by reading every manual I could find. I discovered how much less complex COBOL "records" were to deal with than FORTRAN's "lots of twisty arrays, perhaps related, perhaps not." Cross fertilizing scientific language terseness with data abstraction (user defined types in the ALGOL, PL/I, C familes) would render some intent threads more visible. Structured programming wasn't on my radar yet, but good module design came close to it. We practiced "few GOTOs, especially upward, and mostly 1-in, 1-out design for readability. I remember our "truck factor" in 1970 was about 1. (Truck factor = how many people would need to be "hit by a truck" to kill your project dead as a doornail.) The transition from cards to disk-resident source made inventory visibility much better - Panvalet "directory listings" made it trivial to observe sizes, updates, latest change -- but only one revision deep. It was a no-brainer improvement for most shops. Shared visibility of software sources eases a lot of visibility issues that Royce wanted documentation to solve. It allowed passive "I just wonder..." to be answered easily, reducing a threshhold that used to be "Naw, it would be too hard." In summary, many of Winston Royce's issues relating to the "accidental" risks of software development have been eased by design, capacity, and tool improvements. Many of the "essential" risks still speak to us. His solutions used the feasible tools of the day, today we have options not remotely conceivable then. One interesting twist I've seen is the practice of test-first development, using the tests to force the design decisions, capture and document them in executable (tested) form. The progress in computer languages as human-to-human communication, not merely commands to the computer, has bought us a lot of better options to my way of thinking. BobLee 2003.08.21 ("Just what is your exact way? damnit!") The transition from cards to disk-resident source made inventory visibility much better Really? I can find a deck of punched cards. Since the bad, old card days I have many, many times been stumped by the mystery code which is in someone's secret stash somewhere, all the more invisible because it's just magnetic domains, hidden behind personal account security and the non-conventional naming convention for files and things, in obscuranto. Worse is when they're hacking the stuff in production, right in production, which no one else can get at or see because it is "in production" so has to be walled off safe. Script-y languages make that particular dodge easier. Couple that with the mystery configurations, and random stuff lying around in arbitrary config files, switches, command lines, and so on gives you a truck factor of about .1. If any of several developers (so-called) has as much as a bad hair day, everything stops dead in it's tracks. Management has no corner on the market for self-aggrandizement and control through information hording and obfuscation. I think, Bob, you are talking about access and availability of the artifacts. Done right, source (or anything else) in electronic form can be more accessible and available than a deck O cards. Many, many times I have encountered situations where knowing there was a deck O cards somewhere - locked in a safe, in a basement, behind a pit full of pointed sticks, in the hidden city of Atlantis - would have been an improvement. -- JimBullock, 2003.08.21 Many, many times I have encountered situations where knowing there was a deck O cards somewhere... Of course, knowing which deck (if any!) was desired got very, very tricky! Since refactoring cards didn't respond to "Undo" there were 2 standard ways of updating source: duplicate and refactor or refactor-in-place. Either operation was somewhat messy, and left droppings all over. The truck factor got ugly then. You might find 5 or 6 decks all of the same module(s) in differing versions in someone's desk. The "visibility of shared disk" I refered to was mainframe and later LAN development using server centralization for development partitions. When desktop dasd got big enough to hide clones of the whole system, visibility dived again until SCM systems got popular. A shared development environment has a lot of visibility gains over scatter & integrate. Today, most processes promote into shared SCM / build repositories, and "daily build & smoke" forces visibility. (See Mc_Connell's Rapid Development for visibility practices, or any agile literature.) BobLee 2003.08.21 In card days, there were people who exercised fine configuration control, and there were people who always made a mess for themselves and others. In tape times, there were people who exercised fine configuration control, and there were people who always made a mess for themselves and others. In disk days, there are people who exercise fine configuration control, and there are people who always make a mess for themselves and others. Two questions:
- JerryWeinberg 2003.08.21 I am sorry, Dwayne and Jerry, that there has been some discomfort in this conversation for you. Now that I know, I'll find other words. Indeed, I have used another, better term for being dependent on individual storytelling for information: folklore. The problem is that critical information is resident in folklore. When someone new comes along, the folklore must be recounted. If the stories aren't told, then you are dependent on the one person who knows that particular story for the work to proceed. When you depend on people to remember intricate stories in order to do their work, even those people who know the stories may forget some detail and have to redo something. In my more PC moments, I've even used folklore to describe a goal: "Let's stomp out folklore around here." Another slogan-like thing is: "Why are we telling this story again?" Telling a story to build culture and common goals is one thing. Telling a story to build some single code module somewhere is quite different, and silly a lot of the time. Jerry's four questions are about folklore, I think. Doing configuration control (versioning, baselining, etc.) well avoids the need for retelling the whole past in order to know the configuration you have. Do it badly and your configuration is coupled to the story and the keepers of tales. The whole, rich story is indespensible for working with the people involved with building a system, but, the whole, rich story isn't needed when all you require at the moment is the artifact that's been built. I wonder what happens when we think of "waterfall" or not in terms of folklore? -- JimBullock 2003.08.22 Jim - I think I like Oral History for your use of folklore. Relying on oral history has all the problems of playing telephone, but it has some advantages as warmer communication. The key here is "relying upon". BobLee 2003.08.22 Bob, Jim. In some projects, they don't so much have an oral history as an anal history. That is, some folks constantly strive to "clean up" the history so only good things happen, or at least happen to and by the right people. Anal histories are not very good at keeping track of what you need to know to learn to do it better. - JerryWeinberg 2003.09.22 In Federal government contracts, there is some information that you must keep on paper on file per Federal law. So, we keep that information on hand. What Jim calls folklore we call "corporate memory" and I have heard some managers claim that they didn't document information because they had a lot of corporate memory. It really isn't hard to keep information around, especially today with the ability to burn CDs. As Jerry mentioned earlier, the key is the people and their desire to keep information organized. As far as oral history and anal history go, I worked on a project a few years ago that lasted ten years. The managers above me kept changing in and out so we were having to tell the ten-year story over and over. So, I wrote a history of the project (about 30 pages with lots of figures) and handed a copy to each new manager. They seemed to like that. The key to writing the history is that I wrote it by myself and stated on the first page that I wrote it by myself from my perspective. Had I tried to write it with six other people we would have never finished it. That is the way it goes when people keep records and document the past. Someone saves some information and throws out other information. As hard as we may try to be fair and unbiased, we are human. DwaynePhillips 23 August 2003 Random thoughts as I read through this very interesting topic from top to bottom: When we have a problem, we attempt to figure out what went wrong, how to get it right, and how to get it down on paper. Once it is on paper, it becomes frozen. Some call it good and, therefore, unchangeable. Some call it bad because it is unchangeable. The best, the few (of whom I am often not a member) say it is only paper, take the ideas that work and add others to see if they work as well. Waterfall is a descriptive word, yet there is not complete agreement on what it describes. People seem to agree, at least, Waterfall is sequential. Jacopini proved that mathematical algorithms only require two control forms: choice and repetition. Sequential action was a given. In basic programming, I was taught that adding subalgorithms (substitution?) made the whole algorithm understandable. Waterfall and other process methodologies seem to be classified in just these terms: sequence, choice, repetition, and substitution. Is part of our problem our attempt to turn useful ideas into a form we know we can control with machines? Is the problem, as Jerry frequently writes and Dwayne just reiterated above, that we are all people? The folk in the folklore? MikeMelendez 2003.08.23 Actually, it was know since time immemorial (which means I can't remember not knowing it) that all you need is one control form - really the only instruction you need for any internal processing. (A subtract one, test and (conditional) branch.) That has everything wrapped into one (except substitution): process, test, conditional sequence. A lot of projects go wrong because there really is no condition under which they are capable of choosing a different (and more meaningful) path through their process. The waterfall process says that the condition you can apply is a gate - don't proceed past this point until/unless you have produced and tested certain artifacts. Most "modern" processes explicily say you can have branches other than gates. Sometimes these are represented as branching back, but you can never go back. Heraclitus said, "You can never step in the same stream twice." A waterfall is a stream. If the Greeks knew that, how come so many people today don't? - JerryWeinberg 2003.08.23 Jerry, I don't understand what you're getting at. What is that people don't know? That a waterfall is a stream? That if you "do it over" you get a different experience? --DaveSmith 2003.08.23 What people don't know, or don't act as if they know, is that even if you're in a block on your plan that has the same name as another block, you're not in the same block. Thus, loops drawn on process diagrams are always falsehoods, the kind of thing that managers use to fool their managers into thinking they're doing a good job on the project. Instead of loops, I recommend that the repetitive steps be written out, with subscripts indicating which, for example, TESTING iteration you're doing. The plan might show, say, five iterations of DEVELOP, TEST, which permits managers to ask sensible questions like, "Can you point to which testing box you're now in?" or "You're now in the sixth iteration, but our plan calls for a maximum of five. What's happening?" - JerryWeinberg 2003.08.24 Ah, yes. The Spiral can be used to obfuscate if all you show is a single spiral. Even if you show multiple, you need some way to make visible the "slips" of unfinished work from one iteration to another. 1. What's the common factor in good configuration control? People (note the first word) who realize that they are producing artifacts for the use of other people, including themselves. Configuration control addresses some of the elements of having a useful artifact - knowing what you have, being able to get the same thing again, knowing what has changed or not. The techniques are bookkeeping, and the tools leverage because people sometimes need some help with details. Without the human orientation of offering value to someone else, configuration control techniques or tools are just noise, or sometimes amusing gizmos.
2. What's the common factor in poor configuration control? People oriented toward anything but serving the people who use the artifacts they produce.
1. (extra credit) Do you think the next new storage technology is going to guarantee good configuration control? No. I think I'll claim that tools with greater reach will expand the scope and scale of systems we can manage with any given amount of human awareness, dedication and commitment. I think I'll claim that like schedules and software projects, software systems will expand to fill any tools we come up with. In any case, the guys in the shop next door have access to the same tools we do (most of the time) so the tools aren't a differentiator. Skill with the tools may be. Awareness, dedication and commitment usually are a differentiator.
2. (extra extra credit) What do you think the AYE Conference has to do with configuration control? It's all about people.
--- JimBullock, 2003.08.25 (See, I am trainable.) (extra extra credit) What do you think the AYE Conference has to do with configuration control? This year will be my first to attend I, so I am guessing on this one. It seems that AYE is about people trying to do better for themselves and others. Configuration control is simple in theory, but it can be difficult in practice. We need to put a little ego behind us and sometimes do things that are not our personal favorite in order to preserve the integrity of the product. If the product or the process is not exactly what I want it to be, I am fine with that. It is only a product - it is not me. Something like that. DwaynePhillips 26 August 2003 Yes, a lot of people seem to think that if the product-in-progress is kept under careful control, it will stifle their "creativity." Most of those people don't know the difference between creativity and confusion. - JerryWeinberg 2003.08.28 If the "careful control" goes as far as restricting people to following their origin design, done during the single design phase for coding has commenced, then opportunities for learning get shut out. The other extreme--continous (re)design--shuts out the opportunity of shipping. There's a middle ground, but people sometimes have a hard time letting themselves (or others) get there. --DaveSmith 2003.08.28 1. What's the common factor in good configuration control? Keeping the end in mind. 2. What's the common factor in poor configuration control? Thinking the means are more important than the end. The end never justifies the means; but without the end, the means are meaningless. MikeMelendez 2003.08.29 Thinking the means are more important than the end. That's a nice description of process zealotry, Mike. SteveSmith 2003.09.01 Not to mention anti-process zealotry. The process is just the means. I think arguments about less or more process are self-contradictory. Arguments about less or more control, less or more documentation, or the order that things should get done in make sense, but need to be resolved within a project. If I understand Jerry correctly, the seminal idea in the original Waterfall paper was that of decision points. Jerry calls them gates above. All following process models offer variations of decision points. Why not seek the useful ideas in the "Waterfall" model, rather than seeking to discredit it? Why does the software field feel so strong a need to declare that everything old is bad and only the new is good? But that's a different discussion. MikeMelendez 2003.09.01 Why not seek the useful ideas in the "Waterfall" model, rather than seeking to discredit it? Why does the software field feel so strong a need to declare that everything old is bad and only the new is good? People in other fields are just as guilty. For instance, I hear people talk about Newtonian physics like its useless. That notion is so ridiculous that it gives me a good laugh. A useful theory never dies. It just fades away and then is rediscovered. SteveSmith 2003.09.03 Maybe they follow your HypeCycle, Steve? I guess a theory or a methodology is a technology, too. - JerryWeinberg
Updated: Friday, September 5, 2003 |
