Home | Login | Recent Changes | Search | All Pages | Help

BestPractice

A completely fatuous concept based on two dangerous assumptions:-

a) someone else knows what's best for you

b) you don't have to think about your context.

PhilStubbington 2004.02.22

Brilliant. I laughed out loud.

Contrast with the less fatuous concept of UsefulPractices.

-- JimBullock, 2004.02.22

I've started using simulations in my talks so people can see the differences in context and why some practices are useful sometimes, and almost none of them are useful all the time. Do you have other techniques for helping people understand the difference between BestPractice and UsefulPractices? -- JohannaRothman 2004.02.23

The discussion of UsefulPractices has kind of mutated to include this question, I think. -- JimBullock 2004.02.24

A useful practice is less constrained by cost. For instance, it costs very little to have a peiodic, written exchange of ideas with colleagues.

Sometimes a useful practice is a best practice, at least in my mind. For example, inviting the right people to a meeting. And planning how to use those people.

But, is a best practice always useful? No. Not if you can't afford it. For example, CMM Level 5 compliance for administering this wiki web costs too much. And it provides zero value.

A useful pracitce always generates value. The same thing can't be said for a best practice.

SteveSmith 2004.02.24

You shoulda been there!

At AYE 2003, I had the pleasure of having dinner at the same table where Michael Bolton and Laurent were discussing this very idea ("there is no such thing as a best practice") with a more traditional tester. What a great conversation, I wish I had less to drink and the Cajun restaurant was less noisy.

KenEstes 2004.02.24

Is it the distortion that Best means in all times and all places? Is it that some hide behind the term without giving it meaning? Is it the capital letters?

I've been struggling with this since I picked up "Lessons Learned in Software Testing" and wound up at the ContextDrivenTestingWiki. I have no problem at all with six of the seven principles of the Context-Driven School of Testing, but the second half of Principle Two gives me pause. To quote:

"There are good practices in context, but there are no best practices."

Kaner, Cem, James Bach, Bret Pettichord, Lessons Learned in Software Testing. New York: John Wiley & Sons, Inc., 2002.

I have hope that those here at the AYE Wiki can help as the topic is explicitly forbidden at the ContextDrivenTestingWiki. I would dearly have loved to have heard the conversation Ken Estes mentioned, though I fear it might have been a "No, there aren't. Yes, there are." discussion.

Let me start with what I believe, before someone assigns me meaning I do not have.

1. There are no practices that are, at all times and in all places, useful (including FrequentBuilds).
2. The phrase "Best Practices" is abused more than explained. I react viscerally to plans that include the phrase, "We will use best practices."
3. My current favorite catch phrase is "There is no perfect. There is only better." I use it whenever the conversation offers, "Because it's the right thing to do." in place of an explanation.

I am fascinated by Capers Jones attempt to identify "Best Practices" by using data, fascinated in the sense, "What does it all mean, if anything?"

Jones, Capers, Software Assessments, Benchmarks, and Best Practices. Boston: Addison-Wesley, 2000.

I find very persuasive Warren Harrison questioning the use of the phrase "Best Practices". Persuasive in the sense that I now believe no one, including Capers Jones, knows what is a best practice, as Harrison does.

Harrison, Warren, "Best Practices: Who Says?", IEEE Sofware, January/February 2004, pp. 8-9.

So much is background. So what is my problem? The second half of Principle Two is an absolute without explanation. Does it mean the English word "best" is without meaning in the software context? If so, how do you choose among several "good" practices in a given, very local, context? Isn't the one you choose the "best" choice, given the limits of your knowledge? If not, why did you choose it? Does it mean, there is no a priori best? That's something I would agree with given my belief number 3.

Best, as an English word, is a superlative. To me, that means, i.e. denotes as opposed to connotes, comparison. There is no best without at least two things to compare with. Is that what the second half of Principle Two is saying? There is no best without the specific and reasoned comparison of at least three explicit options within an established context? (Two being just a dilemma.) Such a comparison must be local in time as well as place, so it is impossible to say "This is best," before you've done the comparison.

In the end, I am seeking a means to persuade, starting with myself.

Can you all help me?

MikeMelendez 2004.02.25

I can help with this: " . . . though I fear it might have been a "No, there aren't. Yes, there are." discussion."

I know Michael, Laurent, and Ken somewhat. I cannot imagine any of them engaging in a dogmatic "argument by repeated assertion." Ever. In addition, Ken has a remarable perceptiveness, directness and honesty in reporting what he observes and thinks - almost like a "fair witness" from Heinlein. If any one of them was involved in a substantive conversation about testing practices, it was worth listening to. All three? Be bummed that you missed it. I am. -- JimBullock 2004.02.25

Jim, You took away my one crutch avoiding despair at missing that exchange. I wasn't refering to repetition specifically, but to all the ways we misunderstand each other when our starting points differ. Perhaps, we can talk Michael, Laurent, and Ken into reproducing it here on the AYE wiki?

MikeMelendez 2004.02.25

No need to despair, Mike. They're around and all varying degrees of prolific. I like your poking at this question, so, rather than dispair that I know you thus far only through this particular thin medium, I'll be pleased I know of another likely opportunity to learn something, should I see your name on an article, or at a conference or similar. -- JimBullock, 2004.02.26

When people in our field discuss "best" practices, they mean universally best, as in all projects should use these practices regardless of whether the practice fits. Here are some examples:

I worked at an organization where the developers were interested in performing peer reviews, until management decided to measure how good the developers were by measuring the defects counted in the reviews. (I know, classic mistake.) Once management started ranking people by the defects found in the reviews, the reviews became worse than useless. Now the reviews actively masked defects.

Even using my favorite practice, FrequentBuilds, I'm sure there are times when more frequent builds doesn't help. (See the disussion on UsefulPractices.)

The problem with a "Best" practice is that it's prescriptive, as if there is no other alternative to the practice.

Maybe a better alternative to the absolute would have been, "There are good practices in context, but there are no best practices that we have seen successful in all contexts." (Italics mine)

I am convinced that if you tell people what the objectives are, they will respond using practices that reflect those objectives.

Hmm, on reflection, I'm not sure this helped you at all. It helped me :-) JohannaRothman 2004.02.25

Perhaps the real problem with BestPractice is the various interpretations of the word "best". Some people will interpret it in the least generous way, assuming, perhaps unfairly, that the speaker is setting themself up as an authority laying down an absolute rule, to which they have a negative, and possibly non-rational, reaction. (Not anyone here, of course.)

If someone says "best", you have to take the time to ask them what they mean by "best". If they say "useful", you know (or rather, I tend to assume) they're qualifying the benefit.

-- StephenNorrie 2004.02.25

I agree with Stephen as well, but I wonder whether the term, with or without capitals, has become too corrupted by meaningless use -- that is , the term itself becomes a rationale for an option, prescriptive as Johanna puts it -- to be used rationally in conversation. I'm not saying the user can't be rational in her use, but that the communication will become distorted because of the meaningless weight the term is too often given. Phil, at the head of this thread, sums up the worst distortion.

Perhaps that's a better statement of what I'm coming to believe: "Best practices, in the software context, has lost any identifiable meaning." That takes me back to Warren Harrison. Perhaps, the term "best practices" should not be used until -- and if (my addition) -- it regains a broadly recognized meaning.

MikeMelendez 2004.02.25

I will do my best to remember what was said, but of course I can not claim to have a clear understanding of what was meant by all the participants.

The part I remember was that one woman (sorry do not know her name. Did not meet here at AYE, she was a tester, perhaps from Canada, and was not Sherry Heinze who I know.) was arguing that having testers follow "testing scripts" was a best practice. Michael was arguing that it was not a "best practice" because he could envision situations where it was not useful (expert testers working on a system where they "know" the bugs are not where the scripts told them to look). The woman was arguing that scripts are good for many reasons, it is a good place to start a novice tester (gives them a chance to learn about the system and when they are done testing it is clear what they accomplished. She had a bad experience where a novice tester went "off script" and found no bugs of significance and if he had stayed on script would have found serious bugs).

To me the whole argument boiled down to context/management issues. I think that both of them saw value in test scripts under specific circumstances (for novice testers, for ensuring coverage of the system) but were arguing about if you always wanted to use this one device for every testing job.

Michael seemed to be arguing that really you want the testing goals to be explicit because "test scripts" may not be the best way to get the results you want. He is more in favor of an "exploratory testing" approach but does not discount that a script might be useful to start a novice with. Focus on the results not the procedure since mechanically following a procedure can give you a false sense of accomplishments was my interpretation of his position.

I think her argument was more about "test scripts are a great place to start". When starting in a chaotic organization she could get people to agree that test scripts were a good practice and get people to start focusing on them and she would gather all kinds of data about the product and organization and steer her effort from there. So my interpretation was that it was her management of the situation was key and the scripting effort was a convenient framework to begin working and thinking about all the other problems. I do not think she fell into the trap of the robot manager who believed everything was great if all the scripted tests work, but she wanted some uniformity and repeatability and coverage in her testing process and testing discussions.

So in all I agreed with both sides of this vocal argument. Its the people and not the process that makes it work but process and standards can be useful and people with more experience should consider the standards guides rather then rigid rules. Once again we see there is much debate about exactly what "testing scripts" mean. I think both people would be against rigidly following a test script in every testing situation but how do we know what this best practice actually is if two people in the same field argue about it yet to my ears it seemed as if they were really on the same side.

KenEstes 2004.02.25

Might have been Fiona Charles.

Testing is a pretty good example for exploring "best practices." I think of testing in terms of predicates: assertions about the software. I also think in terms of testing giving me better accuracy in knowing about those predicates than I might get otherwise. So . . .

• Scripts are pretty useful to get direct observations of the software's behavior, in specific cases known ahead of time, and possibly employing people who don't know much about the system being tested.
• Exploratory testing is pretty useful to get both assertions and questions about the software that you hand't considered before hand.
• Regression tests are useful for confirming that something that worked before still works now. That's the old definition, actually.
• Automated regression tests change the price point for doing regression tests, and perhaps the accuracy (as machines are often more precise than people).
• etc.

For any of these best practices in testing, the information the practice creates may be useful, or not. The practice may be the best choice among the practices we know to get that information or not. The information from other "kinds" of testing may be a better investment, or not. We maybe ought to know better than we do about the practices that are out there to choose from, or maybe not.

I think this version of a BestPractice is fair, and valuable: Knowing what we know, among this list of practices, this is the best one for this kind of problem (and here's why we think so.)

-- JimBullock, 2004.02.25

(It probably was Fiona, and I'd rather continue the fascinating conversation here than attempt to replay the one we had there, which obviously has served its purpose.)

Perhaps we can preserve a more general, less context-dependent meaning for "Best practice", something like the following: "This is something that has worked repeatedly, so it costs very little for me to whip it out in any new situation and see if it might help (it usually does), or by its failure generate further ideas."

If you're a two-legged animal, walking is a best practice. There are situations where you can choose to do something else, like crawl or swim, but these tend to be exceptional. It's "best" in a sense that depends not on the context, or, if you will, on a part of the context that (hopefully) varies little over your lifetime, because it's the part within your own skin.

I can imagine that there are "best practices" of that sort for narrower categories of systems, such as "software developer". Thus, "industry best practice", yes, Virginia. I'm willing to bet that all such practices can be expressed in fairly general, non-software terms, because that's the level of generality they'd have to reach in order to fit that definition, "an adaptation to the general facts of life as a software engineer that all develop at some point or other". So, if "version control" is a best practice, it is as a transposition of the idea "ability to look back on previous states of the work in progress serving as known good baselines".

LaurentBossavit 04.02.25

That sounds like two flavors of best practice, for ignorance and experience:

• Ignorance - Knowing relatively little about this situation, this is the most likely best practice.
• Experience - Knowing quite a bit about this situation, this seems to be the best practice.

Or am I confused? -- JimBullock

Ah-ha. Thanks. This Ignorance thing is conflating two ideas:

• Doing some good in producing some useful software.
• Doing some good in understanding more about the situation (so we can do some more, better good in producing some useful software.)

So, at least these things:

• Ignorance - Knowing relatively little about this situation, this is the most likely best practice for producing some useful software.
• Ignorance - Knowing relatively little about this situation, this tool of mine (this practice) might help us understand more about the situation.
• Experience - Knowing quite a bit about this situation, this seems to be the best practice.
• ??Experience - Knowing quite a bit about this situation this tool of mine (this practice) might help us inderstand more about the situation. Maybe by figuring out what we know that "just ain't so??"

That's interesting. I've had the sense that something is off with "frequent builds." Same problem - it has two uses.

• Frequent builds is a very powerful ongoing practice for producing software. It's a pretty good bet a lot of the time when you know little else.
• Frequent builds, or one try it right now and watch what doesn't work build is an absolutely great way to help us understand more about the situation.

The other thought that goes with this, belongs off under UsefulPractices. -- JimBullock, 2004.02.26

To give an example from Laurent's points, I've worked where "version control" was considered so important that everything, including reproduceable binaries, was kept under it, definite overkill or so I believe. Where I work now, I get pushed by some to place our defect tracking software (Bugzilla - Perl/CGI/DBI on MySQL) under "version control", just in case we want to change it. I resist strongly because I want to be able to easily update when Mozilla makes major changes to Bugzilla. My take: "version control" is a useful practice but when and how much you use it depends very much on context. With regard specifically to development, does it really matter if you keep all the steps leading to a product several releases ago? So "version control's" usefulness is also temporally limited. So what does it mean to call "version control" a best practice?

MikeMelendez 2003.02.26

. . . there are branches of engineering where there are recognized and published "best practices" given a certain context.

He's right, there are. I think our biggest problem with talking about practices when we talk about software is dealing with the context. We ignore it.

He argues that software engineering is playing with fire by using the term without a similar meaning.

Also correct. The combination of playing with fire, developing practices, and ill-defined domain is the argument against the SWEBOK, and especially using the SWEBOK for certifications. And it's a correct argument. Cem Kaner is probably the most useful entry point to track developments with that ill-founded idea.

Steve McConnell was pretty much on point for the SWEBOK / certification push. In general, I've found his stuff to be pretty good summaries / surveys of more or less current practice, if somewhat myopic (No, not all software development is developing software products.) and not crediting his sources. That's probably his Microsoft origins at work - they did invent disk operating systems, windowing user environments, "word", "mail", "MQ messaging", etc. etc. Just ask them.

However, McConnell went right off his rocker with this, taking something potentially useful way, way, way too far.

My take: "version control" is a useful practice but when and how much you use it depends very much on context.

Including both "more" and "less." If you know your system will be in use for 40+ years, it makes all kinds of sense to archive and escrow the whole tool chain. That's making a baseline, not version control. If lots and lots of deployed software processing chains include (which means depend on) your particular tool, version control and it's companion change management become a Big Deal(tm).

With regard specifically to development, does it really matter if you keep all the steps leading to a product several releases ago?

Sometimes, yes. Some systems are so critical that keeping around everything you need to rebuild the system up from the raw silicon (and sometimes even that) makes sense. The one time your Mars rover, through absolutely impossible circumstances, gets itself into an infinite reboot loop it's real nice to have "all" the stuff on hand to build yourself a local rover instance to play with. Or when your software is still running in the year 2000, a date by which it "obviously" would have been replaced several times over, maybe the compiler vendor has gone belly up.

If you are talking about keeping the history of particular changes around, that similarly depends. How long will the effects of "before" and "after" be lying around out in the world? How about big effects? The questions comes down to: "For what?" which includes: "For how long?"

Rather than resist strongly, another tactic might be to ask whomever is pushing you to - er - push the bug tracking software under version control: "What do we gain by doing this?" "Version Control" is a solution. What's the problem it fixes?

"Version control" gets conflated - especially in software - with archiving, baselines and repositories. Also change management, configuration management, and target platform, product line, and technology life-cycle issues. It's useful to tease these apart when someone asks for "version control."

Rebuildable snapshot baselines are a lot less burdensome to create and manage than "version control" and may solve the real problem. Or there may be no real problem to solve, other than someone told them that "everything under version control" is a BestPractice. It's just one of several UsefulPractices that have to do with keeping your system available. - JimBullock, 2002.02.27

I take some exception to the 20 year future mark. How many compilers of twenty years ago still run on today's platforms? And if they do, will the resultant binary be the same? What is a reasonable horizon? Or you wind up saving everything just in case another Y2K problem occurs.

That's the point. If you need to be able to maintain the gizmo for 20 years, you need a story for how you'll be able to do that 20 years from now. Some stuff it's more effective to reimplement with current tools & etc. Other stuff not.

I have two demonstrations by example that this does happen. The first is the Y2K problem that was both hyped and real. It's just scarey how many banks handling your money have bits of their transaction processing systems that are binaries from a long time ago, that nobody understands or can reproduce.

The second example is a military tactical system with a designed operational life of 40 years. You whack copies of all the tools away, with emulators and spares of hardware. The "how sure are we that things will change?" argument is actually the other way around for a complex embedded system over 40 years. It's close to a sure bet that some time along there 1) Significantly changing the behavior of the system will be real, real valuable and 2) Some part of the tool chain will have gone away in the meanwhile, unless you take steps to ensure that the tools remain available.

In software, we might take a cue from other engineering disciplines in planning for the future. Large, specialized technical artifacts often contain built in spares because technology and the retail market will move on during the life of the system. Chemical plants often build in spares of specialty pipes and fittings, for example. They also keep the specs to have the stuff manufactured as custom jobs if need be. -- jb

More important, I doubt that most developers latch onto ideas because they are promoted as "best practices". Though, that may be my R&D context, where "best practice" means "heavyweight process, ugh". In my case, those engineers promoting "version control" as right for any source code encountered are twenty-plus year veterans up-to-their elbows in multi-threaded, multi-process, multi-processor code on eight currently supported platforms, with many times that supported in the past (the product is over fifteen years old). They all look like me, only not as ugly. They reached the idea based on their long experience -- with a single product and a pertually small, less than ten, team. I think the lesson is that no one can know all software contexts (the team has grown over the last five years). And that the problem expressed in "A specialist knows more and more about less and less. A generalist knows less and less about more and more," applies without humor. Yet, the software industry is young enough that there are those still active who do remember a time when it was possible to be a renaissance "programmer".

I do think that "best practice" is just one of the phrases we need to be very careful about. "Correct practice" and "the right thing to do" fall into a similar category. And we need to be careful that we do not turn "useful practice" into a mere synonym for the phrases we are avoiding.

To get back to the central point, the most effective argument has been that we had been unable to directly upgrade the previous defect tracking system, Gnats, due to local changes that nobody wanted to give up. One reason we had to change was that our ten-year-old Gnats C source would simply not compile on Linux, our current mainstay, and the last system on which it would compile was overdue for retirement.

MikeMelendez 2004.02.28

With the problem on the table, you can consider the context, and make a choice that seems best for you, at the moment, from porting the local mods up to more current compilers, to reimplementing the local mods, to dropping them as not worth the work. This isn't a technical issue. It's a people issue, wanting to reconcile several goals at once, for free. -- JimBullock, 2004.02.29

PhilStubbington 2005.10.05

But although it's my usual practice to work with real testers, I can't call even that a best practice. I'm currently managing a UAT, with a team of BAs and line ops people. The BAs have written detailed scripts mirroring the documented new production procedures for the line operations people to execute. (This is a new experience for me, and the scripts were written before I arrived.) In this particular narrow context, the course my team is following seems like exactly the right one for their business. -- FionaCharles 5-Oct-2005

I responded to this thread, inadvertently, with an article published a year or so ago in Better Software (better than what?!) Magazine. Best according to whom? For what purpose? Best compared to what other candidates? By what criteria? Measured by what yardstick? Does the "best practice" incorporate any consideration of economics? People? Skills? Quality? Scope? If a best practice is only a best practice in a given context, there is perforce a context in which it's not a best practice - so how can you call it "best"? This is simply language abuse.

By "best practices", people mean "practices that we like", or "practices that we claim to like", or "practices that we don't like, but which someone told us we should", or any number of pathological constructions.

Here's a test, for you, though: take out the "best" or "best practices" when you hear someone say "best practices", and see if the sentence makes more sense or less sense; is more truthful or less truthful.

Examples, culled from :

"QCG is a consulting services organization specializing in financial systems best practices, project management, implementations..." vs. "QCG is a consulting services organization specializing in financial systems, project management, implementations..."

"Welcome to the Best Practices of Technology Integration in Michigan Site." vs. "Welcome to the Practices of Technology Integration in Michigan Site."

"Fig Leaf's senior team of developer's is pleased to offer Best Practices Consulting to our corporate, governement and association clients." vs. "Fig Leaf's senior team of developer's is pleased to offer Consulting to our corporate, governement and association clients." (Note that the senior team of developer's doesn't include proper use of apostrophe's as one of their best practice's.)

MichaelBolton 2005.11.01

Marketing catch-phrases are hardly the best resource to find meaning. This is the discipline that gave us "whiter than white" after all. -- jb

You know, if "they" have a useful answer for even half of the shotgun of questions above, "they" might well be on to something. -- jb

The heuristic is "Take out "'best' OR 'best practices', and evaluate both results. I'll assert that in most cases, at least one of the modified phrases makes more sense with the words omitted than with the words included. It would be fun to see the examples people find, and more fun to see what people could offer as counterexamples.

MichaelBolton 2005.11.02

I think I've found one from that same conference:

"Top 10 Best - and Worst - Patterns and Practices For C# Developers"

I offer it while I'm mulling it over.

MikeMelendez 2005.11.02

PhilStubbington 2005.12.20

SteveSmith 2005.12.21