Home | Login | Recent Changes | Search | All Pages | Help

MoreProcess

In the last few weeks, I've noticed that more process jobs are open -- true QA jobs, not testing jobs with a QA title. I was surprised about this until I looked at the job descriptions. The companies are looking for GMP (Good Manufacturing Practices) experience for software people.

Well, I understand why - the FDA and other regulatory agencies are auditing the companies' processes and finding big holes. Big surprise. So the companies want to hire people to tell them what to do.

In your opinion,

• Do QA people understand software?
• Can a QA person apply GMP to software successfully?
• Is applying more process the best approach, or even a reasonable approach? Is there a better approach?

JohannaRothman 2003.02.03

In my experience, the focus of too many QA people is on improvements to the process model rather than the actual process. These people delude themselves into thinking that just by describing their desired model the desired improvements will happen automatically.

If management's objective is to market improved process, creating lots of process models is an excellent approach, perhaps even a best practice. Management must hide the dirty little secret that the process models have little impact on the actual process itself. But, who cares? It's all about marketing improvement rather than actual improvement.

If you want to improve process, you've got to have the buy-in of upper management to reshape a company's culture because the process is an artifact of the culture so changing the process requires a change in culture. Getting management buy-in is necessary but not sufficient. You also must gain the support of the people who use the process. One way you gain their support is to collaborate with them on the development of the new process model and support them through the process transformation.

SteveSmith 2003.02.03

>>The companies are looking for GMP (Good Manufacturing Practices) experience for software people.<<

I am one of those individuals looking for these folks for my clients as well.

I have a number of past clients that were in the FDA regulated industries. I have also served on a industry-FDA software task force for 2 years, having the opportunity to work with several national FDA software experts and can provide a little insight to your questions.

• Do QA people understand software?

To paraphrase Jim Bullock on another thread on this Wiki, it depends on the person. I have met a few QA folks in the regulated industries who do understand software. Most of these rare birds have learned about software in a different discipline (i.e., aerospace, military) and moved to the regulated industries after many years in that previous discipline. I have also met a large number of QA folks in the regulated industries who know relatively little about software development. This is not surprising based on the previous experiences of the these individuals. In the pharmaceutical field, many QA folks have never actually built (coded) software but are mainly involved in deploying COTS packages and providing validation services for those packages. Many come from the scientific displines, such as Chemistry, Biochemistry, Physics; others come from Medicine, Quality Assurance, etc. with little or no formal computer science or software engineering background.

Many of the QA folks in the pharma industry do quite well in their job since they focus on a very narrow area in the field (validation/testing) but they do not have the breadth or exposure in the other areas of software engineering. This leads to the difference between testing QA versus process QA. For instance, in a group of about 10 or 15 worldwide QA experts at one of my clients I was the only one who actually had done work in the area of software process improvement using ISO 12207, ISO 15504 (SPICE), CMM and TickIT. Most of this can be attributed to the fact that Pharma folks typically focus on testing and validation since they don't do actual software development. Process work is not really indicated in this situation. At one of my other regulated clients, in a IT group of about 40 individuals, I had more software testing experience (as well as the theory of testing) than the entire software testing group. This is not surprising since the testers in this regulated arena focus on testing COTS packages instead of general testing of new software applications.

At one of my pharma clients, we differentiate between Quality Assurance (QA) and Quality Management(QM). QA supports primarily COTS deployment and validation or testing. They have little to do with process management. QM does a lot more with broad quality interests in many areas of software. They do deal with process related issues. With one QM group that I work with we are setting up a global SDLC for all development sites, setting standards for information protection and security, developing standards for performance and load testing, enforcing standards for configuration management, metrics, estimation, project management techniques, best practices for automated development tools, vendor selection, vendor auditing and testing.

I believe it is easier to teach a classically trained software engineer about the GMPs than to teach an employee in the regulated industries about software engineering. However, I rarely find a classically trained software engineer asking me about the GMPs. If they learned about the GMPs (which is not hard, plus all the information is available on the web) they could open up a whole new world in which to work or obtain new clients.

• Can a QA person apply GMP to software successfully?

In general, the QA individuals in the regulated industries apply the GMPs to software all the time. It is really just a question of how well they do it and how close that they comply with these Federal laws. Application of the GMPs to software involves both traditional QA activities as defined above and applying a consistent, documented process to build the actual software. This is where the manufacturing part of the GMPs comes through.

In my experience, the medical device industry is head and shoulders above the other regulated industries (pharma, biotech, blood, food, clinical investigation) when it comes to software production. The medical device side of the FDA (called CDRH) has published many guidance documents on using software in medical devices. Basically these documents cover classical software engineering, something you would find in the many standard SE texts. (These guidance docs cover topics such as Software Validation, Requirements, Human Factors, COTS, Risk and Hazard Analysis, Testing, etc.) The medical device side of the regulated industry has several standards in print such as AAMI SW 68, "Software Life Cycle Processes for Medical Device Software." It's very similar to ISO/IEEE 12207.

What prevents QA folks in the regulated industry from applying the GMPs to software is usually a lack of knowledge of software engineering principles or experience in applying them properly. I have participated in a FDA-Advamed Software Conference for many years; the themes at this conference cover the full gamut of software engineering. The main problem is that industry folks never have the time to attend these conferences, or rarely have the time to read and understand the standards and guidance documents. For instance, the Pharma side of the regulated industry rarely read the medical device software guidances put out by the FDA. Another disturbing trend I found is that QA folks in these industries rarely keep abreast of the literature in the software development field. .

• Is applying more process the best approach, or even a reasonable approach? Is there a better approach?

The process folks would argue that more process is better. Is it reasonable? A big complaint in the regulated industries is the sheer volume of documentation that you create each time you build and deploy software. The costs are huge. A better approach that is reinforced by Barry Boehm's work is to get the most experienced individuals (or team) working on your project. I have seen this again and again in the regulated industry. An experienced team without formal processes can run circles around a less experienced team that has hundred of pages of processes and procedures.

JohnSuzuki 2003.02.04

I have not worked in the regulated industries, having spent all my time in advanced R&D. At the same time, I can't imagine that more or less "process" is either good or bad. "Process" is just a tool. I have found the attitude, that more is better, common among management. That's been true even among management that says, "We don't have time for process right now."

I've worked places where QA and testing are both intended to serve marketing ends. More "process" is better and more "testing" is better. In such places, the developers frequently do not have the time to deal with problems found by testing, let alone conform to a specified process. ISO 9000 has frequently played a part. The most important teaching in such places was to only answer the questions the auditors asked and given doubt about the validity of an answer to tell them, "I don't know but will find out," knowing the auditors' schedule was full and they would be gone in a few days.

I've worked with those who define a "good" process and push for everyone to move to it. They could be QA, developers, or management. They know where they want to go. More common are those believe any change is difficult and hang onto current traditions, often without questioning them. They know where they are. I think these two ends need to be tied together with a map and a willingness to trust the terrain if the map and the terrain disagree.

I know its not the common usage, but I suggest "process" is no more than how we do things. If we confuse that with the paperwork done to help others learn how we do things, we get lost.

MikeMelendez 2003.02.04

These folks want more process because they think it provides them a structure in which to perform. And, they think it will make them more predictable. It might.

I also received two calls for more QA people today. I wonder if this is because the QA staff was canned first during the last round of layoffs.

John, when you said that GMP is applicable to software, I certainly don't disagree with you. What I find in practice is that when companies attempt to apply GMP to software, they end up with a bloated rigid process that works for large projects and doesn't work for small or technically risky projects. So the company's strengths, what got them to market and successful, is obliterated by process. Distressing.

Should we be teaching people about GMP or other process information in school? Where did you learn it? I learned my process knowledge on the job and with some company-sponsored training.

JohannaRothman 2003.02.04

John is bang on. Building from that, one thing manufacturers do is make the distinction between kinds of work, and the associated processes:

• R & D follows one set of processes.
• Product validation is a different kind of thing.
• Manufacturing "QA" tends a lot toward Crosby style repeatability.

When attempting to apply manufacturing "processes" to software, one failure mode (another engineering model that would help software a lot if it could be applied without abuse) is choosing processes from the wrong part of manufacturing as models. Manufacturing "R & D" models work just fine for "R & D" type software activities.

Personally, I think most software work is more like production in a semi-custom discrete manufacturing shop than anything else. Occasionally there are whacks of work that really track toward "repeatability". Lots of screens and transactions in an ERP conversation is one example. There are small amounts of work that correspond roughly to actual R & D, and to product validation. In software, a great deal of so-called R & D isn't, and a great deal of what is R & D for a particular organization doesn't have to be - somebody else has dealt with the problem. This last is a consequence of the limitations of our professional literature compared to the engineering disciplines.

Just MHO, and for once my post is shorter than John's. I'm so proud.

- JimBullock, 2003.2.4

Johanna,

>>What I find in practice is that when companies attempt to apply GMP to software, they end up with a bloated rigid process that works for large projects and doesn't work for small or technically risky projects. So the company's strengths, what got them to market and successful, is obliterated by process. Distressing.<<

I agree with your observation. I once reviewed the validation of a simple half page spreadsheet program in the regulated industry. It took some 25 pages to document the validation/testing package for that simple application and a few weeks of calendar time to complete. The standard process was too bloated to work on small or simple testing/validation project in an efficient manner.

>>Should we be teaching people about GMP or other process information in school? Where did you learn it? <<

From my experience and personal knowledge, the GMP's are not really taught in university curriculms (although some Pharm MS and PhD programs teach introductory classes on the topic). Most individuals in the regulated industries learn from industry teachers, peers or mentors or from on-the-job-training. Many academic types would probably not be good instructors on this subject matter because many don't have any practical experience in this area. Also, the GMPs are not really considered an academic focus or subject area so I suspect that universities would not encourage its faculty to teach or do research in this area.

JohnSuzuki 2003.02.12

We've had this conversation so far without defining what "GMP" means. So what is a "good" process in manufacturing? Which processes? What practices matter? What is goodness in those? One of the nice things about the manufacturing I've encountered is that by and large the practitioners are willing to keep asking these questions, vs. looking for a prescription somewhere. They are pretty adept at the "best current practice" / "keep asking questions" dance, exactly as we are not in software.

Some institutions do teach application of manufacturing or engineering tools to software. The "Quality" & related programs at the Rochester Institute of Technology have applied techniques and models from manufacturing to software. They also teach some process improvement models. Anecdotally, I've heard of research at RIT on application of Taguchi techniques / Robust Engineering Disciplines to software, although I haven't found the paper or the researcher yet. Several universities funded by, through or in association with Nasa have also done so. They seem to be engineering schools with good software programs, mostly. I think some more fruitful models to borrow might come from product engineering and systems engineering (real honest-to-Murphey product engineering and systems engineering from manufacturing.) There are many manufacturing programs out there that teach this kind of stuff. Rensselaer has. I'm sure there are some in Boston. And over the years NASA has funded, or variously been associated with research of this kind at various institutions, although the current "reliable computing platform" initiative looks to be more brochure-ware than content.

Meanwhile, while a prescriptive model like SEI / CMM is attractive, I'm a lot more comfortable with the related ones from Europe that don't suggest a particular consistent "level" of "sophistication" is required in all of a known collection of practices all at once. Some places have a very developed requirements process, while their build & cm is weak, and still they do OK. Others it's the other way around, and still they do OK.

One instrument from Europe - the name escapes me at the moment - produces a kind of plot of an organization's "capabilities." You plot "how far out" you are on each of a bunch of measures, and get a shape. The CMM's approach, grossly oversimplified, would have "levels" for each of the KPAs on each radius, and you "ought to" get a circle - every process is at CMM II, all at once. I think different shapes, meaning different emphasis, are appropriate for different problem spaces. This kind of graphic - it's name also escapes me at the moment - is one of many tools from Systems Engineering and Product Design disciplines, which we might profitably leverage into software, more so than Murphy-Help-Us Six-Sigma, for example.

Just MHO. - JimBullock, 2003.02.13

BobLee 2003.02.13

Yeah, that. I found a free, automated chunk of software on-line sometime in 2000 that would make a graph like this from your answers to a pile of questions about your software practices. That was something like 3 computers and two moves ago, so the software and the link are "somewhere" - I can't locate either at the moment.

JimBullock, 2003.01.13

Thought the second:

If we're talking about borrowing process or practice models from other disciplines, I've got a variation on my previous comment about "several kinds of manufacturing processes." In web-space, I've noticed that there is a stack of different kinds of artifacts, which are deserving of different kinds of process disciplines. One problem that organizations have with web processes is applying a wrong kind of process to one of these layers - too much or too little.

That one's probably worth an article or paper of some kind. If someone wants to coauthor such a paper with me, drop me a line and we'll talk about it. I've been meaning to get this one out there "somewhere" for a while.

- JimBullock, 2003.02.13

Thought the third:

"Do QA people understand software?"

I've met so called QA people who work in manufacturing or other process-heavy fields who, as best I can tell, understand neither manufacturing, nor process. I have met some who's command of spoken language seems sketchy at best, although their parts seem intact, and they do make a lot of noise. I have met others who are very definite and crisply articulate about a great many assertions that are neither observably correct, nor mutually consistent.

I suspect that a QA person from manufacturing (or another non-software discipline) who understands the discipline and QA might stand a chance of applying what they know to software, while one who understands neither is a poor bet. These more mature disciplines have their share of the clueless, including in QA functions. We should not presume that merely because the discipline is relatively mature, all credentialed, certified bearers of knowledge from that domain actually have anything like a clue to share.

Unfortunately, cluelessnes seems to be fairly well distributed in this universe, and acts a lot like a conserved quantity (neither created nor destroyed, simply changes form.)

- JimBullock, 2003.02.13

Any place where that results in a false statement, there you will find trouble. For example, the business Steve mentions abour process models vs. process. Substitute the phrase and you get (with a little grammatical inversion) "models of the way we do things here and now." If the models are something else, like "the way our management would like us to do things someday," or "the way we would do things if we were better human beings," then you are in trouble, and the source of the trouble is rather easy to identify.

Once you start doing this, you can start correcting people's speech when they wax theoretical, or about the past or the future (process in the sky by and by). A process is simply the way we do things. If we don't do them, then we're talking about something else, like a theory of process, or a something written on a piece of paper, or somebody's fantasy. - JerryWeinberg 2003.02.16

OK. Now that I've stopped laughing, I can respond.

It turns out that at a certain place I - er - heard of once, yeah that's it - every time I'd ask "What about . . . " I'd get back: "We don't do that around here." The pattern was usually: complaint about egregious, expensive, disruptive and ongoing bad thing, "What about . . . ", "We don't do that around here . . . " Now I knew enough to try this with the people who both felt the pain of the problem, and could change what they did around here.

After the first couple times, I managed to respond: "Yes, I can see that. I'm suggesting with all this grief over here, that it might make sense to do somethign different. How would we try it?" The response wasn't "No." it was "Huh?" The thing that baffled me was that The idea of somebody doing something differently was so foreign that they just couldn't get it in their heads. Over time I found a few people who thought that way, and worked with a few more who started thinking that way.

So I think there's maybe another kind of process thing that isn't real yet - "candidate process." I think that's maybe where the traction is in making things better. You've got to be able to at least imaging that some other way might become "The way we do things around here."

- JimBullock, 2003.02.22

The following is from my quotes collection, the ones that get appended to my sig at random when I send email :

"The greatest obstacle to transforming the world is that we lack the clarity and imagination to conceive that it could be different. -- Roberto Mangabeira Unger"

I think this also relates to what John said about "industry folks never have the time to attend these conferences".

LaurentBossavit -- 2003.02.23

Interesting. I've been suggested for two jobs recently: A mentor to "application architects" who are essentially internal consultants to project and lead the process . The second was a director of QA position - a sort of govern ALL of the QA that Jim mentions. In the first case I was told that the organziation wanted to follow CMM (on to victory, supposedly). So, I was wondering what prompted this surge in process.

I think the simplest answer is managers thinking a "process" is an proven means of reducing costs. Instead of reading "the way we do things", I kept hearing "procedure". This recipe will get us a tastier and cheaper cake.

I am meeting with some of these people tomorrow and you all have given me some thoughts about questions to ask. I'll let you know if anything worth reporting comes up.

- BeckyWinant 2-23-03

Oooh, oooh, oooh, I can name several things that I - er, this guy I heard about, yeah that's it - wishes he found out before going to the "We don't do that around here" place.

Simple data:

• So how does <this, that the other> happen?
• Who's on the hook to make sure it works?
• How do you decide to ship something?
• Ever not ship something? Based on what?

What they want or expect:

• So how did this position (or contract) come to be open?
• Tell me the story that's supposed to happen. So you hire someone, say me. A week later . . . And after six months, what are things like around here?
• Let's set up some conversations with the folks who will be using the information that comes out of QA. (Then watch what happens.)

I've been amazed, for example, at the willingness of people to pay for "placeholder" QA people - someone to sit there so they can say they have QA and feel all quality and stuff. That's an expensive high.

I made a whole lot of notes after my last J - O - B about things I wished I'd asked. Heard a whole lot of good questions from several colleagues who were considering one job or another over the last year. One of these is Ken Estes. I think there's a guideline / article in there somewhere. I'm not the guy to do it, just now. Too much else going on for me.

I'd be a little suspicious of either place, you mentioned, Becyk, from the info at hand. What's an "application architect" have to do with shepherding the process, especially a CMM derived set of processes? If they can't call things by their right names, you're starting with a problem. The second, "director of it all" is a little less obviously confused, if they realize that these are three different things, and if they are willing to treat them differently. If the idea of director-ness is to take "QA" appropriate for one process and impose it on the other two, well, that's a problem.

Then again, I do live in Seattle, where I finally learned that here, "QA" is local for "testing" and there isn't any ownership of "the process" so the conventional definition of "QA" has nothing to apply to. Along with responsibility for the artifact and authority to determine what it is comes responsibility for the process and authority to determine what that is. So we're always making up "how we do things around here" which amounts to anybody doing work isn't accountable in any fashion for how they do that work. I think I just answered my question about architects herding processes, didn't I? Is the gig in Seattle? If so, I'll invite you over the next time I cook.

- JimBullock, 2003.02.24

Fit the second . . .

Relative to lots of people suddenly deciding they need some of that QA stuff, I've seen it really be a combination of timid management and lack of productivity. Engineering or development or whatever they are called are shipping crap, haven't shipped anything, are fighting, or so on. So we'll get a "process guy" who will make all this go away, without us having to actually address the problem directly. At a minimum, "process guy" creates a convenient scapegoat for our existing bad project management, lack of project charters, or prima donna developers when their ridiculous behavior continues to produce the nothing it has been producing. Not that I have an opinion.

After several trials, I've finally learned to ask myself whether this is a scapegoat position being talked about. In the end the people doing the functional management, of R & D, of Engineering of Production, have to be accountable for their results, and can only be accountable to their boss, not to you the QA guy. So, there are a gaggle of questions to ask about who's on the hook for what. If they're not clamoring for the information one can produce in QA, or the organization (meaning clarity and productivity) one can produce as process herder, they don't want what they say they want. The job is something else.

Just sign me: Made that mistake more than once, but I do learn eventually.

JimBullock, 2003.02.24

Yeah, responsibility for QA (overall quality) is another interesting thing. It seems from my local search that this plays out several ways. In fact, if you can think it, someone else here is too.

Regarding Seattle - one opportunity is working with a colleague from Denver in Seatlle. I'm his "henchman", so at this point (not closed deal) all I know is it is a faciliation of business goals & objectives leading to planning. If I find myself in Seattle, I will definately call the Chez Bullock for an invitation to dinner. (I don't know why... but I know more people in Seattle than any other place I've never lived in).

At a minimum, "process guy" creates a convenient scapegoat for our existing bad project management, lack of project charters, or prima donna developers when their ridiculous behavior continues to produce the nothing it has been producing.
Often I've observed this to be true. Hiring an outside consultant to do this almost makes this assessment a no-brainer. "Process" people in particular need to be savvy about human dynamics and sniffing out stinky politics.

Often when I hear about corporate politics I hear sound bites from a talk Tom DeMarco gave at a Cadre Technologies user conference. Cadre produced a decent case tool for a while. Tom's talk outlined the difference between "good P" and "bad P". After digging into the greek roots of "politics" as a public discourse - a good thing, he talked about bad P where essentially blaming reigned. His catch phrase phrase was "bad P runs downhill" - meaning people at the top of the organization determine the culture. But "P runs downhill" is so memorable.

Background: When Tom spoke at the user conference it was after a merger in which two presdients (I kid you not) were still at the helm because the control issues apparently couldn't be resolved.

BeckyWinant 2-25-03

Jim said, "Then again, I do live in Seattle, where I finally learned that here, 'QA' is local for 'testing.'"

I always thought that it was title promotion. "Quality Assurance Engineers" was sort of like the "Sanatation Engineer" who stops by with the big smelly truck and picks up your garbage once a week.

I don't know how it plays out in garbage collection, but I think misnaming software testers hurts software development.

-- ShannonSeverance 2003.02.25

You are correct. I think the problem is that the title silliness lets developers off the hook. I like to play the "What does it take to do this job?" game.

• If I'm poking at a system to generate some information, that can be valuable, without to many preconditions.
• If I'm "testing" a system, that word usually drags along with it some expectations on the information produced - that it will be reproducible, or at least traceable to something. That the information will be relevant to some particular concerns about the system.
• If I'm doing "quality engineering" well, I had better know what "quality" means, and have some influence on both the artifact and how the artifact is produced.

Unfortunately, many testers feel disenfranchises or are developer wanna be's or both. So they are unwilling to challenge the job requirements for fear of losing the inflated title. Everybody looses when the game is fooling ourselves. What I don't get (yet) is why people involved in testing, a practice that has a very high reality content, get hooked so easily by this one.

- JimBullock, 2003.02.27

I don't think it is a case of applying more process, but understanding what, if any process is currently defined and understood. I have found that for process-immature organizations (CMM level 1, if you like), development often proceeds in the absence of a commonly understood process. This starts with a vague notion of requirements (in most cases without even considering system vs software vs functional, etc.), to an even vaguer understanding of planning, tracking, change management, and so on.

Although there is a lot of truth and sense in some of the earlier posts about process/model abuse, adopting a model such as the CMM in an environment where "the way we do things now" is at best poorly defined and understood, can be a powerful way to steer the organization towards considering, often for the first time, "how they play the game." For these organizations, the CMM gives them a framework to work from, giving 'process rookies' an idea of where they need to start, and as much as anything showing them that to start with process does not have to be perfect and all-encompassing.

The problem is when people look at the model as the design spec. They create a bunch of policies, procedures and templates and think they've got it licked. Process describes how things get done, from understanding the overall goals of the organization, setting expectations for folks, walking the talk, enabling that behavior, validating the resources and expectations with quatifiable measures and oversight.

Initially, QA's job is to mentor the projects in terms of establishing appropriate processes to meet the organization's expectations, and provide feedback to management as necessary when those expectations are unrealistic. In the early stages of concerted process improvement, much of the work should revolve around communication more than on documentation.

For me it's not a question of "applying more processes" but starting by "understanding, communicating and implementing the fundemental processes" to enable the organization to meet their goals.

RobWyatt 2003.10.20

Let me offer a way of thinking about process that helps me out of a lot of jams. Whenever you see the word "process," substitute the phrase, "the way we do things (here, now)."

In this view, there can't be "more process." There's always the amount you have, what you do. There can, however, be more process documents, and more process documentation activity, etc. As yourself, "Is that what we want?" - JerryWeinberg 2003.10.20