Home | Login | Recent Changes | Search | All Pages | Help

SystemModeling

Let's discuss the art/science/philosophy of systems modeling.

I read EstherDerby's article on measurement on the AYE conference site this morning. Great article, good advice, very relevent.

Reading the example about the customer support system made me think of Senge's classic systems thinking simulation, the Beer game, where if each player in the game doesn't take into account the entire system, local optimization can bring the whole system to its knees.

Since I'm a requirements and user interaction guy, it looked to me like the system model was missing the root cause, and therefore only optimizing locally.� Here is my revised model with a layer that talks about the software system quality as the root of the problems with customer support. Basically, in an ideal software system, there would be no customer support needed, so why optimize customer support!� So, if we get better at customer support without improving the quality of the product, we'll still lose customers to our competitors.

- JimJarrett

SteveSmith: Jim, I've been studying the relationships you propose between usability and the other nodes. Help me test the hypothesis that usability has an inverse relationship on all the nodes indicated. I can stretch myself and see how that more usability would create a more satisfied customer and, perhaps high-quality product knowledge and documentation , but I don't get how more usability creates less reboots, less number of problems on each call, less problems requiring more than one call. Inquiring minds want to know... -Steve

JimJarrett: Steve, a higher usability system leads to less user confusion, therefore requiring fewer support calls. I think I made the effects of usability go to those two # of problems measures because there was no measure for "number of support calls." Also, the "# of reboots" measure is probably should have an inverse effect on "Software System Quality" (instead of the way I drew it initially). I've attempted to rework it a bit in the following diagram. (I managed to stop myself and not try to show how higher usability and requirements quality reduce software complexity... for demonstration purposes only, after all).

The interesting aspect of improving the customer support part of the system, to me, is figuring out how customer support can have an impact on system quality. - JimJarrett

SteveSmith: Esther, I went back and reread the article on the web site. I still like it a lot. Regarding the dot with arrows, I now see it represents an inverse relationship. I'm used to seeing just a dot so I wanted to check if I was interpreting correctly. Thanks, -Steve

EstherDerby: Thank you. I learned how to do DOE with black dots to indicate inverse relationships.... the art folks at the mag added the up/down arrows I kept them. The theory is they provide a visual clue to the inverse relationship. - Esther

System boundary is an interesting question (I've got another article related to that floating around somewhere). This plays out in interesting ways in organizations. Sometimes I see folks limiting the exploration of the problem space to factors within their span of control... which I suppose is a natural outgrowth of the "don't come to me with a problem unless you have a solution" mentality prevalent in many management cultures.

BYW:The little case study in the article, the only actual measure the company had was call duration (tracked automatically by the telephone system). The case study comes from a real company. The managers in that company instituted a measure (tied to performance and compensation, of course) on call duration. And the support staff got really good at getting customers off the phone quickly...with the result that customers had to make 10 - 20 (short) calls to get a problem resolved.

SteveSmith: James, span of control is one way to bound the problem. Whether we are addressing the true problem, rather than a symptom seems to me as a matter of perspective. For example, from a customer perspective waiting on the phone is a pain in the butt and a source of dissatisfaction. But, if the telephone support is outsourced, long call times are a source of a paycheck to some person(s) and some company. So, the first order of business for a person or group doing a Diagram of Effects is to determine their customer. -Steve

Incidentally, I uncovered a problem in the Wiki when adding this comment. Someone else had opened this in update mode simultaneously and I had to re-input this comment. JamesWard

Edit collision avoidance is a feature. It's frustrating when it happens, but the server logs show that it's a rare occurance. --DaveSmith

1. There is value in doing the modeling at all, even if you don't spend much time, because it stirs a great deal of thought and commentary - you start asking the right questions. Making the assumptions visible is key.
2. Iteration and refinement is necessary... I haven't gotten anywhere close to a model that I'm comfortable with, but I've learned immensely from the process and discussion so far.
3. Models can run away from you, but they can be pruned back and made to communicate well. It's the classic expansion and contraction cycle in any good design process.

I also ran into a couple of questions:

1. If you have a measure that has a direct effect on another measure, but also has an indirect effect on that same measure through another measure, should you show both? I found myself removing the direct effect, and only showing one "path" in some cases ('tho not all, looking at it again). Any rules of thumb for which effects to emphasize?
2. Sometimes it would be nice to show relative magnitude of the impact of a measure or effect. Any ideas on how to notate that? For example, I think requirements usually have the largest impact on system quality, but I didn't show it in any way in particular.

From JamesWard: When you say "requirements usually have the largest impact on system quality" what specifically do you mean? Are we talking about the nature of the requirements themselves, their completeness/correctness or some other measure of requirements? Complex and/or poorly understood requirements may likely result in a poor quality system, however you measure system quality.

3. How much use does the model have if you weren't part of the collaboration that created it? We seem to lose a lot of the insight and rationale while we model. Anyone used any methods to capture more of the "context" and rationale in the modeling?

SteveSmith: I redrew the diagram, despite my desire to only add items. I gave up trying to figure out how to add nodes and relationships after a couple of failed attempts, especially the one that crashed my machine after working for 45 minutes.

I made explicit what I thought were conceptual measures, rather than actual measures. I played around with what which items I thought were controllable. I added the nodes that are in blue. I changed the relationship between usability and quality software and some others I can't remember because it's bed time. Later, -Steve

4 September 00: Deleted the link between Quality of Problem Resolution and Customer Satisfaction. Added an inverse relationship link between Quality of Problem Resolution and Number and Magnitute of Problems. -Steve

SteveSmith Jim, I've seen excellent requirements studies ignored by the developers. I believe that human intervention is required to make quality work transfer to another area. This kind of intervention is needed to create a positive relationship between the usability of a system and the quality of its documentation and the quality of the knowledge about the system. We may have a very usable system, but the organization may have skipped the documentation and the knowledge transfer.

Regarding feedback from customer support into system quality, we would have to add a new node for that quality. We have a node for software system quality that would, if we added a node, affect system quality. Customer Service would affect system quality also. I think the node for customer satisfaction serves the same purpose so I don't think adding the new node is necessary. -Steve

SteveSmith: Our thinking is very similar about boundaries. I prefer to explore the problem without boundaries and pull back later. -Steve

SteveSmith 4 September 00
I redrew the diagram from the perspective of how variables between customer service (help desk), sustaining engineering (maintenance), and systems engineering (development) affected customer satisfaction. It never stops amazing me how interdependent that our supposedly separate systems are.

This diagram needs a lot more work, but through the diagramming process I now have more clarity about what may lead to increased customer satisfaction. For instance, I belive empathy can buy a lot of customer satisfaction, especially if its coupled with repairs that actually solve problems in a reasonable amount of time.

In the situations I've seen, if empathy isn't present, management must intervene with hiring, firing, training and monitoring. Those actions aren't represented on the diagram.

Another important variable that isn't on the diagram is investment in staffing and resources. If someone else is game, they can figure out how to add that variable.

Finally, I suspect that tradeoffs between variables related to economy, quality, and development determine the number and magnitude of problems. For example, some developers are willing to tradeoff the economy of customer service for more rapid system development. That choice seems reasonable to me as long as it is made explicit, which, in my experience, is rarely done. -Steve

I wonder if we might take a few Wiki pages or paragraphs or illustrations and get three or five or ten or twenty fairly detailed understandings from readers to test whether the intent of their authors was accomplished by the readers? What does it take to do rigorous user testing of expository material? -- DickKarpinski

You're right, without a key, these diagrams seem quite mysterious.

The clouds represent some measurable effect within a system.

Clouds connected with an arrow change in the same direction: as one goes up, so does the other; as one goes down, the other does, too.

Clouds connected with an arrow and a black dot change in an inverse way: as one goes up, the other goes down, and vice versa.

The half shaded box indicates an action with free choice.

This is described in the article (close to the figures). I think the people who have contributed to this discussion have read the article the prompted this thread. An incorrect assumption, as it turns out.

I use this modeling technique with clients sometimes. It takes a few tries for it to become intuitive... but once people get the hang of it, it is a useful tool to discuss the factors contributing to a situation. EstherDerby 10/31/00.