InterviewTipsandTrapsforAssessments

I perform a number of assessments each year for my clients, to really understand what's going on in a project, organization, or functional group. They're intense, exhausting, and I love doing them.

When I coach new consultants or internal folks on how to do assessments, they sometimes fall into interviewing traps. So I thought I'd write up my tips and traps, and see what you've seen.

Interview tips:

use open-ended questions as much as possible
use behavior-description questions as much as possible to get real live examples
use meta questions to ask about what else to ask about
separate strategic questions of management from tactical questions of technical staff

Interview traps:

never ask leading questions, such as "is your manager a bozo?" You won't get an honest answer and the question diminishes your authority, authenticity, and credibility. A lot to lose in one question.
avoid opinion questions such as, "Do you like what you do?" Instead, reframe it as, "What's working for you here?" and "What prevents you from getting your job done?"

Have any other tips or traps to add? -- JohannaRothman 2008.03.26

I have found the following three questions to be valuable for unearthing data:

How's that working for you? (after hearing someone describe a process they are using to solve a problem)
What is the cost of doing nothing? What is the cost to you, personally, of doing nothing?
What questions haven't I asked you that I should have?... Ask them their questions.

SteveSmith 2008.03.26

Interview Tips

What question have I not asked that you don't want me to ask?
How did you come to be here?
How are things for you now?
What would you like to happen?

Interview Traps

Have the person's supervisor in the room while asking them questions
Have the person's peers in the room while asking them questions
Sit across the table from the person while asking them questions (try to sit next to them)

DwaynePhillips 2008.03.27

What question have I not asked that you don't want me to ask?

Tell me more (Dwayne).

That's a genuine inquiry.

I use the words -- tell me more -- frequently in interviews to elicit additional information so please add that to the list of tips.

SteveSmith 2008.03.27

Dwayne wrote (as a trap): Sit across the table from the person while asking them questions (try to sit next to them)

I wonder if there's a gender difference here, Dwayne. In a session last year, Jerry said that when men are talking to each other, they are most comfortable sitting side-by-side (as in a car), while women talking to other women prefer face-to-face. This insight about men came as a startling revelation to many of us women in the room, who had never noticed this behaviour while observing men (or perhaps not being observant about them).

I'm not at all sure that having an interviewer sit beside me would make me feel safer. I look people in the eye when I talk to them (except in a car), and feel uncomfortable with people who don't look me in the eye back. I agree about the table -- it's a barrier -- but I prefer to face an interviewer. -- FionaCharles 28-Mar-2008

A trap I've seen an assessor led into is one where the accessor only gets to talk to a few people who've been briefed on the fictional story that's to be presented. The accessor almost got out of the trap by wandering into a nearby cubicle to ask a question about if/how the resident was working according a document to do with ISO 9001 compliance that the accessor had been given (and which the cubicle dweller knew nothing about). The cubicle dweller stalled, allowing the ruse to continue.

--DaveSmith 2008.03.28

Wow, Dave, thanks for the jiggle. I'd been thinking about the questions, but had forgotten about physical setup and location issues. Other traps I've seen:

doing pair interviews. If the pair has a senior and a junior person, the senior does all the talking or "corrects" the junior person.
Group interviews with people from up and down the hierarchy. Either the senior people lead the discussion, or they decide who to punish later, or the other people don't say anything.
not getting a tour of the facility. Sometimes, people show me what they think I need to see. If I don't ask to see the rest of the facility, I don't know that the testers are working in a lab four floors and half a mile away from the developers--and the people don't tell me because that's normal for them. Or I don't see the cramped cubes. Or something else.

-- JohannaRothman 2008.03.29

Some tips:

know what kind of assessment you're doing and tailor interviews to fit the objectives. Assessing an organization or a project to an external standard (like ISO9000 or CMMI) is very different from a risk/capability assessment to no particular standard other than pragmatism.
depending on the assessment type, an interview team may be a better choice than going solo. (I've always insisted on at least one other interviewer when leading assessments to a standard, even when the standard was an organizational quality manual.) If so, define the rules of engagement with the team beforehand so everyone knows how to play his/her role.
in a team interview, designate one person to lead and one to listen for undercurrents the lead might miss. Empower that person to ask questions following his/her nose. Swap roles in different interviews.
Regardless of the assessment type, predefine a set of questions to use as a guide, but don't get too hung up on following it. Listen for the energy and follow that.
don't worry overmuch about feeling you're in control. Let the interviewee talk. If the interview goes a little off topic, you may learn something important you'd never otherwise hear. You can always gently steer the conversation back if necessary.

A couple of others, especially for new consultants:

Get some training. Interviewing requires its own set of skills. and assessment interviews have their own special variants. A good training course will give you the chance to practice with role plays and debriefs.
Especially when you're starting out, team up with someone more experienced. You will want to develop your own style, but never discount what you can learn from someone who's done it before.

I think any good training course is better than none. I had ISO9000 Lead Auditor training many years ago. Some of it was specific to the standard of course, but I've since been able to generalize a lot of it, and it's proven invaluable. (It also taught me what auditors do and don't expect, and that has been very useful in my consulting.)

--FionaCharles 30-Mar-2008

When doing a private assessment of the health of a development team, I like to look around to see what's on whiteboards or stuck up on the wall. If the walls and whiteboard are all clean, a flag goes up. If the whiteboards look like they haven't been cleaned in a long time (hard to tell, sometimes), a flag goes up. Either way, it leads to better questions to pursue. (I picked this habit up from one of Jerry's books.)

Lately, though, as I work with distributed teams, I've found that this technique isn't as effective. Distributed teams are impelled by circumstance to do their artifact sharing on-line. And without access to the group's Wiki-equivalent and to server logs, it's hard to tell what's really being shared (where "really" means being read). Still, the lack of "information radiators" makes me nervous.

--DaveSmith 2008.03.30

Regarding sitting facing a person across a table or sitting next to them facing the same direction...

My government training told me to always sit across from them so you "can see their beeeedy little eyes." An encouragement to confrontation. Yuck.

I encourage sitting next to the person especially when the two of you are looking at the same piece of paper. I would have the interview materials on the table in front of the two of us so we are all facing them at the same time.

DwaynePhillips 2008.03.31

Across vs. next to: how about splitting the difference by taking the corner of a table?

Hi whomever left this comment. (Who are you?) I interview candidates for jobs by taking a table corner. I tend to interview people in assessments where we're slightly catty-corner, but I need to spread out my papers, which makes my note-taking transparent.

I tend not to sit next to people, because then I miss the visual cues that let me know I need to ask more questions. I don't yet know how to articulate those cues... -- JohannaRothman 2008.03.31

Why are you using a table in the first place? I never put a table between me and the person interviewed. If I can't see their who body, I lose information.

Would you put your interviewee behind a screen? That's what you're doing when you use a table. - JerryWeinberg 2008.03.31

Jerry, do you take notes when you interview people for an assessment? I don't use a table when interviewing for hiring, but I do for assessments--at least, I have until now. -- JohannaRothman 2008.04.01

I was kind of surprised to hear about spreading documents out on a table. I don't typically review documents with interviewees, having always kept interviews and document review as 2 separate processes. I talk to people and I ask for stuff. Then I go away and read and think about the stuff. Any questions that arise from the docs I follow up outside the interview process.

How do other people do this?--FionaCharles 1-Apr-2008

Fiona, I find that when I think I'm going to *just* talk to people, they bring me documents they think I need to see. I frequently find in large organizations that few people really understand the architecture, so I have a paper that shows me the architecture and I mark it up as I go, to see if I understand it. (For one of my clients, that picture was the first architectural picture of the product in 6+ years!) -- JohannaRothman 2008.04.02

Johanna, In answer to your question, I usually don't take paper or computer notes when I'm going an assessment. As soon as I get away, though, I try to capture what stands out for me. That tends to filter out the noise and leave the most significant things.

Not always, though, and I certainly miss things that way, but I find notes intimidate some interviewees. When they give me specific data, like urls, or passwords, or numbers of any kind, I do take out a card and tell them I'm going to write them down. Everyone seems to understand the reason for this, so it doesn't create problems. But I always put the cards away immediately once the data are safely captured.

I would love to take photos of certain scenes, but that's even more disturbing than taking notes. I try to make mental photos as best I can, but a digital or paper photo captures things whose significance I may not recognize until months later. It can be safer if you have a digital camera and immediately show people what pic you just got, allow them to ask that it be erased. (Noticing what they want erased can be most informative.)

I'll have more on Dave's comments at a later date. Watch this space. - JerryWeinberg 2008.04.14

p.s. I should not that I never took notes in school. Never. All the way through the the Ph.D. I think I was absent from first grade when they told the kids to take notes. I believe that prepared my mind for paying attention.

p.p.s. Sometimes I work with a partner, and we take turns taking notes, but as unobtrusively as possible, and always with permission. I've also tried audio recordings from time to time, but there's just too much there to process.

p.p.p.s. When working as an expert witness, I'm usually warned not to take notes, as they can be subpoenaed and used to contradict what I'm saying as a witness. Any notes I take must be preserved, or I could be in big trouble with the judge. jw

Updated: Monday, April 14, 2008