Bad question, bad exam

PM Certification commentary by William Duncan, IPMA-USA Certification Director.

I stopped by a website recently whose proprietor specializes in training people to take a certification exam offered by one of our competitors … “That Other Organization” or TOO. Here was the Question of the Month offered by this training provider:

“Ten stakeholders need to receive communications on an important change. You and your team of 40 are concerned that everyone needs to receive these changes. What is the formula to calculate communication channels on a project? How many communication channels are there?

Now, I realize that this is an open-ended question rather than multiple-choice, and so it may not be truly representative of what’s on the actual TOO exam. I also don’t know if this person has attended an item-writing session, so I don’t know what expertise they have to construct questions.

But this question does illustrate some of the reported problems with the construction of other project management certification exams. As a result, it also illustrates some of the problems that you will not have with our Level D exam.

First, the questions asks about an important concept: the fact that the number of communication channels goes up exponentially as the size of the project team increases. However, I could be well aware of this fact and still:

  • Not know the formula.
  • Have forgotten the formula.
  • Recall the formula incorrectly.
  • Do the calculation wrong.

So there is a good chance here that I know the concept and still get the answer wrong. By definition, that means this is a bad question.

Second, has anyone ever used this formula on a real project? I’m going to guess that the answer is no. At IPMA-USA, we try to avoid asking questions that are of academic interest only.

Next, the question itself has an excess of information that does not affect the answer. The first two sentences are completely irrelevant and simply slow the examinee down. Change the numbers from 10 and 40 to 3 and 7. Same question. I’m guessing that the item-writer was trying to provide context, but extra text does not equate to context. Context is when I give you information that is relevant to your choice of an answer.

Now let’s look at the question itself:

  • “Ten stakeholders.” Not clear whether these 10 are part of the team or not.
  • “You and your team of 40.” Are you part of the team? Is the team 40 people or 41?
  • Both NYT Style Guide and the Chicago Book of Style counsel against mixing number forms. So if the first two sentences were needed, they should read “10 … 40” or “Ten … forty” to make it easier to absorb the information.
  • “Everyone needs to receive these changes.” Who is “everyone”? The team? The 10 stakeholders? Some other group?
  • Let’s assume that we want to send the information to the 10 stakeholders. That means there areten channels since this is a one-way push, not a discussion. So the formula may be irrelevant. This starts to sound like a trick question.

Our promise to you: No trick questions. No excess verbiage. No useless knowledge.

Your Comments?