What Was the Question?

In my last blog entry, Obsessing over Assessment, I recommended choosing question formats that make sense for the level of learning you need to assess. I then went on to discuss the most popular form of test item: the multiple choice question. However, there are other options, and some of them may be more appropriate for use, depending on what you are trying to assess.

Here are five common question types along with a few guidelines as to how and why you would use them.

1. Short Answer Questions - also known as fill-in-the-blank - are best used to assess basic information that learners need to commit to memory. They are helpful for testing terminology, facts and simple computations. Short answer completion items should have only one brief correct answer. Typically, the blank for completing the statement is placed at the end of the test item. Blank spaces for all items should be equal in size, and should not be any larger than necessary.
2. True/False Questions – are second in popularity to multiple choice questions. Like multiple choice questions, they require the participant to select a response. In this case, there are only two options: True or False; Yes or No; Agree or Disagree. They are useful when there is a black and white distinction between two alternatives. There can be no gray areas. Well-written True/False questions are usually stated as declarative sentences that focus on a single idea. A common mistake that test writers make is to put two ideas in one statement, requiring both of them to be true in all cases. Another common error is to tip off the answer by including words such as “always” or “never” in the statement, which usually means it is false. And of course, on the downside test participants always have a 50-50 shot at the answer, so they may be likely to venture a guess.
3. Matching Column Questions - are used to assess content knowledge and associations between ideas. They fit into the category of selection items along with True/False and Multiple Choice. They are constructed by stating the premise for each test item in the first column, and listing options or responses in random order in the second column. Typically, the items in the first column are identified by number, and the response choices in the second column are identified using letters. Some test makers are reluctant to use matching columns because they seem harder to construct than other question types, but they are really very similar in to multiple choice questions. The format of premises and responses is very clear for the test taker, and they are easy to score.
4. Multiple Choice Questions - are everyone's favorite, and rightly so because they are so versatile. They give you the ability to go beyond testing for facts. You can write multiple choice questions to measure learning outcomes that test for knowledge, comprehension, and application and, to a lesser degree, analysis, synthesis and evaluation. A good multiple choice question includes a clear, well-written premise, and a list of reasonable response choices, one of them being the correct answer (which should not always be choice C!). You can also couple them with reading passages, tables or charts that include the correct answers to assess your test takers ability to interpret important information.
5. Essay Questions – are useful for assessing how well test takers can analyze, compare, contrast, evaluate, interpret or integrate ideas. The obvious downsides are that they rely heavily on writing skills and they are a challenge to score. Those issues can be minimized by attending to how the questions are written. If the essay question is too open-ended, it leaves it up to the test taker to decide on which direction to go. Instead write questions that ask specific questions. The person participating in the assessment must be able to clearly identify what it is you want to know from them. Give them parameters, by using phrases such as, “provide five ideas on how you would…” or “give three reasons why you would…” You can also provide direction by asking the test taker to “consider the following factors…” when writing a response. Consider identifying word count minimums and maximums so the person will be able to gauge the level of detail expected. Also, for every question you write, make sure you create a model answer to aid scoring.

For more detailed information, check out Tests & Measurement for People Who (Think They) Hate Tests & Measurement by Neil J. Salkind. Part III of his book is called The Tao and How of Testing. It is a great resource for anyone who needs to construct assessments.

Obsessing over Assessment

The law of attraction is at it again. Testing and assessment is running through my mind and it seems to be running through my life as well. There is a certification project currently being implemented in my company that involves training and testing. I am taking a course on Inquiry and Measurement that also involves testing. And, during this week’s #lrnchat, I was drawn in by a discussion thread in which the pros and cons (well, mostly cons) of assessment were being discussed. Why am I so preoccupied with Level 2 of Kirkpatrick’s Four Levels of Evaluation? It is mostly because of the certification project at work.

Certification is a term that gets thrown around a little too loosely in training departments these days. Put someone through a course and a post-test and BANG, you are certified – or worse, you are not. It is a tricky thing to put together a certification process that is valid and reliable. Very few companies can stand to wait out validation process, so they jump right in and begin training and “certifying” people.

Certifications do not guarantee that the person being certified has learned more than he or she would in a regular training program, but business leaders often feel it is operationally necessary to validate a level of knowledge or skill required to meet goals and targets. If certification is needed or required, here are some things to consider for the assessment process:

Test items should be directly related to learning objectives, which should be directly derived from performance requirements. This may seem obvious but I have seen many tests that have included filler material alongside valid questions.

Test only on important items, not obscure ones. It is not necessary to test someone on small details unless they are critical. Very few corporate employees are doing life-saving work that needs to be tested at a granular level.

Test items should be straight-forward. Don’t try to be tricky. What is the point? It only serves to confuse the learner and adds no value to the assessment.

Choose question formats that make sense for the level of learning you need to assess.  Multiple choice questions are commonly used on knowledge tests because they are easy to score and easy to tie to outcomes – but they are not always easy to write. Good multiple choice questions will have a clear premise in the stem of the question, a correct answer, and reasonable alternative choices. There should not be any throw away responses or convoluted choices such as “a and b, but not c” or “a and c” only. If your test item has more than one correct answer, then rethink the question format. Consider short answer questions or a matching column.

If possible, use randomized test questions. Most learning management systems have this capability. They allow you to create a bank of questions that can be drawn upon at random so that test-takers will be deterred from sharing answers. But don’t make the bank of test questions so large that everyone feels like they are taking a completely different test.

Pilot your test. This is the hard part, because it takes time and patience. You need to let a few people complete the learning experience and take the test to give you the opportunity to analyze the questions. You will want to take a second look at questions that everyone got right, or everyone got wrong, or questions for which many people chose the same incorrect answer.

Create rubrics for skills assessments.  Skills assessment usually requires direct observation.  It is important that all of your assessors are using the same criteria and weights when judging performance.  Validate the process by having multiple assessors review the same performance.  If they are more than a few points off from each other, either redesign the rubric or re-train your assessors.