Questionnaire Design — Order Effects and Order Bias

Order Effects

The question sequence must be carefully considered to avoid order effects — biases caused by the sequencing of the questions. Order effects occur when a question earlier in the study has an influence on the response to one or more questions that follow. Take for example, the sequence of questions listed below:

• Please rate the colour of this drink.
• Please rate the smell of this drink.
• Please rate the amount of cocoa in this drink.
• Please rate the amount of sugar in this drink.
• Please rate the size of chocolate chips in this drink.
• What is your overall rating of this drink?

Placing it at the end, a common error, biases the response to the question on “overall rating”. If for instance, respondents have a positive (or a negative) feeling for the attributes colour, smell, amount of cocoa, etc. it will influence their response on overall rating. For this reason, questions on overall rating of product, service or experience should precede questions about the various attributes that influence the overall rating.

We also need to be mindful of halo effects. If the respondent has a positive (or a negative) feeling for the first one or two attributes (colour, smell), this might cause a halo affecting her response to some of the subsequent attributes (amount of cocoa, amount of sugar, etc.). The halo effect may be contained by randomizing the sequence of the questions that appear on attribute rating.

Order effects can also occur when the text of a question reveals previously unknown facts to respondents. For example, the question “Do you use Timotei shampoo?” makes the respondents aware of the brand Timotei. Hence unaided brand awareness questions are placed near the start of the questionnaire, and questions on factual and behavioural information go before questions relating to attitudes and opinions. For instance, in advertising tracking, brand awareness questions precede questions on purchase behaviour and brand image, and these are followed by questions pertaining to advertising awareness and diagnostics.

Order Bias

Besides the sequencing of questions, the sequencing of choices may also generate bias. This is referred to as order bias — it may occur when a respondent is given the opportunity to select answers in multiple choice questions. For instance, given a choice of flavours, colours, ingredients and so on, respondents have the tendency to select items that appear at the top or the bottom of the list. Order bias may be eliminated by randomizing the sequence of the items in the list so that each has the same likelihood of being in any given position in the list.