Category Archives: Measurement

RuleAsWorded

Measurement – Likert scale

Data were collected by survey with 21 Likert-type items. Each item allowed respondents to rate the frequency of performing specified “green” activities. Environmental experts combined the items into 3 scales (Most Convenient, Most Economical, and Most Environmentally Beneficial). All items use a 5 point scale which has the ordered options: Always, Regularly, Sometimes, Rarely, and Never.

We don’t have access to the survey instrument but I assume each item looked something like:

How often do you remove Roof Racks when not needed?

Always Regularly Sometimes Rarely Never

 

A model for a good 5 point survey scale is a 6 inch ruler. Scale words should be carefully selected to represent the positions of the numbers 1-5.
SixInchRule

Dictionary definitions of the survey options suggest problems with the rating scale. Regularly and Sometimes are out of order. Rarely seems closer to Never than Regularly.
value Survey options Definition (American Heritage® Dictionary) Value (based on Definition)
1 Always At all times, invariably 1
2 Regularly Customary, usual, or normal 3
3 Sometimes Now and then; from time to time; occasionally 2
4 Rarely Not often; infrequently 4.5
5 Never Not ever; on no occasion; at no time 5

The wording violates three Likert item best practices.

Best practice Violation
Symmetric Is Sometimes the midpoint between Always and Never?
Equidistant Is the difference between Always and Regularly the same as the difference between Rarely and Never?
Extremes Some responders shy away from selecting absolutes (Always, Never). Using extremes tends to make the 5 point scale more like a 3 point scale.

Based on the wording, the 6 inch rule used in the survey has:

  • the ends chopped off,
  • 2 and 3 swapped, and
  • 4 close to 5.

It may look something like this:RuleAsWorded

While these problems may not invalidate the data, the poor choice of scale words will add noise to the measurements. This is unfortunate, especially in studies like this with small sample sizes.

 

Measurement – Clarity of of Questions

Survey questions must be unambiguous. Poorly worded questions require the respondents to interpret the meaning. In essence, this has respondents answering different questions. Questions which have high responses, at either end of the scale, should be reviewed. If the extremes indicate problems the rest of the questions should also be reviewed.

HighAlwaysNever
The table below presents three questions with multiple interpretations. Each interpretation would lead to a different response.

Question Presumed intention Alternative reading
Run Appliances When Full Wait until full before starting Once it’s full I always start it
Keep Tires Inflated Maintain recommended psi +- 3 lbs. I don’t drive on flat tires
Refillable Coffee Cups Don’t use disposable cups All cups are refillable