Tag Archives: Green Living

HighMissing

Review – Green Living

Introduction.

I started working with Tableau Desktop just after Stephen and Eileen McDaniel gave a presentation to the Seattle Tableau User’s group (March, 2013).  On Tableau’s community site they named Eileen’s Green Living Dashboard as a good example of visual data presentation.  This sounded like a good initial visualization for me to work with.   I started by doing a quick review.  It turned up several problems, which triggered a more detailed review.  Eventually, the quantity and severity of problems suggested that the visualization should never have been created. The dashboard did turn out to be a great example  ̶  one which offers reviewers many opportunities to discover problems.   The dashboard is available at the McDaniel’s site.

Summary of Review

Story

The dashboard presents a model for improving programs which promote green activities. It suggests working with experts to:

  • list important green activities which are appropriate for the locality,
  • develop a survey which measures the frequency of those activities,
  • classify the activities into meaningful categories,
  • administer the survey,
  • and suggest best practices for local sustainability programs based on the results.

The model presents 21 green activities which are pooled into these three categories: save money, easy to do, and beneficial to the environment.  The results found that “easy to do” activities were done most often, those that “save money” next and those “most beneficial to the environment” least often.  These stated outcomes are no surprise and of little interest.  However, the methodology could be quite interesting.  How were the 21 items ranked and placed into the three categories.  Did each item have a save money score in dollars and cents, an easy to do score  in minutes per day, and an environmental score in yearly pounds of carbon saved?  Unfortunately none of this information is available on the site or from the author.  We are left with So What?.

Survey

The survey method (phone, personal interview, mail, or on-line) and the response rate are not reported or available.

The respondents’ education levels are significantly higher than state norms.

The respondents’ distribution by state is significantly different from census data.

Measurement – Likert scale

The five point scale for the Likert items is poorly constructed.

There is no information on what Likert items were assigned to each Likert scale.

Measurement – Clarity of Questions

The wording of the questions is often ambiguous.

Several of the questions do not apply to all respondents.

Missing data

Analysis of missing data suggests that some of the questions were not applicable for some respondents.   There were numerous questions which were left blank (i.e. missing data) but the three pooled categories: save money, easy to do, and beneficial to the environment were calculated for each respondent.  The process used to handle missing values for pooled categories is not specified.

Transformation

The study is based on aggregated data. Unfortunately the aggregation loses all individual data and renders the dataset useless. The original data are not available.

Transparency

The original data and key information about the survey (method, response rate), the questions (what items went to which scales) and how missing values contributed to the three summary categories are not presented on the site.  Attempts to get additional information resulted in an email stating that they were too busy to respond to questions.

Communicate

There are several minor issues on the dashboard.

  • The average response is displayed with different resolution in each segment (the map shows integer values, the chart one decimal place, and table two decimal places).
  • The map and table show “Count of Persons” while the chart shows “Count of Responses”.
  • The dashboard header uses “Green Lives” while the blog and link use “Green Living”.

These issues foreshadow the lack of rigor apparent in all categories reviewed.  The review found enough serious problems to invalidate the data and the methods.  Therefore and extended review of Communication is not required.

Categories used in this review (see General topics and Special topics in Tools>Checklists) : 
Key: Red=Summarized here and detailed at link, Black=This page only, Strike through=Not reviewed
General: Argument, Communicate, Comparisons, Measurement, Research, Review, Statistics, Story, Transparency 
Special: Causality, Missing, Survey, Time-series, Transformations 

GreenLiveDashboard

Story – Green Living

All text presented with the dashboard is duplicated below. It is paraphrased and clarified in the Story section of the review summary.

Green living dashboard — green activities in the daily lives of Americans

GreenLiveDashboard

Which “green” activities are consumers performing in their everyday lives?

Activities that-

  • Save them money?
  • Are easy to do?
  • Are the most beneficial to the environment?
  • Some combination of the three?

Developing an innovative approach, we combined insights from a recent survey of consumer behavior with expert opinion on green activities and found that:

  • Activities that experts rank as the most beneficial for the environment are not always performed frequently by consumers.
  • Economic benefit to the consumer is a stronger predictor of frequently-performed activities than environmental benefit.
  • However, convenience to the consumer is the best predictor of green behavior!
  • Decision-makers for sustainability programs can tailor this method to their particular location by:
    • Compiling a list of green activities specific to their region.
    • Surveying local consumers and experts.
    • Altering which dimensions are included in assessing the importance of various green activities.
  • “Newcomer” communities can maximize the impact of launching their green programs by:
    • Prioritizing activities that are convenient and economical for the consumer.
    • Motivating consumers with educational programs and incentives.
    • Waiting until the environmental program has gotten off the ground before encouraging activities that are low in convenience and economic benefit- unless they can be financially subsidized.
  • “Veteran” communities can prioritize the activities by environmental benefit:
    • Activities that are most convenient can be financially penalized for non-compliance.
    • Less convenient activities can have incentives for performance.
RuleAsWorded

Measurement – Likert scale

Data were collected by survey with 21 Likert-type items. Each item allowed respondents to rate the frequency of performing specified “green” activities. Environmental experts combined the items into 3 scales (Most Convenient, Most Economical, and Most Environmentally Beneficial). All items use a 5 point scale which has the ordered options: Always, Regularly, Sometimes, Rarely, and Never.

We don’t have access to the survey instrument but I assume each item looked something like:

How often do you remove Roof Racks when not needed?

Always Regularly Sometimes Rarely Never

 

A model for a good 5 point survey scale is a 6 inch ruler. Scale words should be carefully selected to represent the positions of the numbers 1-5.
SixInchRule

Dictionary definitions of the survey options suggest problems with the rating scale. Regularly and Sometimes are out of order. Rarely seems closer to Never than Regularly.
value Survey options Definition (American Heritage® Dictionary) Value (based on Definition)
1 Always At all times, invariably 1
2 Regularly Customary, usual, or normal 3
3 Sometimes Now and then; from time to time; occasionally 2
4 Rarely Not often; infrequently 4.5
5 Never Not ever; on no occasion; at no time 5

The wording violates three Likert item best practices.

Best practice Violation
Symmetric Is Sometimes the midpoint between Always and Never?
Equidistant Is the difference between Always and Regularly the same as the difference between Rarely and Never?
Extremes Some responders shy away from selecting absolutes (Always, Never). Using extremes tends to make the 5 point scale more like a 3 point scale.

Based on the wording, the 6 inch rule used in the survey has:

  • the ends chopped off,
  • 2 and 3 swapped, and
  • 4 close to 5.

It may look something like this:RuleAsWorded

While these problems may not invalidate the data, the poor choice of scale words will add noise to the measurements. This is unfortunate, especially in studies like this with small sample sizes.

 

Measurement – Clarity of of Questions

Survey questions must be unambiguous. Poorly worded questions require the respondents to interpret the meaning. In essence, this has respondents answering different questions. Questions which have high responses, at either end of the scale, should be reviewed. If the extremes indicate problems the rest of the questions should also be reviewed.

HighAlwaysNever
The table below presents three questions with multiple interpretations. Each interpretation would lead to a different response.

Question Presumed intention Alternative reading
Run Appliances When Full Wait until full before starting Once it’s full I always start it
Keep Tires Inflated Maintain recommended psi +- 3 lbs. I don’t drive on flat tires
Refillable Coffee Cups Don’t use disposable cups All cups are refillable

Missing Values – Green Living

Missing values

Always look for and analyze causes of missing data. Causes can be related to respondents attributes (if you don’t have a child it is hard to answer a question on baby food) or poorly designed measurement tool (an unintelligible question is difficult to answer).

The five questions with the highest number of missing values are charted below. Read the questions and see if you come up with a potential problem. Then think up a way to test that assumption.

HighMissingPeople may not have a roof rack or a garden (for drought-resistant plants and compost questions) and they may abstain from coffee. If a question does not apply to a respondent the only options are to not answer (i.e. a missing value) or to select Never. The following chart indicates that questions with high counts of missing values also have high Never rankings, suggesting that they may be poor questions.
NeverHighForMissing

This effect is much stronger than the figure indicates. Bad data transformations, discussed below, significantly reduced the number of Never responses.

Missing values effect subsequent analyses. The Likert scales are composites of several Likert items. How are the scales calculated for when some items are missing? Two common options are to only calculate scales for respondents who answered all items in that scale or to fill in the missing values with a representative value (e.g. the average for all folks who did respond or the average of the items in that scale which the respondent answered, or the grand average of all responses, or…). Many of the 21 questions have missing values. The three scales based on those questions (Most Convenient, Most Economical, and Most Environmentally Beneficial) have no missing values. The missing item values were plugged with a representative value when pooled into the three scales. There is no information available on how the plugged values were calculated.

Transformations – Green Living

Data are often transformed before analysis. This can be as simple as normalizing or reshaping the data, or more pervasive such as done here. Each transformation needs validation. The data in this dashboard were collapsed into a single summary row for each unique combination of State, Gender, Education, and Question. Each new row summarizes the original data by saving the number of responses which were pooled together and the mean of the pooled responses.ExtractMess

As shown in the chart right, this process loses individual data. The distortion of the data becomes more pronounced as the group size increases. The left table below presents the aggregated data from Minnesota for the question “Do you use Cloth Napkins/Towels”. Row 7 is a single row representing the 4 Females with a Bachelors degree in Minnesota. The position of this group on the chart is marked with a pointer.  All we know is that there were four of them and that their answers on the cloth napkin question summed to 11 which gives a mean response of 2.75. The right hand  table below shows the seven possible combinations of the responses (numbers 1-5) which sum to 11. Any one of these could be the original data for record 7. Each of the Minnesota Female BAs are now reported as having a response of 2.75 which is not a valid response on the survey form. If the individual data were available and plotted on the chart above every point would be on one of the integer y-axis lines.

Record Gender Education Group size Mean
1 Male High school graduate 3 4.00
2 Male Some college, no degree 1 3.00
3 Male Bachelors degree 2 3.50
4 Male Masters, JD, MD or PhD 1 1.00
5 Female Some college, no degree 1 2.00
6 Female Associates degree 1 1.00
7 Female Bachelors degree 4 2.75
8 Female Masters, JD, MD or PhD 1 2.00
Which row is the original? Valid responses for record 7 in table left
1 1 1 4 5
2 1 2 3 5
3 1 2 4 4
4 1 3 3 4
5 2 2 2 5
6 2 2 3 4
7 2 3 3 3
The transformation clearly invalidates the entire study.