The UW System is running an online survey this month to begin the second phase of its strategic planning process. Serious concerns have been raised about the design and implementation of the survey. As UWM Professor Emerita Nancy A. Mathiowetz explains in a letter to UWM Chancellor Mark Mone (reproduced below), the survey suffers from major flaws that make it impossible to draw reasonable conclusions from its results.

With Professor Mathiowetz’s permission, UWM AAUP is publishing her letter here in order to bring the survey’s problems to the attention of the Wisconsin public. We hope that the UW System will interpret the results of the current survey in the light of the objections raised here, and will adhere to the best practices outlined by Professor Mathiowetz in any future surveys it may conduct.

—————–

Dear Chancellor Mone,

I am writing to you today as a former member of the UWM faculty and as an expert in survey methodology to express my concern with the UW System’s online survey.

The survey suffers from two critical design flaws that render the data from the survey at best uninformative and at worst, completely misleading.  These design flaws include (1) a self-selected sample and (2) a poorly designed survey instrument.

The fundamental tenet that underlies statistical inference—that is, the ability to draw inferences from a sample survey to the full population—is the selection of a scientific sample.  With the UW System sample, there is no sample—just an open invitation for anyone to respond and to respond as often as he or she would like.  One cannot use the findings from this survey to project to any known population.  This is not my viewpoint alone—I quote below from the American Association for Public Opinion Research and from the Pew Research Center:

When we draw a sample at random—that is, when every member of the target population has a known probability of being selected—we can use the sample to make projective, quantitative estimates about the population. A sample selected at random has known mathematical properties that allow for the computation of sampling error. Surveys based on self-selected volunteers do not have that sort of known relationship to the target population and are subject to unknown, non-measurable biases. (AAPOR)

A non-probability sample is one in which it is impossible to determine the chance that any individual in the population was selected. Lacking this information, we are uncertain as to how well the sample represents the population, and thus how important a given finding based on such a sample actually is. (Pew)

The design of the UW System “survey” is even more egregious than one based simply on non-probability based sampling: the survey allows a user to input information multiple times.  And, due to the nature of an online survey, those without access to the Internet will be unable to participate in voicing their opinions.  The resulting data cannot be used to make generalizations about the views of citizens of the state of Wisconsin regarding priorities for the University.

In addition, the questions themselves are ill conceived to inform public policy makers.  Each of the five topic areas asks multiple questions related to that topic and for which the respondent is asked to rate the statement/idea on a five point scale, from “Not at all important” to “Extremely important.”  A rating scale is useless for setting policy priorities, since the respondent is not asked to make the difficult choice among priorities.  And in fact, the instructions to the respondent indicate that the goal is to determine which of the statements is most important to the respondent, but the actual survey task simply asks for ratings, not a ranking of most to least important.  A ranking approach to measurement would have required respondents to make the difficult decisions that are similar to those of policy makers—in a world of limited resources, what is the most important priority? And the redundancy of the task—29 questions, all using the same 5-point scale—does not motivate the respondent to think carefully about each of the items.

I note that the individual items are poorly worded; the language is vague and the respondent is not faced with real-world choices.  Consider for example the item “Train a highly skilled workforce.” How does this translate into an actionable item by the UW System?  The statement is sufficiently vague that interpretation is left up to the respondent and to the analyst of these data.  And how will the data be interpreted?  What does it mean to say that a respondent rated “Attract and retain top talent” as extremely important?  At what cost?  And talent with respect to teaching?  Research grants?  And let’s think about the reverse finding—what would it mean if 75% of the respondents rate this item as “not at all important”?

In addition to concerns with the design of the study, I urge you and the other administrators on the UW System campuses to request additional information concerning this study.  Who drafted the questionnaire?  Who will be responsible for the analysis of the data and dissemination of findings?  Will data be made available to the public for analysis?  What research goals will the survey data address?  What Institutional Review Board approved the design of this study?  I saw nothing on the web site or anything on the first page of the questionnaire that provided information with respect to contact information for the study director or the IRB office. No data collection effort supported by UWM would be allowed to be fielded without IRB approval and contact information for the study director.

The American Association for Public Opinion Research (AAPOR) lists the following standards for disclosure:

In accordance with minimum disclosure requirements of the AAPOR Code of Professional Ethics and Practice, every survey researcher should disclose each of the following elements in any report that is for public release, or be prepared to disclose this information promptly:

• Name of the survey sponsor
• Name of the organization that conducted the survey
• The exact wording of the questions being released
• A definition of the population under study.  What population is the survey designed to represent?
• A description of the sampling frame used to represent this population
• An explanation of how the respondents to the survey were selected
• The total sample size
• The method or mode of data collection
• The dates and location of data collection
• Estimates of sampling error, if appropriate
• A description of how the data were weighted (or a statement that they were not weighted), and any estimating procedures used to produce the final results
• If the survey reports findings based on parts of the sample rather than the total sample, then the size of the subgroups reported should be disclosed

I strongly urge you to voice your concerns with the content and sample design for this study.  The findings could be more harmful than helpful; the data will only be representative of the views of a self-selected, motivated segment of the population, responding to an ill conceived and poorly worded questionnaire.  If the UW System wishes to collect data from the citizens of the state, there are at least two well qualified academic survey research centers who could design and implement a study that would meet the standards of the survey research profession.

Please let me know if I can offer further assistance.

Warm regards,

Nancy A. Mathiowetz

Professor Emerita
2015 AAPOR Award Winner for Exceptionally Distinguished Service
2007-2008 AAPOR President
Fellow, American Statistical Association