“Evidence-based practices”, “outcome-driven”, and “measuring impact” are terms that are used everywhere in the field. The reliance on data-driven decision-making and trend analysis has led many in the nonprofit field to implement surveys as part of their evaluation toolkit. However, if they are not designed adequately with clear objectives, they can often result in misleading outcomes, unusable information, or to ethics violations. Creating a survey requires meticulous research, design, and analysis to ensure good measurements are created that facilitate standardization and reliable results.

2 Golden Rules of Survey Design

Good design will ensure that differences in responses are attributable to varied opinions rather than other factors like survey administration, terminology, or confusion about what is being asked. Thus, we are going to spend a little time focusing on how to design an impactful survey. Below are 2 Golden Rules of Survey Design.

1. Questions should be designed to ensure all respondents can understand what is being asked in the same way.

This may seem obvious, but it is worth mentioning that you should be clear about what you want people to answer, so they can answer adequately. This means that as you design your questions, you should plan for the following:

• Make sure that everyone comprehends the terminology that you are using. Industry buzzwords, colloquialisms, jargon, or trends must be replaced with language that will make it possible for every person to answer the question with clarity.
• Avoid using the term “not” in your questions because it will confuse the readers. For example, “In your opinion, health care providers that require patients to provide evidence of their insurance coverage before providing services should not be allowed to refuse care?” This question is very confusing and poorly written. The sentence should be shortened and simplified.
• You should be specific about what you are asking, including the time frame you are referring to, defining ambiguous terms, and explaining concepts.
• You should shorten and simplify sentences to avoid any confusion. Long questions should be broken up into multiple questions.
• Ensure that respondents have direct experience with the topic you are asking about. 

2. Pay attention to every word in your question because it will influence the outcome of your survey results. 

This may also seem obvious, but it is important to develop good measurements that facilitate standardization and reliable results. In order to obtain results you can be confident in, it is important to consider the following in the development of your survey:

• Never assume that people interpret terms that may seem obvious to you in the same way ( e.g., age, income, race, gender, height, neighborhood, etc.). These terms need context and explanations. This link to this video elaborates further on this.
• Leading questions like the following should be avoided, “Do you agree that…”, and “Would you agree that only people who are concerned about their health are likely to buy health insurance?” A good and non-leading question does not attach judgment to a decision or suggest a correct answer. Thus, instead of “Are bad parents responsible for the rise in video game use by lazy youth?”, you would ask, “Please identify the top 3 factors that have contributed to increased video game use in the past 5 years.” The first question passes a judgment on both the parents in the question but may lead the respondent to answer in a way that may be contrary to what they really believe to avoid judgment from you. 
• Avoid questions that both make assumptions about the respondents and then force them to make a choice about what you believe to be true. For example, “What hospital in Washington, D.C. was your child born in?” There are multiple assumptions here: 1) the person has a child, 2) the baby was born in Washington, D.C., and 3) the baby was delivered in a hospital setting. This is called a loaded question. Instead, use preliminary questions to determine whether this question applies to the respondent. Then, you would use skip logic to move to the next question that applied to the respondent. In a paper survey, it would say, “Skip to question…” In a survey that uses software, you would select a skip logic function.
• Avoid straight lining because it is easy to go through many questions of a survey and select the same answer every time without reading the options, particularly if the response options are identical.
• Avoid questions that require respondents to do mathematical calculations, remember a sequence of events, conduct estimates, or engage in complex interactions. This may lead many to drop out of the survey early or to answer in a manner that they think will make you think better of them. Therefore, producing misleading results. 
• Ensure that you are only including one variable in each question you ask. Thus, a sentence that asks the following is bad practice, “How would you rate the quality of our services and support?” This is a double-barreled question and can be confusing for both the evaluator and the respondent. This can be fixed by either choosing one variable to measure in the question or by breaking the question up into 2 separate ones.

Conclusion

Surveys can be incredibly useful in gauging people's interpretations of facts, experiences, preferences, situations, or events. However, they are not factual representations of something that has occurred. Therefore, as you interpret your survey results, always remember there are limitations to the reach of surveys because they tap into people's points of view, rather than facts.

If You Haven’t Already, Please Subscribe to Third Sector News

Please don't forget to share this article with your peers and subscribe to Third Sector News to stay in the loop on all of the exciting new programs we are offering our latest research, and tools that only subscribers have access to that will help further your mission!