In 1954, Darrel Huff wrote the bestseller, How to lie with statistics, in which he outlines the misuse and errors in the interpretation of statistics. In the same vein, Mark Monmonier wrote the book How to lie with maps where he describes how maps only present one of many possible stories about the places it depicts. The most recent book has is about data and has the word ‘lie’ in the title is How charts lie by Alberto Cairo, in which he explains that we are still ill equipped to interpret data visualisations.

All these books explain in a brilliant way how data manupiliation can be used to lies by misusing data analysis. Yet I see it often go wrong before the data is analysing stage. Whether deliberately or unintentionally, many errors are already made during the data collection phase. Due to asking incorrect questions, the results are unreliable in advance. This blog describes the seven most common survey questions mistakes that are consciously or unconsciously made while setting up surveys.

1. Leading questions

Did you enjoy reading this informative blog?

If you enjoyed this blog, would you recommend it to others?

Do you agree that bias in survey questions is a bad thing?

A leading question is a question that suggests the answer or contains information the interviewer is looking to have confirmed. This type of questions are often used intentionally in marketing & sales and in courts to prompt respondents towards providing an already determined answer. Unfortunately, leading questions also frequently appear in survey questions. This is often unintentional but still bad as it makes the outcomes of a survey unreliable. The video below shows brilliantly how leading questions lead to preferenced anwers.

2. Assumptive questions

What is your favorite blog-post on this website?

Is this blog post better or worse than others on this topic?

What search engine did you use to find this blog-post?

Assumptive questions are questions in which assumptions about the interviewee have been made. In the case of the above questions, it is assumed that you already know or did things that might not be true. When drawing up a survey question, you cannot, and should not assume that something is true or plausible in advance. Assumptive questions can make answering questions difficult and confusing. To avoid this question bias, it’s always good to include an “other” option in a survey, especially for personal information such as demographics.

3. Social Desirable questions

A recent poll indicated 74% of respondents appreciates this blog post. How do you rate this blog post?

Do you understand the significance of this blog post?

Do you agree that extra attention is needed for the topic in this blog?

Social desirable questions are questions that are posed in a way that the respondents answers socially desirable. Social desirable questions are often related to taboo topics such as sexual activities, illegal behavior, or unsocial attitudes like racism. But also in the questions above I hope you see how most people would answer socially desirable. It is clear that you will not receive objective and representative answers when you ask social desirable questions in your survey.

4. Vague Questions

How do you rate the experience of this blog based on the readability?

Do you like to UI of this CMS?

Are you interested to see the ROI of this site retrieved from the gtag.js?

A vague question is a question that is broad, undefined, unclear and fails to focus on a specific subject area. If a survey contains vague questions, you can be sure that the results will be unreliable. It is therefore important to make use of phrases and terms that are precise and well thought out in advance, which is something that is not always easy. In addition, you should also avoid the use jargon because probably not everyone will understand it, or because it can be interpreted differently. The video below appropriately demonstrates this in a funny way.

5. Double barreled questions

Do you find this blog-post informative and useful?

How often and how much time do you spend reading blogs?

A double barreled question is a question that consists of one question, but in which two questions are asked. It may surprise you, but it double barreled questions are still commonly used. Fortunately, with double barreled questions it is possible to easily correct them by splitting them up. In the case of the above questions, the following question would be desirable:

• Do you find this blog-post informative? Yes / No
• Do you find this blog-post useful? Yes / No
• How often do you read blogs? <1, 1-3, 3-5, >5 times a week.
• How much time do you spend reading blogs? <10 min 10-15 min, 15-20, 20-25 >25 min.

6. Absolute questions

Do you always read the description of a website in a search engine before opening it?

Why have you never visited this blog before?

Yes or No, are you satisfied with this blog?

Absolutes in questions force respondents into a corner where they can’t give useful feedback. These questions usually have the options Yes/No and include wording such as “always,” “all,” “every,” “ever,” etc. If you guide your respondents to answer that they “always” or “never” do something, the chance of inaccurate answers is higher. People are complex and their behavior is not always consistent. Don’t complicate things by forcing them to take a stand.

7. Unanswerable questions

What did you think of when you started reading this blog?

Is this blog post better than all others on this topic?

Do you think other visitors would recommend reading this blog?

In addition to the above mistakes, it often occurs in surveys that there is simply no answer to the question, or that someone doesn’t remember. The question is then formulated in such a way that no answer can be given. This may be because the question is poorly asked, the type of question is too limited (absolute) or like we have seen with assumptive questions when there is no ‘other’ option.

Conclusion

Looking at the 7 examples of bad survey questions , it is cleat that it’s easy to set up a survey wrong. That’s why it’s vital that you test your survey before sending it out. Additionally to asking the right questions you should also not forgot to select the right sample and use the right type of questions in a survey.