What are the problems with questionnaires?

3 INTERVIEW 3.1 STRENGTHS

3.2 LIMITATIONS

4 CASE STUDY 4.1 STRENGTHS

4.2 LIMITATIONS

5 CONCLUSION

6 BIBLIOGRAPHY

1 Introduction

At the beginning of any type of research, it is important for the researcher to determine the most appropriate methodology to carry out the study. The Dictionary of Sociology (1998) defines methodology as a “the methods and general approach to empirical research of a particular discipline”. It is implied that various methods exist to approach a particular research problem, and the researcher should give his or her own set of methods considerable thought.

While factors such as time and costs certainly play an important part in deciding how to approach a particular research problem, the subject of the research itself should ultimately determine the methods used. A good researcher will evaluate all available options prior to making a decision as to which methods to adapt in the light of being the most useful for the study at hand.

Scandura and Williams (2002) note that the impact of management studies often depends ‘upon the appropriateness of the research methods chosen’. This further highlights the importance the researcher needs to place at the selection of the right approach if the end-result is expected to be valuable and meaningful from a management perspective.

When the most appropriate research method - or a mix of various methods - has been established, it is time to start what Gilbert (2001) calls detective work: “Social research involves detective work. You begin with a problem and then ask a number of questions about it, such as ‘what?’, ‘who?’, ‘where?’, ‘when?’, ‘how?’ and ‘why?’”.

A variety of quantitative and qualitative research methods are available to the researcher, ranging from interviews, questionnaires, observation, experiments, to case studies. This paper will focus on three of the above methods, namely questionnaires, interviews, and case studies, in chapters two, three, and four respectively. Each chapter will give a brief introduction about the method, and then highlight the main strengths and limitations of each approach. For the purpose of this paper, induction is defined as a data-driven approach to research, while deduction is seen as theory-driven.[1] As such, the notion of data-driven versus theory-driven will also be explored briefly for each method. Lastly, chapter five will offer a conclusion and the main findings of this paper.

2 Questionnaire

A questionnaire is essentially a structured technique for collecting primary data. It is generally a series of written questions for which the respondents has to provide the answers (Bell 1999). While authors such as Kervin (1999) offer a very narrow definition of questionnaires (whereby the person answering the questions actually records his or her own answers), deVaus (1996) sees a questionnaire in a much wider context (namely as a technique in which various persons are asked to answer the same set of questions).

Care has to be taken in creating a questionnaire; Oppenheim (1996) comments that ‘the ability to write plain English will help’, but that that will not be sufficient. If a questionnaire is well designed, it will motivate the respondents to give accurate and complete information; as such, it should provide reliable and relevant data in return.

A questionnaire can serve as an inductive method with the aim to formulate new theory, where open-ended questions are used to ‘explore a substantive area’ (Gill & Johnson 2001). However, other researchers may use a questionnaire as a means of collecting reliable data in a rather deductive approach in order to test existing theory. As will also be shown in the case of the other two research methods discussed in this paper, a questionnaire thus allows the researcher to adopt either an inductive or a deductive approach, or even a combination of these two.

When developing the actual questionnaire, QuickMBA (Questionnaire Design 2002) explains the three major question-types available to the researcher, namely:

- Open-ended: They have the advantage of offering a wide range of responses that help to capture the flavour of people’s answers, while not influencing the outcome of the question by pre-determining possible responses. On the down-side, answers are often difficult to evaluate and tend to vary in clarity and depth; - Dichotomous (closed-ended): They are used for questions with two possible opposing outcomes, for example ‘Yes’ and ‘No’. They tend to be easier to answer and require less effort when interpreting the results - they are directly comparable to answers by other respondents; and

- Multichotomous (closed-ended): Questions of this type offer a range of possible answers, similar to a multiple-choice test. Again, they tend to be easier on the respondent and equally on the questionnaire-interpreter later on.

2.1 Strengths

If administered properly, questionnaires can prove to be an excellent method to obtain quantitative data about people’s attitudes, values, experiences, and past behaviour (Bell 1999).

Questionnaires allow the researcher to gather a significant amount of data at relatively little cost. Questionnaire distributed by post can be posted to the target group, and the latter can choose to answer whenever it is most convenient for them (Gilbert 2001). Email acts as another delivery channel, and can reduce costs even further.

Of the two main types of questionnaires, descriptive and explanatory, questionnaires allow the researcher to gather data either to explain different phenomena or to explain cause-and-effect relationships between different variables respectively (Gilbert 2001).

2.2 Limitations

One of the main drawbacks of questionnaires, especially ones served through the post or electronic mail, is the often low response rate (Bell 1999). Gilbert (2001) reports that response rates for postal questionnaires can be as low as 20%. Incorrectly or illegibly filled out questionnaires, or even missing answers, will inevitably influence the quality of the data obtained, and have the potential to further lower the number of useable questionnaires.

Questionnaires do not offer the researcher the opportunity to follow up ideas and clarify issues - one of the main strengths of interviews, as chapter three will show. Consideration needs to be given to all aspects of the questionnaire, from design to selecting the correct target-group, in order to obtain the maximum amount of reliable and valuable data (Bell 1999).

Saunders et. al. (2001) notes that questionnaires are not particularly suited for research that requires a significant amount of open-ended questions, for which a (semi-structured) interview might be a better method to pursue.

Saunders et. al. (2001) also describe the limitations of questionnaires with regards to the expected outcome, which might for example highlight trends or attitudes, but will fail to explain the underlying reasons for the outcome. A ‘multi-method’ approach, whereas the researcher combines questionnaires with, for instance, interviews to explain the results, is therefore proposed.

Another limitation of questionnaires can be found in the structure of the method itself. Fixed-choice questionnaires generally assume an un-stated general knowledge of the topic being investigated, and force the respondent to answer questions that he or she might be ignorant of, have a different understanding of based on personal perception, or which are influenced by exogenous factors such as education, culture, age, or societal status (Hyman 1955). A questionnaire has no means of correcting this; the outcome might thus be slightly biased at best, or plainly misleading.

[...]

[1] It is understood that the terms induction and deduction imply a significantly more than simply data-driven or theory-driven approaches to research. A deeper analysis of the above, however, was not seen as appropriate for inclusion in this paper, which deals primarily with the strenths and limitations of the mentioned research methods. For a more thorough discussion of induction and deduction, see Gill and Johnson (2002), chapter three.

Read the ebook

Reflecting back on my legal studies, I often equate survey question development to direct examination and cross-examination of witnesses during a trial. Questions in a courtroom cannot be overly prejudicial to either side, so as to force an answer from a witness or prejudice the objective jury. Therefore, it’s not a stretch to say that developing questions for surveys is much like preparing questions for a trial.

In the courtroom, the ultimate goal of asking questions is to get to the facts and allow the jury to use those facts to present a course of action. In business, the ultimate goal of asking questions is to ensure that respondents’ answers to survey questions are actionable and unbiased.

Overcoming Survey Design Pitfalls

Thanks to advancements in technology, just about anyone can design and issue a survey. Before you help develop your organization’s next survey, consider these ten common problems that can sabotage your efforts:

1. Demographics questions:

Too many surveys start with a series of demographic questions (name, title, address, phone number, email, etc.) that are often unnecessary when surveying current customers. Existing customers will find these questions repetitive and will likely wonder why you don’t have the information already. Besides maintaining a well-organized contact list, look for survey tools that allow for demographic information to be populated. This helps to reduce respondent annoyance and improve your response rate.

2. Inadequate response options:

Make sure to include response ranges that cover all scenarios for the respondents, but make sure the ranges don’t overlap to avoid confusion. For example, one common mistake occurs when providing date range options – be sure to include “less than one year” and an option that covers “X plus years.” Creating a situation in which a respondent isn’t sure what to select can create confusion and result in erroneous response data.

3. Rating level inconsistencies:

When you are asking a number of questions based on a similar rating scale, it is easy to overlook rating level inconsistencies. While several scales could be utilized throughout a survey, be sure that the answer rating – whether it’s 1 to 5 or 0 to 10 – flows consistently on similar type questions throughout the survey. Also, be sure the scales are reflected the same way (i.e. left to right) when offering the same choices.

4. Assuming prior knowledge or understanding:

Do not assume respondents know more than they do about your organization. Examine the question to ensure all the “building blocks” have been established to show that the respondent can knowledgeably respond to the question. Your survey should leave no room for ambiguity or incorrectly rely on presumed prior knowledge from respondents. An example of this would be utilizing acronyms or industry jargon that may not be readily known by all your customers, or could be easily confused, leading to inconsistencies in the data.

5. Leading questions:

For example, “We have recently upgraded our product to become a first-class tool. What are your thoughts on the first-class product?” These questions supply the facts or suggest the answer in the wording of the question. The question itself can “lead” respondents to a particular response. This is often unintentional and is a common mistake when a survey is designed by someone who is too closely associated with the project.

6. Double-barreled or compound questions:

When you review each survey question ask yourself if the question contains more than one “question”. The words “and” and “or” represent possible tell-tail signs of a compound survey question.

7. Question is ambiguous or unintelligible:

A common example arises from survey questions containing “negative” wording, which can easily confuse respondents. Such questions usually include the word “not” and ask respondents to disagree or agree with the statement or position. This is not to be confused with answer choices, which often include the terms “disagree” or “agree” in surveys.

8. Unnecessary questions:

Ask yourself, “What am I looking for that is actionable?” Don’t try to incorporate everything plus the kitchen sink in your survey design. For example, I once saw a survey question that asked: “When you looked at yourself in the mirror today, what was the first thing you thought?” After reading this question, I decided the survey wasn’t worth my time.

9. Excessive open-ended questions:

Assume that you ask ten open-ended questions and receive 100 responses. On average, each respondent writes two sentences per question, resulting in two thousand sentences to be reviewed following the survey. An average written document contains 16 sentences per page, which means you will have to review 125 pages worth of open-ended feedback. Now think about the respondents –many respondents will not write out responses to more than just a few open-ended questions. Keeping your open-ended questions to three or four is generally sufficient. So, choose wisely and use open-ended questions thoughtfully.

10. Lengthy surveys:

Time is money; ask only what you need to know today. Surveys that are too long, too complex, and too confusing will frustrate respondents, resulting in skipped questions or drop-offs. Limit your questions to subjects that you can act upon within the next six months. Demonstrate that you respect the time and effort respondents spend answering your survey. Save more strategic-oriented questions for phone outreach or Customer Advisory Board meetings with your highest priority customers.

Remember that just because technology allows you to create a survey, the ease with which surveys can be issued should not undermine the importance and quality of your survey questions. Satrix Solutions can help you to develop a survey that produces quality, actionable results by avoiding these common mistakes. Contact us today.