View source for Social:Questionnaire construction

{{short description|Design of a questionnaire to gather statistically useful information about a given topic}}
'''Questionnaire construction''' refers to the design of a questionnaire to gather [[Statistic|statistic]]ally useful information about a given topic. When properly constructed and responsibly administered, questionnaires can provide valuable data about any given subject.

==Questionnaires==
Questionnaires are frequently used in [[Quantitative marketing research|quantitative marketing research]] and [[Social:Social research|social research]]. They are a valuable method of collecting a wide range of information from a large number of individuals, often referred to as respondents.

What is often referred to as "adequate questionnaire construction" is critical to the success of a survey. Inappropriate questions, incorrect ordering of questions, incorrect scaling, or a bad questionnaire format can make the survey results valueless, as they may not accurately reflect the views and opinions of the participants.

Different methods can be useful for checking a questionnaire and making sure it is accurately capturing the intended information. Initial advice may include:
* consulting subject-matter experts
* using questionnaire construction guidelines to inform drafts, such as the Tailored Design Method,<ref name=":0">Dillman, Don A., Smyth, Jolene D., Christian, Leah Melani. 2014. Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method, 4th edition. John Wiley: Hoboken, NJ</ref> or those produced by National Statistical Organisations.

Empirical tests also provide insight into the quality of the questionnaire. This can be done by:
* conducting cognitive interviewing. By asking a sample of potential-respondents about their interpretation of the questions and use of the questionnaire, a researcher can 
* carrying out a small pretest of the questionnaire, using a small subset of target respondents. Results can inform a researcher of errors such as missing questions, or logical and procedural errors.
* estimating the measurement quality of the questions. This can be done for instance using test-retest,<ref>Lord, F. and Novick, M. R.(1968). Statistical theories of mental test scores. Addison – Wesley.</ref> quasi-simplex,<ref>Heise, D. R.(1969). Separating reliability and stability in test-retest correlation. American Sociological Review, 34, 93-101. https://dx.doi.org/10.2307/2092790</ref> or mutlitrait-multimethod models.<ref>Andrews, F. M. (1984). Construct validity and error components of survey measures: a structural modelling approach. Public Opinion Quarterly, 48, 409-442. https://dx.doi.org/10.1086/268840</ref>
* predicting the measurement quality of the question. This can be done using the software Survey Quality Predictor (SQP).<ref>Saris, W. E. and Gallhofer, I. N. (2014). Design, evaluation and analysis of questionnaires for survey research. Second Edition. Hoboken, Wiley.</ref>

==Types of questions==
* Closed-ended questions – Respondents' answers are limited to a fixed set of responses.
** Yes/no questions – The respondent answers with a "yes" or a "no".
** Multiple choice – The respondent has several option from which to choose.
** Scaled questions – Responses are graded on a continuum (e.g.: rate the appearance of the product on a scale from 1 to 10, with 10 being the most preferred appearance). Examples of types of scales include the [[Social:Likert scale|Likert scale]], semantic differential scale, and rank-order scale. (See [[Scale (social sciences)|scale]] for further information)
** Matrix questions – Identical response categories are assigned to multiple questions. The questions are placed one under the other, forming a matrix with response categories along the top and a list of questions down the side. This is an efficient use of page space and the respondents' time.
* [[Social:Open-ended question|Open-ended question]]s – No options or predefined categories are suggested. The respondent supplies their own answer without being constrained by a fixed set of possible responses. Examples include:
** Completely unstructured – For example, "What is your opinion on questionnaires?"
** Word association – Words are presented and the respondent mentions the first word that comes to mind.
** Sentence completion – Respondents complete an incomplete sentence. For example, "The most important consideration in my decision to buy a new house is..."
** Story completion – Respondents complete an incomplete story.
** Picture completion – Respondents fill-in an empty speech balloon.
** [[Social:Thematic apperception test|Thematic apperception test]] – Respondents explain a picture or create a story about what they think is happening in the picture.
* Contingency question – A question that is answered only if the respondent gives a particular response to a previous question. This avoids asking questions of people that do not apply to them (for example, asking men if they have ever been pregnant).

==Multi-item scales==
[[File:Multi-item psychometric scale.jpg|thumb|500px|Labelled example of a multi-item [[Psychometrics|psychometric]] [[Scale (social sciences)|scale]] as used in questionnaires<ref name="dx.doi.org">Robinson, M. A. (2018). Using multi-item psychometric scales for research and practice in human resource management. ''Human Resource Management'', 57(3), 739–750. https://dx.doi.org/10.1002/hrm.21852 (open-access)</ref>]]
Within [[Social:Social science|social science]] research and practice, questionnaires are most frequently used to collect [[Quantitative research|quantitative]] data using multi-item [[Scale (social sciences)|scales]] with the following characteristics:<ref name="dx.doi.org"/>
* Multiple statements or questions (minimum ≥3; usually ≥5) are presented for each [[Dependent and independent variables|variable]] being examined.
* Each statement or question has an accompanying set of equidistant response-points (usually 5-7).
* Each response point has an accompanying verbal anchor (e.g., “strongly agree”) ascending from left to right.
* Verbal anchors should be balanced to reflect equal intervals between response-points.
* Collectively, a set of response-points and accompanying verbal anchors are referred to as a [[Social:Rating scale|rating scale]]. One very frequently-used rating scale is a [[Social:Likert scale|Likert scale]].
* Usually, for clarity and efficiency, a single set of anchors is presented for multiple rating scales in a questionnaire.
* Collectively, a statement or question with an accompanying rating scale is referred to as an item.
* When multiple items measure the same variable in a [[Reliability (statistics)|reliable]] and [[Validity (statistics)|valid]] way, they are collectively referred to as a multi-item scale, or a [[Psychometrics|psychometric]] scale.
* The following types of reliability and validity should be established for a multi-item scale: [[Internal consistency|internal reliability]], [[Repeatability|test-retest reliability]] (if the variable is expected to be stable over time), [[Content validity|content validity]], [[Construct validity|construct validity]], and [[Criterion validity|criterion validity]]. 
* [[Factor analysis]] is used in the scale development process.
* Questionnaires used to collect quantitative data usually comprise several multi-item scales, together with an introductory and concluding section.

==Questionnaire construction issues==

Before constructing a questionnaire survey, it is advisable to consider how the results of the research will be used.  If the results won't influence the decision-making process, budgets won't allow implementing the findings, or the cost of research outweighs its usefulness, then there is little purpose in conducting the research.

The research objective(s) and frame-of-reference should be defined beforehand, including the questionnaire's context of time, budget, manpower, intrusion and privacy. The types of questions (e.g.: closed, multiple-choice, open) should fit the data analysis techniques available and the goals of the survey.

The manner (random or not) and location (sampling frame) for selecting respondents will determine whether the findings will be representative of the larger [[Statistical population|population]].

The level of measurement – known as the [[Scale (social sciences)|scale]], index, or typology – will determine what can be concluded from the data. A yes/no question will only reveal how many of the sample group answered yes or no, lacking the resolution to determine an average response. The nature of the expected responses should be defined and retained for interpretation.

A common method is to "research backwards" in building a questionnaire by first determining the information sought (i.e., Brand A is more/less preferred by ''x%'' of the sample vs. Brand B, and ''y%'' vs. Brand C), then being certain to ask all the needed questions to obtain the [[Indicator (statistics)|metrics]] for the report. Unneeded questions should be avoided, as they are an expense to the researcher and an unwelcome imposition on the respondents. All questions should contribute to the objective(s) of the research.

Topics should fit the respondents' frame of reference, as their background may affect their interpretation of the questions. Respondents should have enough information or expertise to answer the questions truthfully. Writing style should be conversational, yet concise and accurate and appropriate to the target audience and subject matter. The wording should be kept simple, without technical or specialized vocabulary. Ambiguous words, equivocal sentence structures and negatives may cause misunderstanding, possibly invalidating questionnaire results. Double negatives should be reworded as positives.

If a survey question actually contains [[Philosophy:Double-barreled question|more than one issue]], the researcher will not know which one the respondent is answering. Care should be taken to ask one question at a time.

Questions and prepared responses (for multiple-choice) should be neutral as to intended outcome. A biased question or questionnaire encourages respondents to answer one way rather than another.<ref>Timothy R. Graeff, 2005. "Response Bias", ''Encyclopedia of Social Measurement'', pp. [http://www.sciencedirect.com/science/article/pii/B0123693985000372 411]-418. ScienceDirect.</ref> Even questions without bias may leave respondents with expectations. The order or grouping of questions is also relevant; early questions may bias later questions. [[Philosophy:Loaded question|Loaded question]]s evoke emotional responses and may skew results.

The list of prepared responses should be collectively exhaustive; one solution is to use a final write-in category for "other&nbsp;________". The possible responses should also be mutually exclusive, without overlap. Respondents should not find themselves in more than one category, for example in both the "married" category and the "single" category (in such a case there may be need for separate questions on marital status and living situation).

Many people will not answer personal or intimate questions. For this reason, questions about age, income, marital status, etc. are generally placed at the end of the survey. This way, even if the respondent refuses to answer these questions, he/she will have already answered the research questions.

Visual presentation of the questions on the page (or computer screen) and use of white space, colors, pictures, charts, or other graphics may affect respondent's interest – or distract from the questions. There are four primary design elements: words (meaning), numbers (sequencing), symbols (e.g. arrow), and graphics (e.g. text boxes).<ref name=":0" /> In translated questionnaires, the design elements also take into account the writing practice (e.g. Spanish words are lengthier and require more space on the page or on the computer screen) and text orientation (e.g. Arabic is read from right to left) to prevent data missingness.<ref name=":02">{{Cite book |last=Pan |first=Yuling |url=https://www.taylorfrancis.com/books/mono/10.4324/9780429294914/sociolinguistics-survey-translation-yuling-pan-mandy-sha-hyunjoo-park |title=The Sociolinguistics of Survey Translation |last2=Sha |first2=Mandy |date=2019-07-09 |publisher=[[Company:Routledge|Routledge]] |isbn=978-0-429-29491-4 |location=London |doi=10.4324/9780429294914/sociolinguistics-survey-translation-yuling-pan-mandy-sha-hyunjoo-park}}</ref><ref>{{Cite journal |last=Wang |first=Kevin |last2=Sha |first2=Mandy |date=2013-03-01 |title=A Comparison of Results from a Spanish and English Mail Survey: Effects of Instruction Placement on Item Missingness |url=https://surveyinsights.org/?p=1741 |journal=Survey Methods: Insights from the Field (SMIF) |language=en-US |doi=10.13094/SMIF-2013-00006 |issn=2296-4754 |doi-access=free}}</ref>  

Questionnaires can be administered by research staff, by volunteers or self-administered by the respondents. Clear, detailed instructions are needed in either case, matching the needs of each audience

=== Methods of collection ===
There are a number of channels, or modes, that can be used to administer a questionnaire. Each has strengths and weaknesses, and therefore a researcher will generally need to tailor their questionnaire to the modes they will be using. For example, a questionnaire designed to be filled-out on paper may not operate in the same way when administered by telephone. These mode effects may be substantial enough that they threaten the validity of the research.

Using multiple modes can improve access to the population of interest when some members have different access, or have particular preferences.

{| class="wikitable" |
|-
! Method
! Benefits and cautions
|-
|Postal
|
* Usually a simple questionnaire, printed on paper to be filled-out with a pen or pencil.
* Low cost-per-response for small samples. Large samples can often be administered more efficiently by using [[Optical character recognition|optical character recognition]]. 
* Mail is subject to postal delays and errors, which can be substantial when posting to remote areas, or given unpredictable events such as natural disasters.
* Surveys are limited to populations that are contactable by a mail service. 
* Reliant on high levels of literacy
* Allows survey participants to remain anonymous (e.g. using identical paper forms).
* Limited ability to build rapport with the respondent, or to answer questions about the purpose of the research.
|-
|Telephone
|
* Questionnaires can be conducted swiftly, particularly if computer-assisted.
* Opportunity to build rapport with respondents may improve response rates.
* Researchers may be mistaken for being [[Telemarketing|telemarketers]].
* Surveys are limited to populations with a telephone.
* Are more prone to social desirability biases than other modes, so telephone interviews are generally not suitable for sensitive topics.<ref>Frauke Kreuter, Stanley Presser, and Roger Tourangeau, 2008. "Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity", ''Public Opinion Quarterly'',  72(5): 847-865 first published online January 26, 2009 {{doi|10.1093/poq/nfn063}}</ref><ref>Allyson L. Holbrook, Melanie C. Green And Jon A. Krosnick, 2003. "Telephone versus Face-to-Face Interviewing of National Probability Samples with Long Questionnaires: Comparisons of Respondent Satisficing and Social Desirability Response Bias". ''Public Opinion Quarterly'',67(1): 79-125. {{doi|10.1086/346010}}.</ref>
|-
|Electronic
|
* Usually administered via a HTML-based webpage, or other electronic channel such as a smartphone app. 
* This method has a low ongoing-cost, and most surveys cost little for the participants and surveyors. However, initial set-up costs can be high for a customised design due to the effort required in developing the back-end system or programming the questionnaire itself.
* Questionnaires can be conducted swiftly, without postal delays.
* Survey participants can choose to remain anonymous, though risk being tracked through [[HTTP cookie|cookies]], unique links and other technology.
* It is not labour-intensive.
* Questions can be more detailed, as opposed to the limits of paper or telephones.<ref>Respicius, Rwehumbiza (2010)</ref>
* This method works well if the survey contains several branching questions. Help or instructions can be dynamically displayed with the question as needed, and automatic sequencing means the computer can determine the next question, rather than relying on respondents to correctly follow skip instructions.
* Not all of the sample may be able to use the electronic form due to accessibility issues, software compatibility, bandwidth requirements, server load, or internet access, and therefore results may not be representative of the target population.
|-
|'''Personally administered'''
|
* Questions can be more detailed and obtains more comprehensive information. However, respondents are often limited to their working memory: specially designed visual cues (such as prompt cards) may help in some cases.
* Interviewers sometimes rephrase questions during the interview, reducing the level of standardisation. [[Software:Computer-assisted personal interviewing|Computer-assisted personal interviewing]] may assist with this.
* Rapport with respondents is generally higher than other modes.
* Typically higher response-rate than other modes.
* Can be extremely expensive and time-consuming to train and maintain an interviewer panel. Each interview also has a cost associated with collecting the data.
* Relatively few limits to the population, so long as an interviewer is granted access.
|}

===Question wording===

The way that a question is phrased can have a large impact on how a research participant will answer the question.<ref name="Shaughnessy2011">{{cite book|last1=Shaughnessy|first1=J.|last2=Zechmeister|first2=E.|last3=Jeanne|first3= Z.|title=Research methods in psychology|url=https://archive.org/details/researchmethodsp00shau|url-access=limited|edition=9th|year=2011|publisher=McGraw Hill|location=New York, NY|pages=[https://archive.org/details/researchmethodsp00shau/page/n179 161]–175|isbn=9780078035180 }}</ref> Thus, survey researchers must be conscious of their wording when writing survey questions.<ref name="Shaughnessy2011"/> It is important for researchers to keep in mind that different individuals, cultures, and subcultures can interpret certain words and phrases differently from one another.<ref name="Shaughnessy2011"/>

There are two different types of questions that survey researchers use when writing a questionnaire: free-response questions and closed questions.<ref name="Shaughnessy2011"/> Free-response questions are open-ended, whereas closed questions are usually multiple-choice.<ref name="Shaughnessy2011"/> Free-response questions are beneficial because they allow the responder greater flexibility, but they are also very difficult to record and score, requiring extensive coding.<ref name="Shaughnessy2011"/> Contrastingly, closed questions can be scored and coded more easily, but they diminish expressivity and spontaneity of the responder.<ref name="Shaughnessy2011"/>

In general, the vocabulary of a question should be very simple and direct, and preferably under twenty words.<ref name="Shaughnessy2011"/> Each question should be edited for readability and should avoid leading or loaded questions.<ref name="Shaughnessy2011"/> If multiple questions are being used to measure one construct, some of the questions should be worded in the opposite direction to evade response bias.<ref name="Shaughnessy2011"/>

A respondent's answer to an open-ended question can be coded into a response scale afterwards,<ref name="Mellenbergh, G.J. 2008 pp. 183–209">Mellenbergh, G.J. (2008). Chapter 9: Surveys. In H.J. Adèr & G.J. Mellenbergh (Eds.) (with contributions by D.J. Hand), Advising on Research Methods: A consultant's companion (pp. 183–209). Huizen, The Netherlands: Johannes van Kessel Publishing.</ref> or analysed using more qualitative methods.

===Question sequence===
Questions should flow logically, from the general to the specific, from least to most sensitive, from factual and behavioral matters to attitudes and opinions. When semi-automated, they should flow from unaided to aided questions. The researcher should ensure that the answer to a question is not influenced by previous questions.

According to the three-stage theory (also called the sandwich theory), questions should be asked in three stages:{{citation needed|date=May 2017}}
#screening and rapport questions
#product-specific questions
#demographic types of questions

==See also==
{{div col||colwidth=30em}}
* Computer-assisted telephone interviewing
* [[Software:Computer-assisted personal interviewing|Computer-assisted personal interviewing]]
* Automated computer telephone interviewing
* [[Official statistics]]
* Bureau of Labor Statistics
* Questionnaires
* Paid survey
* [[Data mining]]
* [[Company:NIPO Software|NIPO Software]]
* [[Finance:DIY research|DIY research]]
* [[Software:SPSS|SPSS]]
* [[Finance:Marketing|Marketing]]
* [[Finance:Marketing research|Marketing research]]
* [[Scale (social sciences)]]
* Statistical survey
* [[Quantitative marketing research]]
{{end div col}}

==References==
{{Reflist|30em}}

{{Social surveys}}

[[Category:Psychometrics]]
[[Category:Market research]]

{{Sourceattribution|Questionnaire construction}}