A Challenge for Psychologists: How to Collect Sensitive Information in Field Experiments
A Challenge for Psychologists:
How to Collect Sensitive Information in Field Experiments
An exciting trend in psychology is the increased use of field experiments to complement more traditional laboratory-based methods. Field experiments allow psychologists to test whether our theories and findings from the lab apply to attitudes and behaviors in the world at large.
As we move our work into field settings, psychologists face a whole new set of methodological and ethical challenges. One important challenge for psychologists in particular to address is how to ensure that study participants report accurately on sensitive information.
This is an issue that psychologists have paid close attention to when it comes to measuring prejudice. However, when we take our research to the field, there are a number of other types of sensitive attitudes and behaviors that we may want to measure. For example, researchers may be interested in measuring peer-to-peer harassment in school settings, intimate partner violence in households, or engagement in antisocial behaviors in postwar societies. These are all outcomes that have come up in field studies by members of my lab at Princeton University.
In my own research, I examine the relationship between perpetrating violence on behalf of a group and identification with the group. Part of this work involves asking former combatants in Liberia to report on different types of violence they perpetrated during the country’s civil war. As you can imagine, this information is quite sensitive, and involves thinking carefully about how to make respondents feel comfortable during the interview.
Types of Sensitive Information
There are different types of sensitive questions, which I group into four categories: antisocial, private, dangerous, and illegal. Antisocial questions involve information that is counternormative but that will not put the respondent at legal or physical risk (e.g., drug and alcohol use, support for violent groups). Private questions ask individuals to report on information that they do not generally feel comfortable sharing with other people, such as information about sexual practices. Dangerous questions are those that could put the respondent (or the interviewer) at physical risk. For example, asking respondents to report on their attitudes toward the government in a repressive authoritarian state could put respondents with negative attitudes in physical danger if the data got into the wrong hands. Similarly, illegal questions are those that involve a real risk of getting particular respondents into legal trouble.
There are a number of existing strategies for asking sensitive survey questions in field experiments. As described by Blair (2015), these strategies fall into the categories of survey administration protections and survey experimental methods. Survey administration protections include strategies such as self-administration of the survey when possible (which is often not the case in developing country contexts), making sure the interviewer and the respondent share important demographic characteristics or past experiences, and emphasizing that identifying information will not be connected to sensitive data.
Survey experimental methods include the randomized response technique, list experiments, and endorsement experiments (see Blair (2015) for an overview of each method). The advantage of these methods is that they protect respondents by not soliciting exact answers to sensitive questions. In other words, in most instances you are not able to determine any particular individual’s exact attitude or behavior simply by looking at their survey data. Additionally, if researchers are interested in measuring sensitive implicit attitudes, then endorsement experiments may be a great tool to use in the field. The disadvantages of these methods are that they necessitate a large sample size, require intensive fieldwork to properly design the questions, and involve assumptions that are often not testable.
A recent additional strategy developed by Blattman and colleagues verifies sensitive data collected in quantitative surveys through in-depth qualitative follow-up with a small subset of participants (Blattman, Jamison, Koroknay-Palicz, Rodrigues, & Sheridan, 2015). This verification exercise allows researchers to identify the direction and magnitude of measurement error on sensitive items in the quantitative survey. While this technique is promising, it requires additional time and resources that many researchers may not be able to spare in their field experiments.
A Role for Psychology
What is the role for psychologists in the development of sensitive survey techniques? While existing methods are ideal for asking about dangerous or illegal information, psychologists have an important role to play in developing methods for asking about antisocial and private information. For these types of sensitive questions, I believe the ideal solution is to focus on the wording of instructions and questions, and on building trust between the interviewer and the respondent during the interview session.
Psychologists focus on the importance of situational factors in our experimental research, and commonly examine how small manipulations can influence outcomes. This often includes manipulating the content of instructions to study participants, subtly manipulating question wording, and manipulating interactions between researchers (experimenters and confederates) and participants. Thus, psychologists are well positioned to provide recommendations and conduct additional research on how to design sensitive survey questions and increase trust between the interviewer and respondent.
Question Wording: An Example
Here is an example of how we can use psychology to inform the development of sensitive questions. Research in psychology tells us that social norms can influence attitudes and behavior. It follows that providing normative information in a survey question – even subtly – may change how individuals respond to the question.
Consider the following survey question that may come up in my area of research: “Were you ever forced to harm a civilian?” Researchers may be tempted to use this wording because it takes the burden of responsibility for engaging in an antisocial behavior off of the respondent. By this logic, respondents may feel more comfortable answering yes because they do not have to admit to willfully engaging in an antisocial behavior. However, this question wording subtly conveys normative information; in this case, it suggests that the researchers believe harming civilians is an undesirable behavior, thus making it less likely that respondents will report engaging in the behavior. Instead, consider wording the question in the following way: “How often were you forced to harm a civilian?” This wording subtly implies that the researchers believe harming a civilian may have been somewhat typical in the respondent’s environment, in turn making the respondent feel more comfortable reporting that they engaged in the behavior.
Increasing Trust: An Example
In my current research in Liberia (with Elizabeth Levy Paluck), we are thinking carefully about how to create a positive rapport between the interviewer and the respondent. When a respondent trusts their interviewer, they are more likely to provide accurate answers to sensitive questions. Additionally, they are more likely to have a good experience during the interview, which is important from an ethical standpoint when the research involves asking about unpleasant past experiences.
One new technique that we are currently piloting in the field is having the interviewer teach the respondent a short breathing exercise at the beginning of the interview session. Once they teach the exercise, the interviewer makes eye contact with the respondent and they breathe together for thirty seconds. The idea is that engaging in this type of activity together helps to create a positive bond at the start of the interview, and creates a more calm and comfortable atmosphere throughout the interview. The breathing exercise can also be repeated at strategic points throughout the interview.
Challenge for Future Research
These are just two examples of techniques that may increase accuracy in the reporting of sensitive questions, and we will need to conduct more research to see whether they actually have their intended effect. As more psychologists tackle this challenge, the hope is that we can develop a set of tools for asking sensitive survey questions that are low cost, easy to implement, and create a positive survey experience for the interviewer and the respondent.
Rebecca Littman is a PhD candidate at Princeton University working with Professor Betsy Levy Paluck. For more on Rebecca, please visit http://www.rebeccalittman.com