Polling in Iraq and Afghanistan
October 17, 2007
Q&A: Matthew Warshaw, D3 Systems
Remarks by Matthew Warshaw at DC AAPOR
Matthew Warshaw is Senior Research Manager for D3 Systems, which specializes in opinion research in difficult environments. D3 has particular expertise in the Middle East, where since 1988 it has developed an experienced network of field research teams. WorldPublicOpinion.org worked with D3 on its nationwide surveys of Iraq (January and September 2006) and of Afghanistan (December 2005 and November 2006) . Warshaw spoke with WPO Managing Editor Mary Speck.
A lot of people wonder about the validity of polls conducted in these situations. How do you respond to this skepticism about polling in countries like Iraq or Afghanistan. Do these polls have the same credibility as polls in more stable countries?
I think they have the same credibility. I think you have to work harder to make the polls credible in environments like Iraq and Afghanistan. In the US, we can set up a random digit dialing poll. We have all the census information. Everything’s there, you don’t have to do as much footwork. The difference in difficult environments, violent environments, is that you have to do your homework first. You have to spend time building up the field team. You can’t order the survey on Tuesday and expect that it’s going to be back the following week. You have to be ready to plan and work through the details.
But you can do this work. You can train interviewers. You can teach people how to do it. You can monitor their work. You can do quality control. There’s nothing stopping you from setting up a reasonable sample of a country. You have to work with the constraints and I think you also have to be willing to disclose what those constraints were. I think you have to be very open about the constraints, things like when we had to replace sampling points because there was violence or when we had to make decisions about women interviewing women because of the cultural problems. As long as you disclose all of that, and you try to take it into account for design effect on the margin of error, then I think these are as reliable as anything else you see.
What about the people being interviewed? I realize this varies by country but, for example, in Iraq and Afghanistan when someone comes to their door, do you find that they tend to be reluctant? How can you be sure they aren’t giving you the response they feel you want to hear, particularly in countries that have been under authoritarian rule or where the respondents may have reason to fear reprisals?
There’s always the problem of socially acceptable responses, socially expected responses. I think a lot of that comes down to the kind of questions you design. You have to be very cautious about the questions you design. If you can, you should pretest your questions as much as possible to see where you might be adding bias, or pushing people in one direction or another. If you can do that, I think that helps.
But I also think that people in most situations want to tell you their stories. I don’t think that people are so afraid or limited that they can’t tell you. There are certainly environments where this might be a real problem. North Korea would probably be a very hard place to go and interview people face to face and get an honest response. But it might be possible. What we find is that most people want to share their thoughts and opinions.
It’s important to keep in mind that it’s not Matt Warshaw in a suit and a tie knocking on somebody’s door. It’s a local person. It’s someone who’s from that province, someone who’s from the place where the [respondent] lives. They look like them. They speak their language, their dialect. Those things are important so you need to build a local field force so that there are fewer reasons for someone to resist being open.
Does it take longer to do interviews in these situations? Is there more ritual involved to win confidence or to establish rapport?
Absolutely. You knock on the door. There can be a much longer formal introduction. You might have to meet several members of the family first. Of course, you might have to do that anyway just to build the Kish grid* or figure out who has the next birthday. People are going to offer you tea. People are going to talk about other things. In places like Iraq, we find that people aren’t getting out of their homes as much and there’s a real eagerness to converse as well. Sometimes the interviewers have to struggle to keep it on topic and focus on questions.
It can take longer. But you start to develop a rapport with people as it goes on and if that rapport is good then the interviewee opens up and more starts to come out. It depends on the skill of the interviewer. It depends on what’s happening that day. And it depends on how honest that interviewer appears to the other person. Are you really there just to ask them some questions? Can you show them what you’re up to? I think the people in Iraq are becoming more open to public opinion polling, more ready to participate. They want to have their voices heard.
What about the special precautions in Iraq and Afghanistan? Are they very different in terms of security?
Every country is going to have something special when it comes to security. Afghanistan and Iraq are each special cases. In Afghanistan, we’re more worried about what happens to female interviewers. We’re more worried about people going into areas that have a higher concentration of Taliban supporters. In Iraq, we’re more worried about Sunni interviewers getting stopped at a Shia checkpoint or vice versa. Many people in Iraq carry two ID cards: one with a Sunni-sounding name, one with a Shia-sounding name. People are inventive about how to protect themselves.
These aren’t just things that interviewers are doing to get interviews done. These are things that Iraqis are doing to get through their daily routine. The interviewers are no different from people that live there. They have to be able to move and live in their own society.
So, unfortunately, there are a large number of precautions and they are different in every country. We have to do different things in Haiti than we do in Iraq. We do different things in Pakistan than we do in Afghanistan. Interviewers just have to be cautious based on the environment that they’re working in. It’s dangerous work. This kind of polling can be dangerous for the people that do it and I think they’re quite brave for being willing to take on that challenge. But I think a lot of them see the public good in providing this kind of information so that others can see what’s on the mind of the population there.
How do you deal with a sudden change in plans because of violence? Have you had to change the locale at the last minute?
These changes at the last minute happen frequently. Somebody will be headed out to a location and they’ll call back in and say there’s a military operation today and fighting started. We’ve had cell phone calls in Afghanistan where we can hear the gunfire in the background. Sometimes the instruction is: let’s wait a couple of days, see if it moves on, if things calm down in that area. And sometimes the instruction is: it’s not going to happen this field period. Here’s another sampling point in that province and that’s where you should try.
The things I worry more about are the random things that you can’t necessarily protect yourself from: the suicide bombings, the roadside bombs, the random route selection, knocking on a door where you have no idea what’s behind that door. That’s what keeps me up at night. I worry about what’s going to happen to one of these interviewers when they come across the wrong household.
It’s a tough challenge. We’re asking people to do something that’s quite dangerous. They’re making a tough decision to go out and do this kind of work. But they’re doing it of their own volition. We’re not forcing them into dangerous areas. We take their advice when they call back and we can tell there’s a real security problem.
What makes this particularly worthwhile? What makes it worth the huge expense, the tremendous logistical headaches and, of course, the risks?
It’s the third voice in the debate. It’s providing what people in those countries think about that situation. Having some knowledge of what the people in these places think themselves is a very valuable tool when enormous global decisions are going to be made about the future of a country, the direction it’s going to take, whether you’re going to have democracy, elections, invasions, sanctions, etc. These are decisions that could impact not just that country but your own society and other societies. There’s an enormous value in knowing what’s on people’s minds.
* Kish grid: a mechanism for selecting a respondent from a household at random so that the entire sample reflects the makeup of the general population in terms of age, gender, family status, etc.
