Remarks of Matthew Warshaw at DC AAPOR
October 17, 2007
Matthew Warshaw, Senior Research Manager of D3 Systems, spoke about doing survey research under adverse conditions at a seminar held by the DC chapter of the American Association for Public Opinion Research (AAPOR) on Oct. 9, 2007. The following is a partial transcript of his presentation.
Why do this in the first place?
I think it’s very important to do this because these are populations that for the most part people don’t want to talk about. We want to prescribe what should happen in these places, not necessarily hear what people living in these situations have to say about their own situation. If you can get in and do polling, you add a kind of third voice to what’s being discussed. It adds another dimension to the debate, another part of the information picture that you wouldn’t see otherwise. It’s an important aspect of trying to find solutions to some of these problems. And despite the difficulties—no census data and other things—I think it’s worth trying to do this.
Respecting Cultural Values
In Afghanistan I have two separate field teams. I have a male field team and I have a female field team. Because I can’t have men interview women and I can’t have women interview men. So I have to have two completely separate field forces. Whereas I might get away with half the number of people I need in another environment, I have to double the size of my operation in Afghanistan.
In different countries, people have different ideas about how work should be done, what an average work day should be, what’s appropriate for going out and doing interviewing and what kinds of questions are sensitive. You would think this would be political questions but sometimes just asking people’s age can be sensitive.
In Port au Prince, we run into the problem that people don’t want to tell us their age because they’re afraid we’re going to use that information against them in a voodoo ritual. If you know someone’s age you can put a curse or a hex, you can do something to them. So people are nervous about giving out their age.
In Afghanistan, we can’t ask male members of the household certain things about women under the age of 17. It’s considered extremely impolite to ask anything about some of the young women that live in the household. They’re supposed to live completely separately. So a male interviewer talking to a male member of the household is walking on difficult ground if he wants to speak to someone about what’s happening to the school age girls in the family, what kind of problems they may have, other issues like that.
Logistical Challenges in Afghanistan
The last semi-census in Afghanistan was in 1979. I would think things have changed tremendously, especially with five million people going to Pakistan and coming back, and wars. So you have to do your best estimates and then start to build your own data base. In Afghanistan, we’re over the 100,000 interview mark at this point and we’re collecting information about the number of people in each household. At some point, we’ll make our own weighting matrix and see what we can come up with.
Afghanistan is not the largest country but it’s an incredibly diverse country from a geographic perspective. You’ve got deserts in some areas, high mountains. You can have snow blocking roadways and at the same time you can have cars overheating in another part of the country. It’s very difficult to get around the country and not a lot in the way of roads and other things. If you think about it as like a wheel with spokes: you often have to come back to the center to go somewhere else. You can’t go out to the end and then go to another spot that might only be ten kilometers away.
Places like Afghanistan, these are all real paper and pencil operations. This is all about getting somebody who can get out into a neighborhood and find the appropriate person to talk to, hopefully get them away from the rest of the family, because you have large groups of people all living together. You may not be able to do a completely private interview. It generates a lot of attention. You come into a small village, they haven’t had visitors in a long time, everybody wants to know what’s happening.
Women [interviewers] have a tremendously difficult time of just doing their jobs. They have to make sure they’re done at a certain hour of the day because they can have physical problems on the street. We’ve had a lot of interviewers attacked by people who don’t think it’s appropriate that they should be out working. They can run into problems at different houses that they come to. We find that certain hours of the day work better than others because the men tend to be out somewhere else and so it’s easier to get in and out. But completely separate field teams.
Afghanistan is a mostly rural society. From our estimates, from what we can gather from previous census data, updates that are done by the central statistics office, some work done by the world food program and our own work, it’s about 77-78 percent rural. So it’s a logistics issue of getting out to all these rural places and then only about 20 or 22 percent of the population is living in urban areas.
We split the sample at the outset in Afghanistan between men and women. This is an unusual way to approach survey research but, because of the restrictions that we have, it’s really only the way to efficiently get work done. And so, right out of the gate, we have sampling points that are going to be all female sampling points and sampling points that are going to be all male sampling points. And we’ve spent a lot of time looking at the data doing different design effect calculations and what we’ve discovered is that it’s more about distance between locations than it is about gender or ethnicity in Afghanistan that create differences in the opinions.
So what we have to be careful about is that we don’t end up clustering lots of female sampling points in one area and then there are no male sampling points or vice versa. But it’s a huge design effect problem for us and we’ve had to work around that but it’s really the only way to make sure we get both men and women into the sample.
We select the households using random walk methodology. To get to that point, at this stage we think we have a fairly good idea of the number of people that are in each province in Afghanistan. We have a pretty good sense of the size of the different districts so we can proportionally distribute the number of interviews down to the district level.
But beyond the district—who knows? There could be four thousand people in one village, three thousand in another. It’s very hard to tell. There isn’t enough information about those spots. So we end up having to treat them almost equal and we do it as a simple random distribution of villages that are given an assigned starting location off of maps and grids that we have and then they’re conducting a random walk in rural areas, it’s usually every third household. And we use Kish grids* to select [respondents]. As I’ve said we split the sample male/female, so female interviewers are making a listing of the female members of the household and then they’re selecting them according to the Kish grid. The males are making a listing of the male members of the household and then selecting the male according to the Kish grid.
We’ve got a three call back policy. They can’t just take whoever happens to come to the door and be friendly enough to talk to them. We have to spend a lot of time sending people back to some of these villages to do the checks and to make sure it’s happening. It can be difficult, especially in some of these extremely rural spots. We go to places up in the mountains and spending the whole day just to get to some of these locations. And then it’s quite an expense to send a supervisor or someone back. But it’s a randomized process and so the interviewers know that they could be checked up on at any time and they won’t get paid if they find out that there’s cheating and other things. So we uncover it. We uncover problems all the time. That’s how you train the field force. That’s how you teach them you’re serious about what you do.
And then we have to get everything back to one central location to key punch it in. Forget about scanners, forget about handheld PDAs that are sending it back and you’ve got instant data. We’ve actually got to have people sit down with paper questionnaires and punch this into computers and then we have to go back and validate a certain percentage of those so we do double entry to make sure that the key punchers are doing their job right. And then we’ve got to spend a lot of time going through and cleaning up all the mistakes that the key punchers have made. So it’s a very, very labor intensive process.
Growing Violence in Iraq
And then there’s Iraq, which is a very different situation. We have a much better workforce in Iraq. The infrastructure, although very, very badly damaged is much more manageable. When we came to Iraq, we found lots of people with PhDs, a lot of people who had studied sociology. There were people that were ready and willing to do this work. When we got to Afghanistan, we had a hard time finding people that could read and write. One of the first tests that we give interviewers in Afghanistan is they have to stand in front of us and read from the questionnaire. A lot of people just can’t do that. In Iraq, it’s a completely different situation: well-educated population, easier to set up the operation there.
What’s changed in Iraq is that it is significantly more violent than Afghanistan and it’s become a much more difficult thing to work there. We started with a more centralized operation, the way that we work in Afghanistan, with everything centralized out of Kabul. That’s the way that we used to work in Iraq but then anytime something happened to the central part of the operation, the rest of it fell apart. So we now have almost 18 separate offices, one for each province in Iraq so they can operate independently from each other. And month to month, one province will have a problem and another province will have to fill in for that team. So they have to be interchangeable.
It’s set up where we’ve got an overall quantitative team director and a qualitative team director. The quant team has regional trainers that travel around doing training for the different groups. There are 19 field supervisors. There are 18 provinces; we have two main supervisors for Baghdad because it’s a large area. And we’ve got a pool of over 200 interviewers at this point. And the qualitative team is growing.
In Iraq, it’s the insurgency and the breakdown of law and order—that’s what makes doing the survey research difficult. Random sampling in places like Iraq and Afghanistan has an added dangerous component to it. You don’t know what you might find behind the door.
In Afghanistan, recently, we had some people knock on a door and they happened to come across a house where there were Taliban members hiding out. They got held and we had to get people involved negotiating their release. In Iraq, we’ve had people stopped at checkpoints and held by different groups: criminal groups asking for money, political groups wanting different things.
And in Iraq, especially, there are the ethnic divisions. It’s a serious problem. We have to have separate teams. We can’t send Sunnis from Anbar into Baghdad to interview Shias. We can’t send Shias from Basra to do work in Sunni areas. We can’t send Kurds anywhere other than Kurdish areas. Where we started out with a unified field force in Iraq, we now have all of these separate field forces that have to operate separately on a linguistic basis and an ethnic basis.
Once again, Iraq doesn’t have a census. We’re using updates that were done before the elections to try to estimate the number of people so that when they did the elections they could distribute things evenly. And we’re using information from the world food program and the United Nations. But once again in Iraq with all of this movement internally, with people changing neighborhoods with ethnic strife, with the war, all the other things that are happening, it’s hard to know how stable those populations have been. So, once again, once we get down below a certain level, we’re doing a simple random sample. We’re doing the best area probability we can to try to cover the geography and then come back and see what we have and how closely it matches the other data that are out there.
We use some things that are called “place codes.” There was a lot of satellite imaging that was done of Iraq in the 1990s for the United Nations to help with the distribution of aid. So you can pinpoint actual settlements and we had over 11,000 that were pinpointed on a map and over a couple years, we’ve been able to visit the vast majority of them and classify them: Are they urban or rural? Do they exist? Have they disappeared? Are there new ones? If so, we’ve added them in.
* Kish grid: a mechanism for selecting a respondent from a household at random so that the entire sample reflects the makeup of the general population in terms of age, gender, family status, etc.