Chris Blattman on the incubation of ideas until opportunity strikes

blattman-roberts

Is a factory job better than a cash grant and some training? Chris Blattman and Stefan Dercon have a recent study in Ethiopia where they test these two options with a randomized-controlled trial. Back in December, Chris Blattman discussed with study with Russ Roberts on the EconTalk podcast.

In one interesting bit, Blattman highlights how holding onto an idea and repeatedly seeking an opportunity to implement it can ultimately bear fruit. I transcribed it (abridging a little for readability).

Since you have 300 people lining up for these jobs, instead of taking the first 50 in line who are qualified for the job and hiring them, why not see if we can find a factory owner who will find 150 who are qualified and instead of taking the first 50, we’ll flip a coin and we’ll take 50 out of those 150 qualified applicants as random and we’ll follow them over time and we’ll look at what happens to their incomes and their health and their career trajectories.

I had this idea as a graduate student 10 or 12 years ago, and I always thought, “Every time I meet a factory owner I’m going to feel him out. And I did. Once in a while I’d be on a plane to Uganda to work on one of my projects, usually related to poverty or conflict, and maybe I’d sit by a factory owner, and I’d say here’s this idea that I have, and they’d usually look at me kind of funny. They wouldn’t leap at the possibility. I was just this person they met on a plane, and I was a graduate student. I probably didn’t approach it well, and so it never really materialized.

So I was at a conference in London and there was an Ethiopian businessman who was sort of a real estate mogul. He was giving a talk to a group of development economics at the International Growth Centre, and I approached him afterwards and said, “That was terrific,” and I really enjoyed talking to him and we kept chatting and I said, “I had this idea. I think that your firms not only help achieve growth, but I think they might actually be tools of poverty alleviation. Here’s an easy way to answer that question.” And he said, “That sounds great. Let’s do it.” And so literally five or six weeks later we were on the ground in Ethiopia doing the first randomization.

I recommend the whole conversation.

Getting true voluntary consent for your field experiment may be harder than you think

ufahamu

Kim Dionne recently interviewed Melissa Graboyes on the excellent new Ufahamu Africa podcast. Graboyes wrote a history – The Experiment Must Continue: Medical Research and Ethics in East Africa, 1940-2014. I’m reading the book, and Graboyes provides a rich picture, filled with first-person reports from East African participants in research.

Dionne asked Graboyes for key takeaways. Here’s one on consent, which very much applies to social science research.

People regularly mistake the idea that they are participating in an experiment and it’s designed to benefit them personally, rather than the experiment is designed to generate data that can be used to answer important questions and hopefully get us closer to solving some important problems. This disconnect is really profound and it jeopardizes consent.

The basic components of consent laid in the Nuremburg code, laid out in European guidelines, laid out in U.S. national law, is that consent has to be informed, it has to have understanding, and it has to be voluntary. So we can inform people by giving them a consent form translated into Swahili. We can inform them by reading that consent form in Swahili. But if they don’t understand what we’re saying and can’t accurately describe back the kind of experiment they’re participating in with the commensurate risks and benefits that go with it, that’s not really voluntary consent, and it jeopardizes the idea that they are autonomously choosing to participate. I think that there’s a lot of research going on that is stumbling at that step, that we’re formally checking all the boxes we need to, but that we’re not adhering to the real meaning of what that rule is supposed to be about. [I’ve edited very slightly for readability.]

The interview has much more, and the podcast overall is a delight. I recommend it and Graboyes’ book.

The respective value of theory and of field experiments

img_2762

This is Antoinette Schoar of MIT.

On theory:

Theory is a way of organizing your thinking. In the end it’s setting up a hypothesis so that you have something to test. That’s why I also think that theory is most useful when it gives enough structure that it can actually allow you to refute hypotheses. The most frustrating theories are the ones that are so flexible that they fit any finding. This doesn’t help me to make better sense of the world. But it’s important to start from a theoretical framework rather than just a story because it forces you to be more precise. I think of myself primarily as an empiricist, so theory to me is very helpful when it helps me to unearth new empirical insights.

On field experiments:

What I really like about running field experiments is that even if you’re working with one bank or one NGO because you have to engage with the organization to implement something on the ground, if you want to do a good job, you have to get involved in the details. Which means that you get constant feedback on what is feasible and what is implementable or what is practical and what is just a pipe dream. What sounds great in an ivory tower may be impossible in the real world.

Of course, there are many more things in economics than theory and field experiments, but they are two important things.

From Tim Ogden’s Experimental Conversations: Perspectives on Randomized Trials in Development Economics 

Why don’t people take up savings products?

img_2595

This is Xavier Giné at the World Bank

Part of the take-up problem, especially in the case of savings, is that some of these products are pretty crappy. If we see no demand for these products, maybe that’s a good thing actually. If you put money into one of these accounts, check the balance a few times, make a few withdrawals, half or all the money has been eaten up by fees. So the characteristics of the product are very important.

That’s from Tim Ogden’s Experimental Conversations: Perspectives on Randomized Trials in Development Economics.

Beyond savings products, this is important because there is a temptation to lump interventions together (How effective are home-based child care visits? How effective is teacher training?) when in fact there is massive diversity in the particulars of the interventions. Sara Nadel and Lant Pritchett have referred to this as “high dimensional design space,” and Popova et al. document it in the case of teacher training interventions.

There is no reason to believe — ex ante — that all interventions in the same “category” will have the same effect. Indeed, in work Anna Popova and I did analyzing randomized controlled trials of education interventions, we found that in many cases, the variation in impact within categories exceeded the variation across categories, per the table below.

variation

So, as Giné says, “the characteristics of the product are very important.”

3 questions to guide every experiment, from Dean Karlan of IPA

img_2266

I always try to reduce things to three questions—even in simple interventions like passing out nutritional supplements to infants and toddlers. These are theory-driven questions that you should be using whenever you’re justifying any intervention whatsoever. 

And the questions are 

  1. “What’s the market failure? Why isn’t the market and the invisible hand working?”
  2. “How does this intervention specifically solve the market failure?”
  3. “What’s the welfare impact of solving the market failure?”

Karlan makes the point that experiments should absolutely be driven by theory. But theory doesn’t have to mean three pages of math. The theory can be simple, and the questions above sum up what your theory should predict.

from Tim Ogden’s interview with Dean Karlan in Experimental Conversations: Perspectives on Randomized Trials in Development Economics

What do African schoolteachers think about mathematics?

math-education-in-africa

George Bethell has put out a report via the World Bank, Mathematics Education in Sub-Saharan Africa: Status, Challenges, and Opportunities. It brings together an array of analysis on current mathematics performance and teaching around the continent.

I’d like to highlight two elements. First, a survey in 6 African countries shows teacher attitudes towards math. Now, it’s just 70 teachers per country (roughly 50 primary and 20 secondary), so this is suggestive only. Key findings: Almost all teachers agree that mathematical skills are useful for everyone (except in DRC). But between 24 percent and 62 percent of teachers don’t believe that “everyone has the potential to be good at mathematics,” depending on the country, and a majority of teachers in all countries except Cameroon believe that “you have to have the right sort of brain to be good at mathematics.” It’s easy to imagine that these kinds of beliefs then play into the way teachers teach. (You can read more about it in Appendix A of the report.) Of course, I also don’t know how different these are from other parts of the world. The point is not whether these 6 countries are worse or better than other countries, but that teachers in these country have these particular beliefs about math education which may well affect pedagogy.

what-do-african-teachers-think-about-math

The report also lays out what he sees as the big research questions that remain in this area.

  1. How can countries in SSA monitor trends in mathematical achievement?
  2. How do learners understand mathematical concepts as demonstrated by their teachers? How do they approach mathematical problems?
  3. How effective are the textbooks currently being used to teach basic mathematics in SSA?
  4. How can national assessments of student achievement in mathematics be improved so that they provide policy makers and teachers with the information needed to improve outcomes in mathematics?
  5. Where Open Educational Resources have been used as the basis of, or to supplement, formal teacher education development programmes, have they been effective?
  6. Which of the e-learning and m-learning technologies in the classroom have the greatest potential to raise levels of numeracy and mathematical competence? What are the challenges of introducing e- and m-learning technologies – especially in fragile states?

Calling all researchers! To work!

math-questions-to-be-answered

The poor don’t use cash transfers on alcohol and tobacco. Really.

cash-transfers-and-temptation-goods

Two years ago, Anna Popova and I put out a working paper examining whether beneficiaries of cash transfer programs are more likely than others to spend money on alcohol and cigarettes (“temptation goods”). That paper has just been published, in the journal Economic Development and Cultural Change.

The findings of the published version do not vary from the working paper: Across continents, whether the programs have conditions or don’t, the result is the same. The poor don’t spend more on temptation goods. But for the published version, we complemented our vote count (where you sum up how many programs find a positive effect and how many find a negative effect) with a formal meta-analysis. You can see the forest plot below. (The results are not substantively different from the “vote count” review that we did in the working paper and maintain in the published version as a complement to the meta-analysis.)

meta-analysis

As you can see, while there are only two big negative effects, both from Nicaragua, most of the effects are slightly negative, and none of them are strongly positive. We do various checks to make sure that we’re not just picking up people telling surveyors what they want to hear, and we’re confident that cannot explain the consistent lack of impact across venues.

Why might there be a negative effect? After all, if people like alcohol, we might expect them to spend more on it when they have more money. We can’t say definitively, but even unconditional transfer programs almost always come with strong messaging: Recipients hear, again and again, that this money is for their family, that this money is to make their lives better, and so on and so on. We know from others areas of economics that labeling money has an effect (called the flypaper effect).

So you can be for cash transfers or against cash transfers, but don’t be against them because you think the poor will use the money on temptation goods. They won’t. To quote the last line of our paper, “We do have estimates from Peru that beneficiaries are more likely to purchase a roasted chicken at a restaurant or some chocolates soon after receiving their transfer (Dasso and Fernandez 2013), but hopefully even the most puritanical policy maker would not begrudge the poor a piece of chocolate.”

Over the last 100 years, most US blacks migrated from the South to the North. Did they find what they sought?

img_2195

Yes and no. Better income and better social conditions, but also a black-white pay gap that changed little over time. Why?

Leah Platt Boustan, UCLA economist (and my friend), just wrote a book on it, Competition in the Promised Land: Black Migrants in Northern Cities and Labor Markets.

This is from James Ryerson’s New York Times review

In her rich and technical account, the economist Leah Platt Boustan employs the tools of her trade — resourceful matching of data sets, rigorous modeling of labor phenomena, sweeping use of census figures — to analyze the demographics and economics of the Great Migration as a whole… Her investigation both deepens our understanding of what we think we know and adds new complexities and wrinkles.

I expect it’s excellent.

Has the war over RCTs been won?

experimental-conversations

Esther Duflo and Abhijit Banerjee of MIT and JPAL weigh in during an interview in Tim Ogden’s forthcoming book, Experimental Conversations, which I am enjoying thoroughly:

Esther: “I think it’s been completely won in that I think it’s just happening. A lot of people are doing it without us. It’s being used. I think it is now understood to be one of the tools. The argument within the economics profession [over the value of RCTs] had two main consequences, both good. First, it raised the profile. If something was debated, people began to believe it must be significant. Second, it did force us to answer the challenges. There were a lot of valid points that were raised and it forced us to react. We’ve become more intelligent as a result.”

Abhijit: “I am less certain that it has been won. The acid test of whether an idea has come to stay is that it becomes something that no one needs to justify using. … RCTs aren’t there yet: it is true almost everyone is doing them, but many of them are taking the trouble to explain that what they do is better than a ‘mere RCT.’ We need to get to the point where people take RCTs to be the obvious tool to use when possible to answer a particular class of empirical questions.”