How many times do you have to test a program before you’re confident it will work somewhere else?

I heard this question at an impact evaluation training event a few weeks ago. I’ve heard some variation on it many times. Wouldn’t it be grand if there were a magic number? “5 times. If it works 5 times, it will work anywhere.” Alas, ’tis not so.

But Mary Ann Bates and Rachel Glennerster have a good answer in their new essay in the Stanford Social Innovation Review:

Must an identical program or policy be replicated a specific number of times before it is scaled up? One of the most common questions we get asked is how many times a study needs to be replicated in different contexts before a decision maker can rely on evidence from other contexts. We think this is the wrong way to think about evidence. There are examples of the same program being tested at multiple sites: For example, a coordinated set of seven randomized trials of an intensive graduation program to support the ultra-poor in seven countries found positive impacts in the majority of cases. This type of evidence should be weighted highly in our decision making. But if we only draw on results from studies that have been replicated many times, we throw away a lot of potentially relevant information.

Read the whole essay or my blog post on other aspects of the essay.

Researchers as healers or witches?

“A researcher [mtafiti] is an important person because he indeed is the one who discovers everything [anayegundua kila kitu].” – Mzee Thomas Inyassi

Melissa Graboyes describes how research participants in Tanzania see the medical researchers who come to them for samples and information. On the one hand, “East Africans noted the similarity between researchers and doctors: they both gave out medicine and helped the sick recover.” On the other hand…

As healers and witches are understood to rely on the same skills, once researchers were compared with healers, it was not such a stretch to compare them to witches. … Witch doctors often work at night and want blood. … Researchers also worked at night, collecting blood samples by going door to door or collecting night-biting mosquitos by walking around in the bush. For both witches and researchers, blood was valued above all other substances and its use was shrouded in secrecy.

This, from Graboyes’ intriguing book The Experiment Must Continue: Medical Research and Ethics in East Africa, 1940-2014.

Lest you think this is limited only to medical research, consider the following passage from Kremer, Miguel, and Thornton’s randomized evaluation of a girls’ scholarship program in western Kenya:

There is also a tradition of suspicion of outsiders in Teso, and this has at times led to misunderstandings with NGOs there. A government report noted that indigenous religious beliefs, traditional taboos, and witchcraft practices remain stronger in Teso than in Busia (Were, 1986).

Events that occurred during the study period appear to have interacted in an adverse way with these preexisting factors in Teso district. In June 2001 lightning struck and severely damaged a Teso primary school, killing 7 students and injuring 27 others. Although that school was not in the scholarship program, the NGO had been involved with another assistance program there. Some community members associated the lightning strike with the NGO, and this appears to have led some schools to pull out of the girls’ scholarship program. Of 58 Teso sample schools, 5 pulled out immediately following the lightning strike, as did a school located in Busia with a substantial ethnic Teso population. (Moreover, one girl in Teso who won the ICS scholarship in 2001 later refused the scholarship award, reportedly because of negative views toward the NGO.)

Witches or healers?

One take away from this is that researchers need to do more to make sure participants understand what they are participating in.

The EJAQ: The Grand Prize of Economics Publishing?

For performers, one illustrious distinction is the EGOT, for people who have won an Emmy, a Grammy, an Oscar, and a Tony. Only 12 people have ever won all four, including Audrey Hepburn, Mel Brooks, and Whoopi Goldberg.

Is there an equivalent distinction for economists? That would have to be the EJAQ (or the JAQE, if you prefer), for those who have published in Econometrica, the Journal of Political Economy, the American Economic Review, and the Quarterly Journal of Economics.* (And no, Papers and Proceedings doesn’t count for this – that might be the case-sensitive EJaQ.)

Who’s won it? I looked up the research histories of a handful of scholars and identified a few EJAQs right off the bat! (I’ve updated the post with recommendations from others.)

  • Colin Camerer (E, J, A, Q)
  • Anne Case (E, J, A, Q)
  • Matthew Gentzkow (E, J, A, Q)
  • James Heckman (E, J, A, Q)
  • Larry Katz (E, J, A, Q)
  • Michael Kremer (E, J, A, Q)
  • John List (E, J, A, Q)
  • Ted Miguel (E, J, A, Q)
  • Dominic Roner (E, J, A, Q)
  • Jesse Shapiro (E, J, A, Q)
  • Doug Staiger (E, J, A, Q)
  • Robert Townsend (E, J, A, Q)
  • Fabrizio Zilibotti (E, J, A, Q)

Others that have been reported (but I haven’t had time to verify) include Daron Acemoglu, Josh Angrist, Susan Athey, Amy Finkelstein, Muriel Niederle, Jean Tirole.

Who am I missing? Are you next in line for the EJAQ? I also identified scholars who are short just one: Marianne Bertrand is short an Econometrica and Esther Duflo is short a Journal of Political Economy.

* This grand prize is the joint brainchild of Markus Goldstein and me. It comes only with bragging rights.

Update: It has been suggested that an alternative, even more exclusive club would be the REJAQ, where we add the Review of Economic Studies to the four above. Take your pick!

REJAQ winners would include Ted Miguel, Dominic Rohner, Jean Tirole, and I’m sure a few others.

Update 2: This could be called the JAQE, if you prefer. I’ve updated it with new information. 

Nothing new under the sun — Education edition

Some years ago, I was evaluating an education program in The Gambia (read the evaluation here), soon after the government had outlawed corporal punishment in schools. We included a question about it in the evaluation and learned that there was a gap between legislation and practice, which the government then sought to resolve. 

Of course, controversy over corporal punishment in schools isn’t new, but I was surprised to see it debated in Noli me tangere, the 1887 novel by Filipino writer José Rizal. A frustrated schoolteacher recounts that after reading several books, his views changed: 

Lashings, for example, which since time immemorial had been the province of schools and which before I had seen as the only effective way to make children learn (that is how they have accustomed us to believe), began to seem far removed from contributing to a child’s progress, completely useless. I became convinced that when one keeps the switch or the rod in view reasoning is impossible… I began to think that the best thing I could do for these children was to develop confidence, security, and self-esteem.

So he eliminates corporal punishment. 

Little by little I held back the switch. I took the whips home and replaced them with emulation and belief in oneself.

Like any good experimenter, he evaluated short and medium run impacts.

In the beginning it seemed as though my method was impractical: a lot of them stopped studying altogether. But I pressed on, and I noticed that little by little their spirits rose. More students attended class, and more often. And when one day one was praised in front of everyone, the following day he learned twice as much.

But the local priest and the parents didn’t buy it and demanded he return to the traditional system.

I had to renounce a system that after a great deal of effort had begun to bear fruit.

Poor guy, but I imagine there are a number of reformers today who can feel his pain. 

The quotes are from Harold Augenbraum’s translation of the book.

Sure we should all be feminists, but how to raise one? A review of Chimamanda Ngozi Adichie’s Dear Ijeawele, or A Feminist Manifesto in Fifteen Suggestions 

On a recent road trip, my wife and I read aloud Chimamanda Ngozi Adichie’s new book Dear Ijeawele, or A Feminist Manifesto in Fifteen Suggestions, a collection of advice to Adichie’s childhood friend on how to raise her young daughter as a feminist. 

This is a small book — just 63 pages — but don’t confuse it with Adichie’s OTHER small book on feminism, the 2014 We Should All Be Feminists, which was based on a public talk of the same name. The first book is the what (be feminist!) and the second book is the how (15 suggestions!).

This is a readable, thought- and discussion-provoking collection. Each suggestion is followed by a few pages of discussion, which is where the richness lies. Suggestion #6, for example, is “Teach her to question language. Language is the repository of our prejudices, our beliefs, our assumptions.” But here’s how that plays out concretely in the discussion: “Teach her that if you criticize X in women but do not criticize X in men, then you do not have a problem with X, you have a problem with women. For X please insert words like ‘anger,’ ‘ambition,’ ‘loudness,'” etc. 

Here’s another: Suggestion #4 is “Beware the danger of what I call Feminism Lite,” which is the idea of “conditional female equality,” which gives rise to ideas like “men are naturally superior but should be expected to ‘treat women well.’ No. No. No. There must be more than male benevolence as the basis for a woman’s well-being.” Yes! Yes! Yes!

I didn’t agree with every idea, but each was well argued such that I couldn’t dismiss it without careful consideration. 

Obviously I could just quote the whole book. But I won’t. Go read it yourself. It won’t take you long, but you’ll be thinking about it long after, and — if you’re like me — encouraging others to do the same.

Fun fact: The book describes some sexist attitudes in order to combat them. Our 12-year-old son — listening to snatches from the backseat — commented that the book sounded pretty sexist to him. Correct on the passage but not on the book as a whole, son: Context!