Last week I attended the conference for Research on Improving Systems of Education. I just post
ed a little round-up of research presented over at Development Impact.
As an added treat, you can hear Shwetlena Sabarwal doing an impression of my presentation style here.
One key feature of test-based accountability systems in the U.S. is that every student receives not only a test score but also a label based on their performance. Massachusetts, the state that we study, assigns students labels of Failing, Needs Improvement, Proficient, or Advanced by determining cut-points with which it divides the finer-grained test-score distribution into performance regions. … These labels provide no additional information beyond the test scores on which they are based; they are simply coarse summaries of a student’s performance. We focus on responses to labels that have no state-defined consequences for students.
This, from a paper by John Papay, Richard Murnane, and John Willett, in the Journal of Human Resources.
Using a regression-discontinuity design, we find persistent effects of earning a more positive label on the college-going decisions of urban, low-income students.
Here’s an open-access, earlier version of the paper. And here’s the key figure, on the left comparing students with an “advanced” versus a “proficient” label, and on the right, comparing the “proficient” to the “needs improvement” label:
On January 1, 2010, Andrew Evans boarded a city bus in Washington, DC. His goal was to travel by bus all the way to Ushuaia, Chile, where he would board a ship to Antarctica — if he made it in time! No tickets were purchased in advance, and the route was laid out only vaguely. In The Black Penguin, Evans tells his story, and what a ride it is! As Publishers Weekly puts it, “Sketchy border guards, close calls with violence and natural disasters, and intriguing characters fill vignettes that range from hair-raising to hilarious.” Evans engages the people around him consistently and introduces us to a steady stream of wonderful friends, along with a few foes. His trip across the Americas is interwoven with memories of growing up gay in a conservative faith (Mormon) and the strain that placed on him in middle America, in his church, and with his family.
Salaam, from Busia to Kampala, from Maceió to Recife), and Andrew’s vivid descriptions took me back to my own bus rides. I remember riding a matatu in rural western Kenya: When the bus got a flat tire, there was no jack, so the conductor asked all the male passengers to lift the minibus while he and the driver changed the tire. Evans writes: “To travel is to know the unfairness of the world, time and time again.” It’s true, and yet traveling by bus also — for a brief period — brings people from a wide range of social groups into contact. Evans spins that into a compelling, moving narrative. I wouldn’t miss this travel memoir.
A new paper by Paul Christian and Brian Dillon poses this question: “Does a consistently seasonal diet during childhood have long-run effects on human capital formation?” They use Tanzania’s Kagera Health and Development Survey — a 19-year panel survey — to answer the question. As you can see from the figure below, Tanzania has dramatic seasonality: Children have very different access to food in some parts of the year than in others.
Christian and Dillon develop a structural model — which you can read all about in the paper — and use the household data to estimate it.
Here is a taste of the results:
We find a robust, negative relationship between consumption seasonality and human capital formation. Across specifications, the negative relationship between seasonality and human capital is 30-60% of the magnitude of the positive relationship between average consumption and human capital (in the same units). … The effects of seasonality on height is greatest for children in utero and during infancy, during the critical first 1,000 days of life. Effects on education are most pronounced for older children, suggesting that behavioral channels such as dropping out of school to help on the farm are more important in this sample than early life impacts on cognitive performance. When we further allow for heterogeneity by both age and gender, we see that the height effects during infancy are concentrated among girls, while the education effects during adolescence are largely driven by boys.
A new study was just published in the Review of Educational Research: Identifying Effective Education Interventions in Sub-Saharan Africa: A Meta-Analysis of Impact Evaluations, by Katharine Conn from Columbia University. Here is a subset of findings (emphasis added):
I identify educational interventions with an impact on student learning in Sub-Saharan Africa. After a systematic literature search, I conducted a meta-analysis synthesizing 56 articles containing 66 separate experiments and quasi-experiments and 83 treatment arms…. A key finding is that programs that alter teacher pedagogy or classroom instructional techniques had an effect size approximately 0.30 standard deviations greater than all other types of programs combined. Limited evidence further suggests that pedagogical programs that employed adaptive instruction or teacher coaching were particularly effective.
In case you don’t have access, the earlier, open-access dissertation version has the same sample and the same findings reported in the abstract.
That version was one of six reviews that Anna Popova and I synthesized in our paper What Really Works to Improve Learning in Developing Countries? An Analysis of Divergent Findings in Systematic Reviews (open-access version).
I heard this question at an impact evaluation training event a few weeks ago. I’ve heard some variation on it many times. Wouldn’t it be grand if there were a magic number? “5 times. If it works 5 times, it will work anywhere.” Alas, ’tis not so.
But Mary Ann Bates and Rachel Glennerster have a good answer in their new essay in the Stanford Social Innovation Review:
Must an identical program or policy be replicated a specific number of times before it is scaled up? One of the most common questions we get asked is how many times a study needs to be replicated in different contexts before a decision maker can rely on evidence from other contexts. We think this is the wrong way to think about evidence. There are examples of the same program being tested at multiple sites: For example, a coordinated set of seven randomized trials of an intensive graduation program to support the ultra-poor in seven countries found positive impacts in the majority of cases. This type of evidence should be weighted highly in our decision making. But if we only draw on results from studies that have been replicated many times, we throw away a lot of potentially relevant information.
Read the whole essay or my blog post on other aspects of the essay.
“A researcher [mtafiti] is an important person because he indeed is the one who discovers everything [anayegundua kila kitu].” – Mzee Thomas Inyassi
Melissa Graboyes describes how research participants in Tanzania see the medical researchers who come to them for samples and information. On the one hand, “East Africans noted the similarity between researchers and doctors: they both gave out medicine and helped the sick recover.” On the other hand…
As healers and witches are understood to rely on the same skills, once researchers were compared with healers, it was not such a stretch to compare them to witches. … Witch doctors often work at night and want blood. … Researchers also worked at night, collecting blood samples by going door to door or collecting night-biting mosquitos by walking around in the bush. For both witches and researchers, blood was valued above all other substances and its use was shrouded in secrecy.
This, from Graboyes’ intriguing book The Experiment Must Continue: Medical Research and Ethics in East Africa, 1940-2014.
Lest you think this is limited only to medical research, consider the following passage from Kremer, Miguel, and Thornton’s randomized evaluation of a girls’ scholarship program in western Kenya:
There is also a tradition of suspicion of outsiders in Teso, and this has at times led to misunderstandings with NGOs there. A government report noted that indigenous religious beliefs, traditional taboos, and witchcraft practices remain stronger in Teso than in Busia (Were, 1986).
Events that occurred during the study period appear to have interacted in an adverse way with these preexisting factors in Teso district. In June 2001 lightning struck and severely damaged a Teso primary school, killing 7 students and injuring 27 others. Although that school was not in the scholarship program, the NGO had been involved with another assistance program there. Some community members associated the lightning strike with the NGO, and this appears to have led some schools to pull out of the girls’ scholarship program. Of 58 Teso sample schools, 5 pulled out immediately following the lightning strike, as did a school located in Busia with a substantial ethnic Teso population. (Moreover, one girl in Teso who won the ICS scholarship in 2001 later refused the scholarship award, reportedly because of negative views toward the NGO.)
Witches or healers?
One take away from this is that researchers need to do more to make sure participants understand what they are participating in.
This Saturday, the Etisalat Prize for Literature will be awarded to an African writer for her or his first novel. I read the three books on the shortlist and make recommendations as to which will most heal the ills in your life over at African literary site Brittle Paper.
For performers, one illustrious distinction is the EGOT, for people who have won an Emmy, a Grammy, an Oscar, and a Tony. Only 12 people have ever won all four, including Audrey Hepburn, Mel Brooks, and Whoopi Goldberg.
Is there an equivalent distinction for economists? That would have to be the EJAQ (or the JAQE, if you prefer), for those who have published in Econometrica, the Journal of Political Economy, the American Economic Review, and the Quarterly Journal of Economics.* (And no, Papers and Proceedings doesn’t count for this – that might be the case-sensitive EJaQ.)
Who’s won it? I looked up the research histories of a handful of scholars and identified a few EJAQs right off the bat! (I’ve updated the post with recommendations from others.)
Others that have been reported (but I haven’t had time to verify) include Daron Acemoglu, Josh Angrist, Susan Athey, Amy Finkelstein, Muriel Niederle, Jean Tirole.
Who am I missing? Are you next in line for the EJAQ? I also identified scholars who are short just one: Marianne Bertrand is short an Econometrica and Esther Duflo is short a Journal of Political Economy.
* This grand prize is the joint brainchild of Markus Goldstein and me. It comes only with bragging rights.
Update: It has been suggested that an alternative, even more exclusive club would be the REJAQ, where we add the Review of Economic Studies to the four above. Take your pick!
REJAQ winners would include Ted Miguel, Dominic Rohner, Jean Tirole, and I’m sure a few others.
Update 2: This could be called the JAQE, if you prefer. I’ve updated it with new information.
…you’re voting on the next book, and you get into a debate over whether you should choose the winner based on first-past-the-post, instant-runoff, or the Borda count. All this despite Arrow’s Impossibility Theorem telling us that no tallying system will be perfect.
Did I say too many economists? I meant just the right number.
David Evans' personal blog 