Even if technology improves literacy, is it worth the cost?

Ben Piper reports on insightful work that he and co-authors have done comparing various education technology intervention in Kenya in terms of both effectiveness (do they improve reading ability?) and the cost-effectiveness (what’s the cost per reading gain?).

I recommend his full post (or the research paper it’s based on). Here are a couple of highlights:

When compared to traditional literacy programs, the more intensive ICT interventions did not produce large enough gains in learning outcomes to justify the cost. This is not to say that each of the ICT interventions did not produce improvements in students’ reading ability…. [But] the cost-effectiveness of all of these programs might still be significantly lower than a clear investment in high quality literacy programs…. In additional to monetary cost, an opportunity cost existed…. Many of the teachers, tutors, and students lacked exposure to technology and the time and energy spent on learning how to use the technology reduced the amount of time for instructional improvement activities. 

When costs are considered, there are non-ICT interventions that could have larger impacts on learning outcomes with reduced costs; one such option could include assigning the best teachers to the first grade when children are learning how to read, rather than to the end of primary school as many schools do.

Economists will disagree with the standard errors if I understand the specification right: Randomization is at the district level and I don’t believe the authors cluster the standard errors. 

But I don’t think that will change the fundamental message here: Even if there are some gains from education technology, we have to ask when they will be most likely to be worth the cost.


Economic Development in 20 minutes to middle schoolers

Earlier this week I had 20 minutes each to speak to four classes of middle schoolers about my career. I talked about economic development. I used a presentation (available in full here). Given that it was the antepenultimate day of school, the students and teachers appeared very engaged.

  1. I showed the students four families from Gapminder’s Dollar Street initiative — one from India, one from Burundi, one from Ukraine, and one from Colombia, and I had students vote (by raised hands) on which family they expected was poorest and which was wealthiest.

slide 2

2. Then I introduced the four families in turn, and I expressed their monthly income in terms of the price of school lunches at the middle school where I was speaking. (The Ukrainian family’s monthly income was the equivalent of 3,100+ school lunches, more than the most insatiable student should ever consume.)

slide 3

3. Having highlighted the massive gaps in income between families, I invited the students to vote (again, by raised hands) on the number of people in the world currently in extreme poverty. Hint: According to the latest estimates from Cruz et al. (2015), we’re at 700 million.

slide 4

4. Then I showed — in two ways — how much poverty has decreased over time. First I showed the figure below from Our World in Data. I also showed the evolving chart on income and life expectancy from Gapminder. (Technical difficulties precluded showing the actual evolution over time, but at least I could show screenshots of the beginning and the end.)

slide 9

5. I then highlighted geographical concentrations of poverty.

slide 12

6. Then I gave two very simple definitions of economics:

  • Macro: Why are some countries rich and others poor
  • Micro: How can poor families get out of poverty?

7. What does economic growth look like? Here’s some of the variation, where countries on the top left are those that grew the most: low income in 1960, high income in 2014.

slide 13

8. I then invited the students to suggest what makes countries grow. We talked about a few possibilities.

slide 15

9. We then returned to the growth map and differentiated between two high-growth countries: South Korea, which produces goods for trade (I had all the students with Samsung devices raise their hands) versus Equatorial Guinea, which produces a natural resource for trade (oil). We talked about the different implications for inequality.

10. I then talked through the two objectives of the World Bank: to encourage growth and to end extreme poverty. (To be more precise, the “twin goals” are to encourage “shared prosperity” — growth that benefits the bottom 40% of the population — and to end extreme poverty.)

11. Then, since education is an area I work in actively, I highlighted the relationship between learning and economic growth, using data from Hanushek et al. (2008).

slide 18a

12. I then asked how many of the 7th graders could read a sentence: All of them claimed that ability. I then showed data from the Early Grade Reading Barometer on the percentage of 2nd graders in various countries who couldn’t read a single word, which of course predicts future literacy.

13. I talked about what I do specifically, with a few examples (including a few funny stories).

slide 20

14. And finally, I talked them through how I got to my current job and reminded them that it’s not just economists working in international development.

slide 21

Many thanks to all those who gave suggestions. I used several of them and would have enjoyed using others if I’d had more time (either to prepare or with the classes).

The next day, I received a number of thank you notes from students. This one took the cake.

thank you

The impact of student performance labels on later schooling

One key feature of test-based accountability systems in the U.S. is that every student receives not only a test score but also a label based on their performance. Massachusetts, the state that we study, assigns students labels of Failing, Needs Improvement, Proficient, or Advanced by determining cut-points with which it divides the finer-grained test-score distribution into performance regions. … These labels provide no additional information beyond the test scores on which they are based; they are simply coarse summaries of a student’s performance. We focus on responses to labels that have no state-defined consequences for students.

This, from a paper by John Papay, Richard Murnane, and John Willett, in the Journal of Human Resources.

Using a regression-discontinuity design, we find persistent effects of earning a more positive label on the college-going decisions of urban, low-income students.

Here’s an open-access, earlier version of the paper. And here’s the key figure, on the left comparing students with an “advanced” versus a “proficient” label, and on the right, comparing the “proficient” to the “needs improvement” label:

advanced-proficient cutoff


Is it possible to travel by bus from Washington, DC, to Antarctica? Nearly.

black penguinOn January 1, 2010, Andrew Evans boarded a city bus in Washington, DC. His goal was to travel by bus all the way to Ushuaia, Chile, where he would board a ship to Antarctica — if he made it in time! No tickets were purchased in advance, and the route was laid out only vaguely. In The Black Penguin, Evans tells his story, and what a ride it is! As Publishers Weekly puts it, “Sketchy border guards, close calls with violence and natural disasters, and intriguing characters fill vignettes that range from hair-raising to hilarious.” Evans engages the people around him consistently and introduces us to a steady stream of wonderful friends, along with a few foes. His trip across the Americas is interwoven with memories of growing up gay in a conservative faith (Mormon) and the strain that placed on him in middle America, in his church, and with his family.

Here’s a taste of Evans’s preparation, demonstrating his mastery of the fake wallet technique familiar to travelers:

I dressed carefully, stashing fifty dollars in the bottom of one shoe, then a hundred dollars folded tightly into a removable waistband under my pants, and then another fifty in my shirt pocket. Then I packed an old wallet with cancelled credit cards, old student IDs, and a wad of five-rupee notes from India sandwiched between two twenty-dollar bills. If anyone did steal my wallet, they would truly steal trash. I had been mugged once before — in Kiev — and the thieves walked away with about a million dollars of defunct Zimbabwean currency.

And here’s a bit of drama in Bolivia:

It happened in slow motion — the stove-size chunk of granite dropping right down in front of us, crushing the truck in the opposite lane and toppling it over, followed by a rush of rocks and debris. The hard brake jerked us forward, stopping at the pile of rocks. Our bus driver leapt down from the bus and began kicking at the windshield of the wrecked truck, turning the glass opaque with stars. Gripping the rubber lining with his bare hands, the driver peeled away the entire windshield, then reach inside and yanked the keys out of the ignition.

I’ve ridden a number of buses in my time (from Provo to Boston, from Nairobi to Dar es

Andrew Evans's journey to Antarctica
Andrew Evans’s journey

Salaam, from Busia to Kampala, from Maceió to Recife), and Andrew’s vivid descriptions took me back to my own bus rides. I remember riding a matatu in rural western Kenya: When the bus got a flat tire, there was no jack, so the conductor asked all the male passengers to lift the minibus while he and the driver changed the tire. Evans writes: “To travel is to know the unfairness of the world, time and time again.” It’s true, and yet traveling by bus also — for a brief period — brings people from a wide range of social groups into contact. Evans spins that into a compelling, moving narrative. I wouldn’t miss this travel memoir.

Full disclosure: Andrew Evans is my cousin and dear friend. I even show up briefly in the book: After Andrew hitches a ride on a milk truck in Costa Rica, the driver offers him a beverage: “Though I had broken the rule about accepting rides from strangers, I was still not accepting drinks from strangers. That’s exactly how my cousin got drugged and robbed on a bus in Uganda.” That’s me!

To improve educational outcomes, help households to smooth consumption throughout the year

A new paper by Paul Christian and Brian Dillon poses this question: “Does a consistently seasonal diet during childhood have long-run effects on human capital formation?” They use Tanzania’s Kagera Health and Development Survey — a 19-year panel survey — to answer the question. As you can see from the figure below, Tanzania has dramatic seasonality: Children have very different access to food in some parts of the year than in others.

histogram of seasonality

Christian and Dillon develop a structural model — which you can read all about in the paper — and use the household data to estimate it.

Here is a taste of the results:

We find a robust, negative relationship between consumption seasonality and human capital formation. Across specifications, the negative relationship between seasonality and human capital is 30-60% of the magnitude of the positive relationship between average consumption and human capital (in the same units). … The effects of seasonality on height is greatest for children in utero and during infancy, during the critical first 1,000 days of life. Effects on education are most pronounced for older children, suggesting that behavioral channels such as dropping out of school to help on the farm are more important in this sample than early life impacts on cognitive performance. When we further allow for heterogeneity by both age and gender, we see that the height effects during infancy are concentrated among girls, while the education effects during adolescence are largely driven by boys.


How to improve the quality of education in Africa? Start with the evidence from Africa!

A new study was just published in the Review of Educational Research:  Identifying Effective Education Interventions in Sub-Saharan Africa: A Meta-Analysis of Impact Evaluations, by Katharine Conn from Columbia University. Here is a subset of findings (emphasis added):

I identify educational interventions with an impact on student learning in Sub-Saharan Africa. After a systematic literature search, I conducted a meta-analysis synthesizing 56 articles containing 66 separate experiments and quasi-experiments and 83 treatment arms…. A key finding is that programs that alter teacher pedagogy or classroom instructional techniques had an effect size approximately 0.30 standard deviations greater than all other types of programs combined. Limited evidence further suggests that pedagogical programs that employed adaptive instruction or teacher coaching were particularly effective

In case you don’t have access, the earlier, open-access dissertation version has the same sample and the same findings reported in the abstract. 

That version was one of six reviews that Anna Popova and I synthesized in our paper What Really Works to Improve Learning in Developing Countries? An Analysis of Divergent Findings in Systematic Reviews (open-access version).