The Factory of Policy Relevant Education Research that Is Ben Piper

piper

Ben Piper is an education researcher with at least three outstanding qualities. He’s incredibly productive, with at least 11 publications in the last 12 months. His work is deeply policy relevant, answering questions that policymakers are asking: How can we implement teacher coaching at scale? How much scripting in learning is helpful for teachers? And he’s a really nice person, per my several interactions with him.

Here’s a cursory review of those last 11 publications. You can read about more here.

  1. How much does learning one language help you learn another? In Kenya, comparing kids instructed in Kiswahili to those instructed in English reveals that it goes both ways. “Cross-language transfer of reading skills: an empirical investigation of bidirectionality and the influence of instructional environments” (with Kim)

 

  1. Is it possible to scale up an effective education pilot? In Kenya, a scaled program showed impacts on learning. This paper documents how. “Scaling up successfully? Lessons from Kenya’s Tusome national literacy program” (with Destefano, Kinyanjui, & Ong’ele)

 

  1. In a decentralized system, how do countries provide resources for early child education programs? Qualitative evidence from Kenya “Scaling Up Early Childhood Development and Education in a Devolved Setting: Policy Making, Resource Allocations, and Impacts of the Tayari School Readiness Program in Kenya” (with Merseth & Ngaruiya)

 

  1. After comparing three models for improving literacy in Kenya, a combination of “PD, teacher instructional support and coaching, 1:1 student books, and structured teacher lesson plans” was most effective AND most cost-effective. “Identifying the essential ingredients to literacy and numeracy improvement: Teacher professional development and coaching, student textbooks, and structured teachers’ guides” (with Zuilkowski, Dubeck, Jepkemei, & King)

 

  1. Across 13 countries & 19 projects, “structured teachers’ guides improve learning outcomes, but that overly scripted teachers’ guides are somewhat less effective than simplified teachers’ guides that give specific guidance to the teacher but are not written word for word for each lesson in the guide.” “Effectiveness of teachers’ guides in the Global South: Scripting, learning outcomes, and classroom utilization” (with Sitabkhan, Mejia, & Betts)

 

  1. In the era of free education, why do parents choose private schools? “Drawing on parent survey and interview data, as well as interviews with national policy makers, we found that parents who chose LCPS for their children were more driven by quality concerns than were public school parents.” “Parents, quality, and school choice: why parents in Nairobi choose low-cost private schools over public schools in Kenya’s free primary education era” (with Zuilkowski, Ong’ele, & Kiminza)

 

  1. Are you a big fan of mother-tongue instruction? In Kenya, “assignment to the mother-tongue group had no additional benefits for English or Kiswahili learning outcomes beyond the non-mother-tongue group, and that the mother-tongue group had somewhat lower mathematics outcomes.” “Examining the secondary effects of mother-tongue literacy instruction in Kenya: Impacts on student learning in English, Kiswahili, and mathematics” (with Zuilkowski, Kwayumba, Oyanga)

 

  1. Summer reading loss: Not just for rich countries anymore. Big losses during school breaks in Malawi. “Is summer loss universal? Using ongoing literacy assessment in Malawi to estimate the loss from summer breaks” (with Slade, Kaunda, King, & Ibrahim)

 

  1. Is it possible for a large-scale education technology program to work? In Kenya, tablets went to 1,200 instructional coaches. This study documents usage and effectiveness. “Implementing large scale instructional technology in Kenya: Changing instructional practice and developing accountability in a national education system” (with Oyanga, Mejia, & Pouezevara)

 

  1. Here’s how we’re working to use research to affect large-scale education policy “Improving learning in Sub-Saharan Africa using rigorous research designs

 

  1. How does teacher coaching affect teachers’ attitudes and practices in Kenya? (I’m not sure, since this one ) “Instructional coaching in Kenya: Supporting teachers to improve literacy outcomes” (with Zuilkowski)

Ben and his co-authors are working in real time, in the real world, which means that not every study has a massive sample or a perfectly clean identification strategy. (Many do, but not all.) But policymakers are making decisions in real time, in the real world, and Ben’s work contributes to better policymaking. Don’t get me wrong: I love big samples and perfectly clean identification strategies. But I also posit that we can learn a lot from different kinds of studies.

You might have noticed Stephanie Zuilkowski as an author on several of these papers. She’s another extremely active education researcher, with lots of other interesting work. Maybe I’ll be do a full rundown another day, but in the meantime, check out her work here.

Advertisements

What I’ve been reading this month – July 2018

July included a lot of family vacation, which — for me — translates to wonderful memories but less reading time. So it’s a short list for July!

The Goldilocks Challenge: Right-Fit Evidence for the Social Sector, by Mary Kay Gugerty and Dean Karlan — With an increasing emphasis on measuring the impact of non-profits and other pro-social organizations, simple monitoring can get neglected. Yet monitoring systems are fundamental to every organization, for understanding whether they’re delivering the services they intend to deliver. Gugerty and Karlan offer a set of clear principles for monitoring systems that aren’t too burdensome nor too slight, but just right. I wrote a fuller review at the Development Impact blog.

The Regional Office Is Under Attack, by Manuel Gonzales — There’s a team of women assassins. And they’re going up against another team of women assassins (the titular “regional office”), one of whom has a robot arm. There are references to the actual mission of the regional office — say, suppressing “a den of werewolves, or a nest of vampires” or battling someone’s “dead wife from the bowels of hell” or a “demon horde” — but the whole novel revolves around one assassin on each side and their stories. It’s lots of fun, full of pop-culture references (She had “one real option — to ‘Die Hard’ it John McClane style”) and life wisdom (“She’d rather they’d just given her her job to do and not this management position because what a pain in the ass managing people was turning out to be”). I found the pacing imperfect, but I had a great time. NY Times review by Kelly Braffet: “it’s rollicking good fun on the surface, action-packed and shiny in all the right places; underneath that surface, though, it’s thoughtful and well considered.”

When You Reach Me, by Rebecca Stead — This is one of those books that is technically science fiction, but you wouldn’t know it until you’re well into the book. It opens like a 1970s family drama revolving around Miranda, a sixth-grader, and her single mom. Then anonymous notes start appearing with strange requests. It’s all mysterious, maybe even a little bit eerie, but it all comes together in grand fashion. I listened to the audiobook in the car with my whole family, and once we got going, we couldn’t stop. NY Times review by Monica Edinger: “Smart and mesmerizing.”

Incidentally, the children’s classic A Wrinkle in Time plays a role in both The Regional Office and When You Reach Me, although unfortunately not in The Goldilocks Challenge. (Come on, Gugerty and Karlan!)

Give to panhandlers?

Bryce Covert has written a thoughtful, balanced, carefully researched piece on whether to give to panhandlers. “On the whole, all the evidence, from the statistical to the spiritual, points in one direction: if you can give, you should give. It won’t solve the problems of mass homelessness or impoverishment. But it will improve someone’s life ever so slightly and briefly.”

She also quotes Anna Popova’s and my work on how the poor tend to spend cash: “Overwhelmingly, they found that giving cash ‘had no impact on spending on alcohol and tobacco,’ Evans said. ‘In a number of cases, it even seemed to have a negative impact—people spent a lower proportion of their budget on these temptation goods.’”

After that, I go off a little bit on how we needn’t judge the poor’s spending habits, even if they did decide to go and buy a beer: “‘Do we get rid of an effective way of helping the poor just because there are a couple of people who don’t use the money in the way that we think is the most constructive?’ he [Dave Evans] asked. Perhaps, he went on, a trip to the liquor store isn’t necessarily unhelpful. ‘If a poor person wants to buy a beer and that’s going to help them feel better at the end of the day, is that something we should criticize or be concerned about?’”

I’ve often thought: Oh, rather than donate to panhandler, I’ll give the same amount to some organization that helps the poor more systematically. Here’s Covert on that: “As the economists I spoke to pointed out, most people are not likely to take the dollar they would have otherwise given a panhandler and donate it to a nonprofit later. And while service organizations do a lot of good, what they do is generally something different than give money directly on the street, one American to another—a service that has its own merit. Just as the man I saw on the median needed something other than what I’d thought to give, there is value in the simple handoff of cash in a personal encounter.”

I believe there are multiple defensible stances on what to do when someone asks for money on the street. But Covert uses evidence and reasoning to rule out those stances based on false presumptions about the poor.  Read her article. It’s much better than my quotes.

What I’ve been reading this month – May/June 2018

PachinkoPachinko, by Min Jin Lee — This captivating epic follows a Korean family over decades during the Japanese occupation of Korea, migrating to Japan early in the novel. Tash Aw (Guardian): “A long, intimate hymn to the struggles of people in a foreign land.” Krys Lee (NY Times): “Each time the novel seems to find its locus — Japan’s colonization of Korea, World War II as experienced in East Asia, Christianity, family, love, the changing role of women — it becomes something else. It becomes even more than it was.” I loved everything except when a character warned me, “If you like everything you read, I can’t take you that seriously.” Hey, I don’t like everything I read.

freshwaterFreshwater, by Akwaeke Emezi — When Ada is born in southern Nigeria, she brings a spirit along with her. Not her spirit, though. It’s another, who shares control of her thoughts and actions. Over time, more spirits appear, one after a sexual assault in college. The result is wild swings of personality and shifts in gender identity. Emezi (who has told their own story of gender dysphoria) provides a potent voice to depression and despair. Emezi writes, “After you have let the wilderness in you come out and play, after you have spilled your darkness in front of a stranger, it can be difficult to look at them in the sentience of daylight.” And later, “it was as if staying alive just gave everyone else time to leave you.” I never completely got lost in the story emotionally, but I respected it. Susan Straight (LA Times): “Emezi’s lyrical writing, her alliterative and symmetrical prose, explores the deep questions of otherness, of a single heart and soul hovering between, the gates open, fighting for peace.” Tariro Mzezewa (NY Times): “Remarkable and daring… “Freshwater” builds slowly, but that only crystallizes how fractured Ada and her personalities are. As the voices in her head get louder and grow hungrier, the story gains momentum.”

dionne 1Doomed Interventions: The Failure of Global Responses to AIDS in Africa, by Kim Yi Dionne — Through a healthy mix of ethnographic fieldwork, original survey collection, and large-scale survey analysis, Dionne shows that international donors care a lot more about HIV/AIDS than African do. And yes, that includes Africans who have family members with HIV. One reason is that even Africans who are infected with HIV share a lot of their needs with their neighbors — clean water and good jobs, to start. “By privileging donor priorities over citizen priorities, global elites cripple states’ abilities to implement policies representing citizens’ interests.” The title is overstated: Internationally funded AIDS treatment has saved countless lives. But “looking at our failures to improve the human condition can help us formulate better strategies and approaches going forward.”

therecoveringThe Recovering: Intoxication and Its Aftermath, by Leslie Jamison — I just loved this mix of recovery memoir, an exploration of the historical link between addiction and literary creativity, the racial and class divide of addiction public policy, and more. “All my life I had believed — at first unwittingly, then explicitly — that I had to earn affection and love by being interesting, and so I had frantically tried to become really…interesting.” A little sprawling but I really enjoyed it. As an added bonus, development economists Dean Jamison and Julian Jamison make cameos in the book. “My oldest brother, Julian, taught me how to solve an equation for x when I was seven. ‘Great,’ he said, ‘but can you solve when x is on both sides?”

three shadowsThree Shadows, by Cyril Pedrosa (translated from French by Edward Gauvin) — How far will a parent go to save their child from death? This is an urgent little fable in a graphic novel format. “In this our springtime there is no better, there is no worse. Blossoming branches burgeon as they must. Some are long, some are short.”

Doughnut Economics: Seven Ways to Think Like a 21st-Century Economist, by Kate Raworth — Raworth proposes that the field of economics has led to a narrow global focus on economic growth, and that instead we should focus on a social foundation for all the world’s citizens, as well as protecting the environment. The book is filled with nuggets on the history of economic thought (“At the end of the nineteenth century, the sociologist and economist Thorstein Veblen berated economic theory for depicting man as a ‘self-contained globule of desire'”) and the power of a good diagram (“if drawing new pictures sounds frivolous to doughnutyou — like mere child’s play — believe me it is not”). Raworth includes lots of policy proposals (including LOTS of taxes) with little consideration of the incentive effects of taxes, and I sometimes felt like she was attributing self-interest itself to economics — whereas I’m inclined to believe it predates the discipline. I also didn’t feel like she reckoned with how attractive a single goal — or a handful of goals — is to get your mind around (more stuff — i.e., GDP growth! less poverty!) as opposed to a long list of items (social equity! clean water! housing! protect ozone! something about phosphorous!). Still, I admire her consistent optimism, boldness, and creativity, and as she quotes the poet Taylor Mali, “Changing your mind is one of the best ways of finding out whether or not you still have one.”

crazy rish asiansCrazy Rich Asians, by Kevin Kwan — An untenured economics professor at NYU takes a ten-week summer vacation during which she doesn’t once think about her research? Now THAT sounds like fantasy! Seriously, it’s a fun, light book about a Chinese-American economics professor who doesn’t realize that her boyfriend — also a professor — is one of the wealthiest (and sought-after) men in Singapore until she joins him on a summer vacation. Lots of drama ensues!

Truly Madly Guilty, by Liane Moriarty — Three families attend a barbecue in the suburbs of Sydney, Australia. Something bad happens. The effects ripple. This multi-family drama reveals itself bit by bit with great humanity. Moriarty truly madly guiltyemploys rotating narrators but focuses on two women who have been uncomfortable friends since childhood. On childhood: “No one warned you that having children reduced you right down to some smaller, rudimentary, rudimentary, primitive version of yourself, where your talents and your education and your achievements meant nothing.“ On white collar malaise: “Sam felt himself break out in a cold sweat at the thought of how little he was achieving at work. He had to get something done today. This couldn’t go on much longer. He was going to lose his job if he didn’t find a way to focus his mind.”

chastGoing into Town: A Love Letter to New York, by Roz Chast — Chast — whogoing into town - detail you might know from her New Yorker cartoons — writes that “this book is a sort-of guide and also a thank-you letter and a love letter to my hometown and New Yorkers everywhere.” It’s fun and useful. I laughed aloud repeatedly, and I will surely give a copy to future New York-bound friends. Chast loves New York! She shows that you can do ANYTHING in that city.

shadeShade the Changing Girl (Volume 1): Earth Girl Made Easy, by Cecil Castellucci et al. — Okay, so a bird creature (Loma Shade) from another planet steals a “madness coat” from a museum and uses it to take over the body of an teenage earth girl in a coma. It turns out that the earth girl was manipulative and mean, and people on Loma’s home planet are after her — they want that coat! Occasionallyshade - detail confusing but gorgeous visuals and fun story, with a healthy appreciation of poetry: “There is a poem for every feeling. It’s what gets me through when fear threatens to overwhelm.” Included among the “best graphic books of 2017.”

Binti: The Night Masquerade, by Nnedi Okorafor — In this, the climax to binti masqueradethe Binti trilogy, Okorafor fuses taut, tragic action and surprises while furthering the themes of alienation (in this case, with actual aliens!) and personal growth.

Ms. Marvel: Mecca, by G. Willow Wilson et al. — If you haven’t read the meccaadventures of Kamala Khan — aka Ms. Marvel, Pakistani-American Muslim superhero — then go do it right now! Start with No Normal. Mecca — volume 8 in the series — continues the tradition!

Harry Potter and the Half-Blood Prince, by J.K. Rowling — I just read this harry potterto a couple of my kids. It’s still a great read!

Imagine Wanting Only This, by Kristen Radtke — “There is little linear plot in Imagine Wanting Only This, even though each of the eight chapters finds Radtke at a slightly different stage of her life … becoming obsessed with the history of ruins and disasters. At each point, she seeks answers to her nagging life questions while alsoimagine attempting to escape her reality” (Arnav Adhikari in The Atlantic). I had trouble engaging with the existential questions at the heart of this book, but Radtke captures perfectly the reason that I go to sleep listening to things — to escape the million thoughts in my head (see picture). Included among the “best graphic books of 2017.”
imagine - detail

Advice for impact evaluations with government: Drop the baseline

karthik2

This was a case where we did randomization without a baseline. I highly recommend this when you’re working with a government because the biggest risk is implementation failure. You’ll spend a lot of time doing the baseline – spend time, spend money – and have the intervention not be implemented. So when you’re working with the government, it’s better to get power by doubling the sample of your endline and just randomized with administrative data so you’ll get the same amount of power but you reduce risk up front.

That is Karthik Muralitharan speaking at the RISE conference today. Of course, he didn’t have to say that doubling your sample also increases your likelihood of randomization resulting in balanced intervention and comparison groups, thus making a baseline less necessary.

Update: This prompted a very active discussion on Twitter, which you can read in full here. Below are a few points.

Ultimately, there are a number of factors to consider — the potential sample size, the probability of implementation failure, the importance of baseline covariates for your analysis. But still, where there is serious concern that the program may not be implemented as expected — and especially if there are decent administrative data — it’s worth consideration. I’ll give the penultimate words to Karthik’s co-author, Abhijeet Singh.

First, responding to Pauline and Andrew’s point.

Second, to Cyrus and Seema’s points.

And the last word to Karthik himself.

 

There are many more comments, but I won’t embed them all here. You can read the full conversation here on Twitter.

Activity for teaching regression discontinuity design

One method for evaluating the impact of a program is regression discontinuity design (RDD). This works when an intervention (to be evaluated) is assigned based on a score of some sort. For example, a welfare program that is assigned for all households below a certain level of income, or an education program that is assigned to all students above a certain test score (or below a certain test score). In short, this method compares individuals who are just above and below the cut-off for assignment to the program, since they are very similar (except for 1 or 2 points on a test or a small amount of income). You then adjust for those small differences statistically, but the intuition is that you’re comparing people who are very similar, except that one group gets the intervention.

When I teach impact evaluation, an activity where students get up and move around can be helpful for more at least two reasons: It can make a point visually, and it can keep people from falling asleep. Here’s an activity I came up with for demonstrating the concept behind RDD, and it has worked pretty well.

Tell the students that we are evaluating the impact of an injection that is supposed to increase the height of recipients. Every participant under a certain height will receive the injection. How can we evaluate it?

Have all the students line up in a row by height. (With a big class, use a subset of students.) Pick a couple of students of similar (but not identical) height in the middle and explain that this is the height cut-off. The shorter student on the left will receive the injection, and the taller student on the right will not.

Now, if we were to compare the height of the tallest student (to the right of the group) and the shortest student (to the left of the group), we wouldn’t have a good sense of the impact of the injection, since their heights are already so different. But if we compare those who are just below the qualifying height (getting the injection) to those just above (not getting the injection), then differences we observe are likely to be due to the injection.

I’ve done this activity with adults in more than one country, and it’s been effective and fun.

Any ideas for how you’d make this activity better? What activities do you use to teach impact evaluation methods?

The image at the top of this post is from Impact Evaluation in Practice (Second Edition)