The Promise of Teacher Coaching and Peril of Going to Scale

This, from Matt Barnum’s review of a recent meta-analysis on teacher coaching, by Kraft, Hogan, and Blazar.

First, what is coaching?

Teacher coaching involves “instructional experts work[ing] with teachers to discuss classroom practice in a way that is” individualized, intensive, sustained, context-specific, and focused.

What’s the finding?

“We find large positive effects of coaching on teachers’ instructional practice,” the authors write… Similarly, the 21 papers that looked at student achievement found notable positive results, on average.

But before you get too excited

When the research examines large-scale programs (with more than 100 teachers involved), the benefits, relative to small coaching initiatives, are cut roughly in half. 

Read the whole article or the meta-analysis itself.

Cash Transfers and Health: Evidence from Tanzania

My paper, “Cash Transfers and Health: Evidence from Tanzania” (with Holtemeyer & Kosec of IFPRI) has been published in the World Bank Economic Review. The paper is attached. Here is the abstract:

How do cash transfers conditioned on health clinic visits and school attendance impact health-related outcomes? Examining the 2010 randomized introduction of a program in Tanzania, this paper finds nuanced impacts. An initial surge in clinic visits after 1.5 years—due to more visits by those already complying with program health conditions and by non-compliers—disappeared after 2.5 years, largely due to compliers reducing above-minimal visits. The study finds significant increases in take-up of health insurance and the likelihood of seeking treatment when ill. Health improvements were concentrated among children ages 0–5 years rather than the elderly, and took time to materialize; the study finds no improvements after 1.5 years, but 0.76 fewer sick days per month after 2.5 years, suggesting the importance of looking beyond short-term impacts. Reductions in sick days were largest in villages with more baseline health workers per capita, consistent with improvements being sensitive to capacity constraints. These results are robust to adjustments for multiple hypothesis testing.

This is a deep analysis of the health investments and impacts stemming from cash transfers in Tanzania. Here are some other resources from the same experiment:

  1. An open access working paper version of the attached paper is available here (which is substantively the same as the published version), and a summary blog post is here.
  2. A broader analysis of program impacts (beyond health) is available here. A quick summary of those results is available here.
  3. All of the data from the Tanzania community-based conditional cash transfer evaluation are available here.

Why scaling up fails

Scaling up a successful small-scale program involves changes to the program. In some cases, that includes a shift in providers, from individuals employed by a private agency to civil servants. This could have both a quality effect (moving from specialists to generalist civil servants) and a motivation effect — both intrinsic, since specialist agencies might employ people who care more, and extrinsic, since non-government agencies might find it easier to fire people.

Lisa Cameron and Manisha Shah have a new paper that examines the scale-up of a sanitation program in Indonesia. From the abstract:

This paper evaluates the effectiveness of a widely used sanitation intervention, Community-Led Total Sanitation (CLTS), using a randomized controlled trial. The intervention was implemented at scale across rural East Java in Indonesia. CLTS increases toilet construction, reduces roundworm infestations, and decreases community tolerance of open defecation… We also examine the program’s scale up process which included local governments taking over implementation of CLTS from professional resource agencies. The results suggest that all of the sanitation and health benefits accrue from villages where resource agencies implemented the program, while local government implementation produced no discernible benefits. [emphasis added]

Okay, so when the government took over, the program didn’t work. Why? They explore a number of mechanisms. The data suggest that the problem is NOT the quality of the facilitators: 

In the field one hears a lot about the importance of the “quality” of the facilitator. In order to test whether the RA facilitators are “better” than the LG facilitators, we collected information from respondents on their perceptions of how charismatic/persuasive the facil- itators were… There is no significant difference in the average reported persuasiveness of the facilitators.

What else?

The intensity of implementation is…greater in RA villages (driven by facilitators making more visits).

More people had heard about the program in RA villages, and “RAs appear to be more effective at reducing tolerance to open defamation.”
This points to motivation rather than quality, in this particular case.

The paper reminds me of Kerwin & Thornton’s work on teacher training in Uganda, in which a “full” version of the program had large impacts on student learning, whereas a lower cost version used government employees (whose job is to train teachers) as well as fewer materials: 

A cost-effectiveness comparison of the two programs reveals the low-cost version to be slightly more cost-effective than the full-cost one… However, focusing on the “headline” measure of letter name knowledge hides significant drawbacks to the low-cost version of the program: the cost-effectiveness result is reversed when considering the overall reading score index, and the low-cost version of the program causes a small (but statistically-insignificant) decline in students’ English speaking ability… Most concerningly, the low-cost program causes large and statistically-significant reductions in several aspects of writing ability – of about 0.3 SDs – relative to the control group. In contrast, the full-cost version of the program improves writing scores across the board, with the effects on several exam components being statistically significant.

In that case, reduced inputs may also play a role. But it is potentially additional evidence that a shift in implementer can have a major impact. 

Bold et al. find something related when examining the scale-up of a contract teacher intervention in Kenya: 

What constraints arise when translating successful NGO programs to improve public services in developing countries into government policy? We report on a randomized trial embedded within a nationwide reform to teacher hiring in Kenyan government primary schools. New teachers offered a fixed-term contract by an international NGO significantly raised student test scores, while teachers offered identical contracts by the Kenyan government produced zero impact. Observable differences in teacher charac- teristics explain little of this gap. Instead, data suggests that bureaucratic and political opposition to the contract reform led to implementation delays and a differential interpretation of identical contract terms.
[emphasis added]

Last week, on the World Bank’s Development Impact blog, I wrote about an experience with scaling up an education pilot in Kenya where the pilot was explicitly implemented using existing government programs, where government actors are playing roles already included in their job descriptions. The pilot was effective, and results on the scale up come in later this year. Fingers crossed.

What do researchers owe their participants?

Back in the 1990s, a group of sex workers in Nairobi, Kenya, seemed to be immune to HIV. Scientists have drawn repeatedly on that group in ongoing efforts to develop a vaccine. Today I read a news piece by Melanie Gosling, reporting that, “Nairobi sex workers…want to establish a code of conduct for researchers in an attempt to get some benefit from the decades of studies they have taken part in.”

It highlights the tension in research, which principally benefits people other than those providing data (whether through words or blood samples). One researcher quoted is Gosling’s piece put it bluntly: “Researchers are not there to solve their problems.”

And yet, those providing data don’t always have a clear understanding of that. As historian Melissa Graboyes has said, “People regularly mistake the idea that they are participating in an experiment and it’s designed to benefit them personally, rather than the experiment is designed to generate data that can be used to answer important questions and hopefully get us closer to solving some important problems.”

That’s consistent with a quote from the piece on Kenyan sex workers:

A male sex worker, who wanted to be known as Jonathan, said one of several problems they had with the research was that the consent forms were difficult to understand.
“So you sign contracts you don’t understand. Then maybe they give you drugs that make you drowsy or dizzy, but you didn’t know this,” he said.

Part of the solution to that immediate problem of understanding is simple, clear consent forms. As Rachel Glennerster, Executive Director of J-PAL, tweeted: “I favor clear simple forms, not long complex ones that U.S. health regulations require” (edited slightly to translate from Twitter-ese).

But this points to more than that. Participants in studies provide data. Hopefully that data benefits someone besides the researchers involved. Hopefully it contributes to better policies. But chances are that it won’t benefit the participant directly, at least not much. An obvious solution is to pay participants in some form. But how do you identify the right amount?

Some researchers ask her [one of the sex workers] for help, such as “mobilise five or ten ladies and we will pay you”. After spending time doing this, she is given “a small token, maybe enough to find food to eat that day. But we are making their work easy”.

The alternative that Nairobi’s sex workers are proposing is a novel one: Organize. Then they can collectively negotiate what they see as appropriate compensation for participating in research.

How do you believe that researchers should benefit their research participants?

Thrilling roller coaster with a long wait — a review of Kwei Quartey’s Gold of Our Fathers

Kwei Quartey writes mysteries that give a window into current Ghanaian social issues. Some authors with African ties — Quartey was born in Ghana and grew up between Ghana and the US — cringe at the assumption that they are writing “ethnographic texts dolled up as literary fiction,” as Taiye Selasi put it. Quartey — to the contrary — embraces that role in his crime novels: His first book with detective Darko Dawson of the Ghana Police Service explored polygamy and traditional religion, his second dealt with street kids, his third was set in the oil industry, and — in Dawson’s fourth outing — we come to the gold mining industry in Gold of Our Fathers.

Dawson is a highly capable detective. He’s not corrupt. He’s faithful to his wife. (He almost cheated in the last book, but he just couldn’t do it!) He occasionally smokes pot but he’s trying to kick the habit, with broad success. He’s so capable that at the beginning of this book, he is transferred far from the capital, Accra, to fill in for a deceased officer in Obuasi, a rural gold-mining district in central-southern Ghana. As soon as he arrives a Chinese immigrant miner is found murdered. Dawson is on the job!

My favorite Darko Dawson novel is the second, Children of the Street. This one, by contrast, felt like two novels in the one. The second half is captivating. The first half is paced very slowly (it took me weeks to work my way through) and has far too much language that sounds like “teaching Westerners about Ghana,” as in this exchange:

“How far, boss?” the sergeant asked, slang for “How goes it.”

or this one

Dawson: “I’m broke—can you mobile me a little cash?” 

Dawson’s wife Christine: “Okay—I’ll send what I can by MTN Money.”

Presumably Dawson knows how he and his wife send money and she wouldn’t clarify. There are many other examples. 

I’ve never been to Ghana, and maybe this won’t bother you if you haven’t either, but I prefer the unapologetically immersive style that lets readers catch up on local culture rather than bringing them along by the hand.

If you haven’t read Darko Dawson, start with the first or second book. This book is a great ride if you don’t mind waiting a while to get to it.

Magic, mayhem, and inspiration — A review of Tove Jansson’s Finn Family Moomintroll

A few months ago, I was in Finland, and I asked two education professionals over dinner, “What are the books that every Finn has read, whether in school or outside?” They gave me two: Seven Brothers, by Aleksis Kivi (published in 1870), and the Moomin books, by Tove Jansson (published between 1945 and 1993).
moomin and snufkin
When I returned home, I picked up Finn Family Moomintroll from the library and read it aloud with my 6-year-old daughter. It is delightful and crazy and continually surprising. Moomintroll is the protagonist, a good-natured, roundish creature who lives in Moomin Valley with his mother, his father, and a whole pile of other creatures. Early on, Moomintroll and his friends Snufkin and Sniff find a hat. Back home, they use it as a wastebasket and discard some eggshells in it. The eggshells then turn into mini-clouds that Moomin and his friends are able to fly around on. All kinds of other adventures ensue. At one point, a character discovers a significant amount of gold.
“I wonder what you’re all thinking of doing with the Snork’s gold?” said Snufkin.
“I think we shall use it to decorate the edges of the flower beds,” said Moominmamma, “only the big bits, of course, because the little ones look so rubbishy.”
Two creatures — Thingumy and Bob — show up who speak largely in spoonerisms: “‘Don’t nake any totice,’ whispered Bob.” Ultimately, a magician called the Hobgoblin shows up — it turns out the hat was his — as he realizes that the ruby he was seeking in the valleys of the moon is actually in Thingumy and Bob’s suitcase. The Hobgoblin then grants wishes to everyone, not without a little mischief: The Muskrat had been reading a book entitled On the Uselessness of Everything; his wish is merely for his book to be returned, but when he gets it back,
“‘On the Usefulness of Everything’,” read the Muskrat. “But this is the wrong book. The one I had was about the Uselessness of Everything.”
But the Hobgoblin only laughed.
Peppered throughout the inspired lunacy are these little nuggets of anti-materialism and optimism. I keep wanting to compare these books to others: They’re like Winnie the Pooh but less safe (in the best possible way). They’re like the Smurfs but smart. But it’s ultimately a futile task: the Moomin inhabit their own world entirely. What the book lacks in narrative momentum, it makes up for in inventiveness and hope. I loved it.
Bits and pieces
  • My daughter and I are now reading Tales from Moominvalley (another book in the series), and I’m reading the collected Moomin comic strips that Jansson also wrote to all my children together.
  • Moominmamma on education: “Moomins go to school only as long as it amuses them.”

How to read a lot more books

While hopefully not as annoyingly as Aziz Ansari’s Saturday Night Live character The Bookworm, many of us would like to read more.

Two recent articles highlight how to read more. If you want all their tips, read the articles. Here, I’ll highlight those that resonate with me. In Neil Pasricha’s 8 ways to read (a lot) more books this year, he talks about going from five books a year to 50. He recommends the following steps among others; I just chose my favorites. The bolded words are his; the rest is my editorializing.

  1. Centralize reading in your home.  He suggests having a bookshelf rather than a TV at the center of your living space.
  2. Make a public commitment. Among some of my siblings, we usually run some sort of reading challenge from year to year. Sometimes it’s a bingo board where you try to reach books in a wide range of categories. This year, we’re trying to each read at least ten books by non-U.S. authors.
  3. Reapply the 10,000 steps rule. Here, he refers to the idea of walking 10,000 steps, taking advantage of every opportunity. “When do I read now? All the time. A few pages here. A few pages there. I have a book in my bag at all times.”

He shares a wonderful (undocumented) anecdote about author Stephen King:

A good friend once told me a story that really stuck with me. He said Stephen King had advised people to read something like five hours a day. My friend said, “You know, that’s baloney. Who can do that?” But then, years later, he found himself in Maine on vacation. He was waiting in line outside a movie theater with his girlfriend, and who should be waiting in front of him? Stephen King! His nose was in a book the whole time in line. When they got into the theater, Stephen King was still reading as the lights dimmed. When the lights came up, he pulled his book open right away. He even read as he was leaving.

In Charles Chu’s In the time you spend on social media each year, you could read 200 books, he talks about reading 200 books each year. One tip is to “go multi-medium.” Chu writes:

If your goal is to read more, you can’t be picky about where you read or what mediums you use. I read paper books. I read on my phone. I listen to audiobooks. And I do these things everywhere—on park benches, in buses, in the toilet… Wherever I can. Make your reading opportunistic. If you have a chance, take it. If you don’t have a chance, find one.

I’ve found this to be my single best strategy for reading a lot. Right now I’m reading an ebook on my phone, listening to an audiobook on my phone, listening to another audiobook on my swimming iPod (you know, so I can swim and listen to books at the same time), and reading one print book around the house. That way I can read anytime, anywhere.

Chu also recommends stepping away from social media and TV. In his not-at-all-judgmental way, he writes:

Here’s how much time a single American spends on social media and TV in a year: 608 hours on social media. 1642 hours on TV. Wow. That’s 2250 hours a year spent on TRASH. If those hours were spent reading instead, you could be reading over 1,000 books a year!

Unlike Mr. Chu, I enjoy my TV consumption and my social media use, and I learn from them, like this bit I learned from Joan Cusack the TV show A Series of Unfortunate Events:

32654688170_227b8dfa13_o

That said, I’d probably be happier and wiser with a little less time on Facebook and more with my nose buried in a book. I remember visiting the wonderful Tinkertown Museum in Albuquerque, New Mexico. Over 40 years, Ross Ward carved and collected his way to an amazing museum collection of wooden figures. Near the exit, I saw this sign.

ward television

[Source]

So my aspiration isn’t to carve a museum of wooden figures, but I do like reading books. Maybe I’ll read or listen to 50 this year.

Happy reading!

Bonus reading: