Author: David Evans

Michael Kremer on how RCTs lead to innovation

David EvansLeave a comment

The modern movement for RCTs in development economics…is about innovation, as well as evaluation. It’s a dynamic process of learning about a context through painstaking on-the-ground work, trying out different approaches, collecting good data with good causal identification, finding out that results do not fit pre-conceived theoretical ideas, working on a better theoretical understanding that fits the facts on the ground, and developing new ideas and approaches based on theory and then testing the new approaches.

This is from an insightful interview with Michael Kremer, Harvard economics professor “generally given credit for launching the RCT movement in development economics with two experiments he led in Kenya in the early 1990s,” and my graduate school advisor.

The interview is in Tim Ogden’s book Experimental Conversations: Perspectives on Randomized Trials in Development Economics.

 

How Steve Levitt convinced me to give my son extra screen time right now

David EvansLeave a comment

Alternative title – Quick take: “The Behavioralist Goes to School,” by Levitt et al.

This evening my middle-school son was negotiating for additional screen time, and I proposed that he receive that additional time based on incremental improvement of his grades. But we had just discussed Levitt et al.’s recent paper testing loss aversion with student incentives over dinner, so he opted to start with the extra screen time and then lose it if the grades failed to improve. (His grades are pretty good anyway; just sayin’.) And I couldn’t not go with the evidence, so this had better work.

Back to the paper: Do students respond to incentives if the reward comes right away? Do they respond to non-monetary incentives? Does offering those incentives once crowd out intrinsic motivation?

If those are your questions, then Levitt et al. have some answers in “The Behavioralist Goes to School: Leveraging Behavioral Economics to Improve Educational Performance.” Here’s the abstract:

We explore the power of behavioral economics to influence the level of effort exerted by students in a low stakes testing environment. We find a substantial impact on test scores from incentives when the rewards are delivered immediately. There is suggestive evidence that rewards framed as losses outperform those framed as gains. Nonfinancial incentives can be considerably more cost-effective than financial incentives for younger students, but are less effective with older students. All motivating power of incentives vanishes when rewards are handed out with a delay. Our results suggest that the current set of incentives may lead to underinvestment.

Here’s some detail on the motivation from the authors: “One of the biggest puzzles in education is why investment among many students is so low given the high returns. One explanation is that the current set of long-run returns does not sufficiently motivate some students to invest effort in school.”

And here’s a little more detail on the study and results.

They focus on three features of past incentive programs:

  1. There is a time gap between when students exert effort and when they receive the reward.
  2. Rewards are offered as gains (not losses).
  3. Rewards are monetary.

After some proof-of-concept testing, they ultimately run their field experiments with 5,000+ students in Chicago public schools.

On point 1 (the time gap), they use a test where possibility of rewards are announced to students immediately before the test (so it’s a test of immediate effort, not preparation) and the rewards are given immediately after, as compared to a treatment where the reward is delivered a month later. Receipt of the reward is determined by improvement relative to a baseline test several months before. Depending on the setting, the test is a low-stakes diagnostic reading or math assessment.

“We find that large incentives delivered immediately, whether financial [$20 cash] or nonfinancial [trophy worth $3], have a significant impact on test performance of about a tenth of a standard deviation. In stark contrast, rewards delivered with a one month delay have no impact, nor do small financial rewards [$10 cash].” “As far as we know, ours is the first study to demonstrate that student responsiveness to incentives is sensitive to the size of the reward.”

To test point 2 (gains versus losses), they vary whether students receive the reward before the test and then have to return it immediately after testing or they receive the reward after the test. “In the pooled estimates, the coefficients on losses are roughly twice the magnitude of the analogous ‘gain’ treatments, but are not statistically different from those treatments.”

On point 2 (monetary rewards), they test non-monetary rewards – a trophy worth $3 – against the large and small monetary rewards. “In the pooled results, the point estimates for non-pecuniary rewards (framed either as a gain or a loss) are somewhat smaller than those for the $20 treatment and much larger than those from the $10 treatment.”

Gender: “Our findings with respect to gender are consistent with a wealth of prior research that shows boys tend to be more sensitive to short-term incentives than girls, which may be due in part to gender differences in time preferences.”

Age: “In general, we see similar results across young and old students, with the exception of nonfinancial incentives framed as losses, where we find large positive effects on young students and small negative impacts on older students.”

Do these incentives affect subsequent test performance? The low financial incentives (which had no impact in the short run) lead to negative impacts on tests a few months later. The other incentives have no statistically significant impact and have a mix of positive and negative point estimates.

My short take away: Nuance, nuance, nuance. Student motivation is probably largely overlooked, and offering incentives can have a positive effect. But if you’re doing student incentives, test them out before committing at scale. Although List et al. don’t find pervasive evidence of problems after the incentives are removed, they do find a little, and a couple of other studies (here and here) have as well.

Bonus reading – a few other papers on financial incentives for students

Should you worry about your legacy? Lessons from a time-traveling poet, a TV comedy writer, and a crime novelist

David EvansLeave a comment

In a 1916 story entitled “Enoch Soames,” Max Beerbohm recounts the tale of a hack poet (Soames) who frets to his friend (Beerbohm) about his inability to enjoy the fame that will assuredly accrue to him posthumously. If only he could step into the Reading Room of the British Museum one hundred years hence: “Think of the pages and pages in the catalogue: ‘Soames, Enoch’ endlessly.”

Of course, the devil happens to be sitting at the next table and offers Soames the chance to do just that in exchange for his soul. Soames zooms to the Reading Room a century ahead. He checks for himself in the card catalogue. Nothing. He checks a few encyclopedias. Nothing. Finally, he finds a book on “English Literature: 1990-2000.” There he finds himself in the following passage: “For example, a writer of the time, Max Beerbohm, who was still alive in the twentieth century, wrote a story in which he portrayed an imaginary character called ‘Enoch Soames’ — a third-rate poet who believes himself a great genius.” History hasn’t forgotten Soames; it fictionalized him. 

This fear for our legacy recurs: Just yesterday I saw an episode of the Dick Van Dyke show that aired in 1963, where the title character — a TV comedy writer — laments, “All I write are jokes. Nothing I write has any real permanence about it. [It is said] on the television once and it’s gone forever.”

Maybe this desire to be remembered is all overblown. Perhaps the late great detective novelist, Robert B. Parker, had it right. When asked how his books would be viewed in 50 years, he replied, “Don’t know, don’t care.”

But in the present he brought great pleasure to many.

Notes

Quick take: “I failed, no matter how hard I tried”: A mixed-methods study of the role of achievement in primary school dropout in rural Kenya, by Zuilkowski et al.

David EvansLeave a comment

In Kenya, virtually every child enrolls in primary school, but many don’t complete it. Stephanie Simmons Zuilkowski, Matthew Jukes, and Peggy Dubeck use mixed methods to explore why.

Three findings stood out to me:

  1. In interviews with both youth and with parents, the youth (age 14-15, mostly, but some older) were “universally” characterized as the main education decision makers. In many cases, parents encouraged them to stay in school but the youth opted to drop out.
  2. Lower performing youth were more likely to drop out of school. This isn’t surprising but it’s useful to see it quantified. It comes out in both the quantitative and the qualitative work here.
  3. Free primary school isn’t free (and I’m not even talking about pure opportunity cost; I’m talking about simple out-of-pocket costs).

Okay, to the study! They point out why cross-sectional studies may miss the point in understanding dropout rates:

A cross-sectional study may identify proximal factors affecting dropout risk—perhaps pregnancy or the need to work for pay (Ball 2012)—but not the earlier factors that put the child on the trajectory toward dropout. In interviews with parents and teachers, proximal reasons for dropout may become the post-hoc rationale for a child’s dropout obscuring the underlying trigger factors.

Finding 1: Who decides on dropouts?  Admittedly, it’s a small sample for this part: They spoke with 21 youth and 20 parents. In most cases, the interviews were conducted separately. Of the youth, half had dropped out. Here is the key finding: “In our interviews with the dropouts in this sample, the youth were described universally as the principal educational decision-makers, both by the parents and by the youth themselves.” Notably, both youth and parents talked about the importance of education. “The stories of all 11 children who dropped out began with some variation of: ‘I wasn’t doing well in school.’” Many of the quotes highlight relative performance and the inability to get extra help. To me, this points back to the importance of structuring education systems that help teachers to teach to the right level (see here and here for more on that). Many of these children simply weren’t getting instruction at their current level.

Finding 2: “A student with a literacy composite score one standard deviation above average would have fitted odds of dropout that are 40% lower than those of the average scorer. A student with a numeracy score one standard deviation above average would have fitted odds of dropout that are 17% lower than those of the average scorer.” N=2,500+

Finding 3: “Despite the official abolition of school fees, all 13 schools the sampled youth attended had charged fees for extra teachers, books, or materials. Nine of the 21 interviewees—five students and four dropouts—said they had been sent home to get money for fees or materials. Children who could not gather the required amounts were not generally allowed back in class.”

I recommend the paper.

my genre fiction is telling me i should be reading literary fiction

David Evans1 Comment
About halfway through Robert Parker’s School Days, the detective Spenser sits down for a drink with his client, the elderly Lily Ellsworth:

“You seem an honest man, sir,” she said.
“‘Let be be the end of seem,'” I said.
She smiled faintly.
“‘The only emperor,'” she said, “‘is the emperor of ice cream.'”
“Very good,” I said.
“My generation read, Mr. Spenser; apparently yours did, too.”
“Or at least I did,” I said. “Still do.”

Here, Spenser is quoting Wallace Stevens’ 1922 poem, “The Emperor of Ice Cream,” and Lily responds with the next line of the poem. The subtext — by my reading — is that in their generations, they read literature. (They’re not, after all, quoting Stephen King or V.C. Andrews.)

Two ironies stand out. The first is that Spenser actually misquotes the original poem; the line is “Let be be finale of seem” (which is less intuitive to the modern ear, but so be it). The second, of course, is that I’m reading this in a pulpy detective novel.

And yet, I adore Parker’s Spenser novels. I don’t read any other writer who matches Parker for witty dialogue. I haven’t read one in a while, and I found this on the free book shelf at the library in a beach town this summer, and reading it was like spending a few hours with a dear old friend. Spenser characterizes himself well: “I am persistent, and fearless, and reasonably smart.”

A few other quotes that I enjoyed:
  • On persistence: “Keeping at it is one of my best things.”
  • On overdoing things: “A thing worth doing…was worth overdoing.”
  • On making do: “It’s a poor workman who blames his tools.”
  • On persistence (II): “I don’t know how smart you are,” he said. “But I’ll give you stubborn.”
  • On expertise in your field and out of your field: “You been a fighter…and you stay in shape, you don’t lose that many fights outside the ring.”
  • On seeking truth: “You probably can’t figure out the truth, if you think you know ahead of time what the truth is supposed to be.”
  • On self-mastery:
“You need to work on your inhibitions,” I said.
“Controlling them?” Rita said.
“No,” I said. “Acquiring some.”

Quick take: “Will More Higher Education Improve Economic Growth?” by Hanushek

David EvansLeave a comment

I hear consistent talk of the need to invest in higher education in low- and middle-income countries, so I was interested to see Eric Hanushek’s paper asking if more higher education will improve economic growth, which is forthcoming in the Oxford Review of Economic Policy.

As Hanushek says, “one does not get electrical engineers and computer scientists without investing in higher education.” And yet, his regression results (growth regressed on cognitive skills, non-tertiary schooling levels, and tertiary schooling levels) suggest no significant association between higher tertiary and economic growth. “These  results suggest the possibility that a number of countries are following a misplaced investment strategy if their goal is to improve economic growth. They might be better off spending on the margin to improve basic skills in earlier schooling (where they can be subsequently built upon in university) than simply expanding colleges and universities with existing basic skills.”

One country does show a strong positive relationship between years of tertiary and economic growth: the USA. He argues that the high quality of US universities and their ability to attract high skilled migrants, many of whom stay to work in the US, may explain that result.

Tertiary may be important for many reasons, including the formation of institutions and future leaders. But this pass at the data don’t suggest a strong growth argument.

What do you think?

Quick take: “Education Quality and Teaching Practices” in Chile

David EvansLeave a comment

On this morning’s commute, I caught up on a new NBER Working Paper: “Education Quality and Teaching Practices,” by Marina Bassi, Costas Meghir, and Ana Reynoso.

Here is the abstract: “This paper uses a RCT to estimate the effectiveness of guided instruction methods as implemented in under-performing schools in Chile. The intervention improved performance substantially for the first cohort of students, but not the second. The effect is mainly accounted for by children from relatively higher income backgrounds. Based on the CLASS instrument we document that quality of teacher-student interactions is positively correlated with the performance of low income students; however, the intervention did not affect these interactions. Guided instruction can improve outcomes, but it is a challenge to sustain the impacts and to reach the most deprived children.”

Why no effect for lower-income students? To expand a bit on the abstract: “The most striking result from the table is the association between better student teacher interactions (reflected in a higher CLASS score) and the performance of low income students. In effect, one additional standard deviation in the principal component of CLASS scores is associated with a higher SIMCE test score for low income students of between 15% and 20% of sd units. These results are robust to adjustments in p-values to control for the FWE rate. For higher income students, effects are smaller and in some cases insignificant.”

So if the quality of interactions is particularly important for lower-income students, and the intervention isn’t affecting those interactions, then that could explain the differential effects. It’s an interesting hypothesis, and it points to the ongoing need to better understand what’s happening in the classroom.

Here’s a little more on the intervention: “The main intervention of the program was to support teachers through a modifed method of instruction by adopting a more prescriptive model. Teachers in treated schools received detailed classroom guides and scripted material to follow in their lectures.”

spend a few delightful hours in this bookshop, with good book people

David Evans2 Comments
a review of The Storied Life of A.J. Fikry, by Gabrielle Zevin

A.J. Fikry owns a book shop in a small town, and a recurring episode in the novel is the book club hosted by the local police chief. In one playful exchange, the chief and the other participants disagree about a book, and Fikry’s response draws cheers from the book club. I caught myself laughing aloud. Not because the scene was hilarious, but because the characters in this book became my dear friends so quickly, and I was part of this light, silly moment.

The story centers around Fikry — who starts grumpy and grows less grumpy — and the people who come into his life. I read it in just a few days, because every time I picked up my phone, I wanted to see what would happen next. (Thank you, Kindle app!)

Here is an extended taste, from an early exchange between Fikry and a publishing company representative:

“How about I tell you what I don’t like? I do not like postmodernism, postapocalyptic settings, postmortem narrators, or magic realism. I rarely respond to supposedly clever formal devices, multiple fonts, pictures where they shouldn’t be—basically, gimmicks of any kind. I find literary fiction about the Holocaust or any other major world tragedy to be distasteful—nonfiction only, please. I do not like genre mash-ups à la the literary detective novel or the literary fantasy. Literary should be literary, and genre should be genre, and crossbreeding rarely results in anything satisfying. I do not like children’s books, especially ones with orphans, and I prefer not to clutter my shelves with young adult. I do not like anything over four hundred pages or under one hundred fifty pages. I am repulsed by ghostwritten novels by reality television stars, celebrity picture books, sports memoirs, movie tie-in editions, novelty items, and—I imagine this goes without saying—vampires.”

I found this gem randomly, browsing online bookshelves. Late in the book, Fikry muses, “Why is any one book different from any other book? … We have to look inside many. We have to believe. We agree to be disappointed sometimes so that we can be exhilarated every now and again.” I’m glad I took a chance. I was exhilarated.

The book is thoroughly delightful.

Here are a few other lines I enjoyed:

  • On embracing the unexpected: “She doesn’t want to become the kind of person who thinks that good news can only come from calls one was already expecting and callers one already knows.”
  • On expecting structure in life: “He doesn’t believe in random acts. He is a reader, and what he believes in is narrative construction. If a gun appears in act one, that gun had better go off by act three.”
  • On priming: “She was pretty and smart, which makes her death a tragedy. She was poor and black, which means people say they saw it coming.”
  • On getting to know people: “You know everything you need to know about a person from the answer to the question, What is your favorite book?”
  • On empathy: “Empathy…is the hallmark of great writing.”
  • On book jackets: “Jackets are the redheaded stepchildren of book publishing. We blame them for everything.”
  • On blurbs: “Blurbs” are “the blood diamonds of publishing.”

a rich, elegant retelling of the rise and reign of King David

David Evans1 Comment

 

As Alana Newhouse writes in the New York Times, this novel, “a thundering, gritty, emotionally devastating reconsideration of the story of King David — makes a masterful case for the generative power of retelling.” Her review is short and excellent. You might want to skip mine and just go read hers.

 

I grew up on the stories of the Bible, including the stories of David: David and Goliath (wait, there were giants?), David and Bathsheba (don’t look, man!), Saul with his thousands versus David with his tens of thousands (not your fault, but a totally unfortunate bit of PR). But like most Bible stories and other traditional stories, the details are pretty sparse. Why did David kill so many innocents in his outlaw years, seemingly unnecessarily? Why didn’t David punish his son Amnon after he raped his half-sister Tamar? Why was Bethseba bathing on that roof? Why did most of David’s sons seem to turn out so badly?

 

Brooks fills in the scriptural narrative with motives, passions, and details, as told through the eyes of Nathan the Seer. Having a seer as your narrator, incidentally, while not technically granting you an “omniscient third-person narrator,” comes pretty close. In the course of the book, Nathan interviews those who knew David in early life, he recounts what he himself has seen, and he drops into visions of far-off, often terrible events. The prose is beautiful. For example, when describing the relationship between three brothers who have fought beside David for many years, she writes, “To say these three were close does not do the matter justice. They had shared more than a womb. They were knit together by the rind of scar tissue that comes after long, bloody service.”

 

As a consumer of legendary tales, I really value Brooks’s effort to demonstrate one way that the sparse tales could be filled in with real people rather than the paper cut-outs so often encountered in scripture and elsewhere. What was the relationship between David and Abigail like? and Michal? and what about that friendship with Jonathan? Of course, one could fill in the story in different ways. But by detailing one way, Brooks opens up our imagines to how we might think about the characters in other legendary narratives.

 

A recurring theme deals with the ends justifying the means, with David repeatedly killing or taking other action for what he views as a greater good. “It was necessary,” he tells Nathan over and over again. This refrain both sickens the seer and — ultimately, sometimes — makes sense to him.

 

I listened to the audiobook, which was well narrated by Paul Boehmer.

 

Note: This book has violence, sex, and sexual violence. Just like the Bible. Oh, and it has strong language. Not to be all judgy, but I hope that’s not what pushes you over the edge.

Never Stop Believing!

David Evans1 Comment

“Never stop believing.” –Journey No wait, Anthony Doerr, in All the Light We Cannot See

Marie Laure is a blind French girl who adores the novels of Jules Verne. Werner is a German orphan and engineering whizz who makes his way into the army. The book alternates between their experiences during the course of World War II. The story is engaging, and the prose is lovely. As Sara Nelson at Amazon wrote, “This is not a book you read for plot…This is a book you read for the beauty of Doerr’s writing.” It took home the 2015 Pulitzer Prize for fiction.

I enjoyed it but didn’t adore it. The plot had a few too many coincidences and I just never got completely caught up in it.

I listened to the audiobook, which was capably narrated by Zach Appelman, but I really wish the audiobook had used two narrators, a man for Werner and a woman for Marie Laure. Having only a male narrator gave the subtle (and probably unintended) impression that the story was more from Werner’s perspective, despite the fact that Marie Laure is a wonderful protagonist who more than holds her own.

Here are a few bits that I noted:

On looking up from my smartphone: “Open your eyes and see what you can with them before they close forever.”

On Journey: “You must never stop believing. That’s the most important thing.”

On being awesome: “Don’t you want to be alive before you die?”

On science: “Science, my lad, is made up of mistakes, but they are mistakes which it is useful to make, because they lead little by little to the truth.” (Jules Verne, quoted here)

On naive interpretations of others’ experience: “To shut your eyes is to guess nothing of blindness.”

On the world beyond sight: “Beneath your world of skies and faces and buildings exists a rawer and older world, a place where surface planes disintegrate and sounds ribbon in shoals through the air.”

On what a bibliophile is thinking as she hides in the basement after her house is bombed: “If only she had brought her novel down with her.”

And here are books that figure prominently within the book: