How Steve Levitt convinced me to give my son extra screen time right now

Alternative title – Quick take: “The Behavioralist Goes to School,” by Levitt et al.

This evening my middle-school son was negotiating for additional screen time, and I proposed that he receive that additional time based on incremental improvement of his grades. But we had just discussed Levitt et al.’s recent paper testing loss aversion with student incentives over dinner, so he opted to start with the extra screen time and then lose it if the grades failed to improve. (His grades are pretty good anyway; just sayin’.) And I couldn’t not go with the evidence, so this had better work.

Back to the paper: Do students respond to incentives if the reward comes right away? Do they respond to non-monetary incentives? Does offering those incentives once crowd out intrinsic motivation?

If those are your questions, then Levitt et al. have some answers in “The Behavioralist Goes to School: Leveraging Behavioral Economics to Improve Educational Performance.” Here’s the abstract:

We explore the power of behavioral economics to influence the level of effort exerted by students in a low stakes testing environment. We find a substantial impact on test scores from incentives when the rewards are delivered immediately. There is suggestive evidence that rewards framed as losses outperform those framed as gains. Nonfinancial incentives can be considerably more cost-effective than financial incentives for younger students, but are less effective with older students. All motivating power of incentives vanishes when rewards are handed out with a delay. Our results suggest that the current set of incentives may lead to underinvestment.

Here’s some detail on the motivation from the authors: “One of the biggest puzzles in education is why investment among many students is so low given the high returns. One explanation is that the current set of long-run returns does not sufficiently motivate some students to invest effort in school.”

And here’s a little more detail on the study and results.

They focus on three features of past incentive programs:

  1. There is a time gap between when students exert effort and when they receive the reward.
  2. Rewards are offered as gains (not losses).
  3. Rewards are monetary.

After some proof-of-concept testing, they ultimately run their field experiments with 5,000+ students in Chicago public schools.

On point 1 (the time gap), they use a test where possibility of rewards are announced to students immediately before the test (so it’s a test of immediate effort, not preparation) and the rewards are given immediately after, as compared to a treatment where the reward is delivered a month later. Receipt of the reward is determined by improvement relative to a baseline test several months before. Depending on the setting, the test is a low-stakes diagnostic reading or math assessment.

“We find that large incentives delivered immediately, whether financial [$20 cash] or nonfinancial [trophy worth $3], have a significant impact on test performance of about a tenth of a standard deviation. In stark contrast, rewards delivered with a one month delay have no impact, nor do small financial rewards [$10 cash].” “As far as we know, ours is the first study to demonstrate that student responsiveness to incentives is sensitive to the size of the reward.”

To test point 2 (gains versus losses), they vary whether students receive the reward before the test and then have to return it immediately after testing or they receive the reward after the test. “In the pooled estimates, the coefficients on losses are roughly twice the magnitude of the analogous ‘gain’ treatments, but are not statistically different from those treatments.”

On point 2 (monetary rewards), they test non-monetary rewards – a trophy worth $3 – against the large and small monetary rewards. “In the pooled results, the point estimates for non-pecuniary rewards (framed either as a gain or a loss) are somewhat smaller than those for the $20 treatment and much larger than those from the $10 treatment.”

Gender: “Our findings with respect to gender are consistent with a wealth of prior research that shows boys tend to be more sensitive to short-term incentives than girls, which may be due in part to gender differences in time preferences.”

Age: “In general, we see similar results across young and old students, with the exception of nonfinancial incentives framed as losses, where we find large positive effects on young students and small negative impacts on older students.”

Do these incentives affect subsequent test performance? The low financial incentives (which had no impact in the short run) lead to negative impacts on tests a few months later. The other incentives have no statistically significant impact and have a mix of positive and negative point estimates.

My short take away: Nuance, nuance, nuance. Student motivation is probably largely overlooked, and offering incentives can have a positive effect. But if you’re doing student incentives, test them out before committing at scale. Although List et al. don’t find pervasive evidence of problems after the incentives are removed, they do find a little, and a couple of other studies (here and here) have as well.

Bonus reading – a few other papers on financial incentives for students

Quick take: “I failed, no matter how hard I tried”: A mixed-methods study of the role of achievement in primary school dropout in rural Kenya, by Zuilkowski et al.

In Kenya, virtually every child enrolls in primary school, but many don’t complete it. Stephanie Simmons Zuilkowski, Matthew Jukes, and Peggy Dubeck use mixed methods to explore why.

Three findings stood out to me:

  1. In interviews with both youth and with parents, the youth (age 14-15, mostly, but some older) were “universally” characterized as the main education decision makers. In many cases, parents encouraged them to stay in school but the youth opted to drop out.
  2. Lower performing youth were more likely to drop out of school. This isn’t surprising but it’s useful to see it quantified. It comes out in both the quantitative and the qualitative work here.
  3. Free primary school isn’t free (and I’m not even talking about pure opportunity cost; I’m talking about simple out-of-pocket costs).

Okay, to the study! They point out why cross-sectional studies may miss the point in understanding dropout rates:

A cross-sectional study may identify proximal factors affecting dropout risk—perhaps pregnancy or the need to work for pay (Ball 2012)—but not the earlier factors that put the child on the trajectory toward dropout. In interviews with parents and teachers, proximal reasons for dropout may become the post-hoc rationale for a child’s dropout obscuring the underlying trigger factors.

Finding 1: Who decides on dropouts?  Admittedly, it’s a small sample for this part: They spoke with 21 youth and 20 parents. In most cases, the interviews were conducted separately. Of the youth, half had dropped out. Here is the key finding: “In our interviews with the dropouts in this sample, the youth were described universally as the principal educational decision-makers, both by the parents and by the youth themselves.” Notably, both youth and parents talked about the importance of education. “The stories of all 11 children who dropped out began with some variation of: ‘I wasn’t doing well in school.’” Many of the quotes highlight relative performance and the inability to get extra help. To me, this points back to the importance of structuring education systems that help teachers to teach to the right level (see here and here for more on that). Many of these children simply weren’t getting instruction at their current level.

Finding 2: “A student with a literacy composite score one standard deviation above average would have fitted odds of dropout that are 40% lower than those of the average scorer. A student with a numeracy score one standard deviation above average would have fitted odds of dropout that are 17% lower than those of the average scorer.” N=2,500+

Finding 3: “Despite the official abolition of school fees, all 13 schools the sampled youth attended had charged fees for extra teachers, books, or materials. Nine of the 21 interviewees—five students and four dropouts—said they had been sent home to get money for fees or materials. Children who could not gather the required amounts were not generally allowed back in class.”

I recommend the paper.

Quick take: “Will More Higher Education Improve Economic Growth?” by Hanushek

I hear consistent talk of the need to invest in higher education in low- and middle-income countries, so I was interested to see Eric Hanushek’s paper asking if more higher education will improve economic growth, which is forthcoming in the Oxford Review of Economic Policy.

As Hanushek says, “one does not get electrical engineers and computer scientists without investing in higher education.” And yet, his regression results (growth regressed on cognitive skills, non-tertiary schooling levels, and tertiary schooling levels) suggest no significant association between higher tertiary and economic growth. “These  results suggest the possibility that a number of countries are following a misplaced investment strategy if their goal is to improve economic growth. They might be better off spending on the margin to improve basic skills in earlier schooling (where they can be subsequently built upon in university) than simply expanding colleges and universities with existing basic skills.”

One country does show a strong positive relationship between years of tertiary and economic growth: the USA. He argues that the high quality of US universities and their ability to attract high skilled migrants, many of whom stay to work in the US, may explain that result.

Tertiary may be important for many reasons, including the formation of institutions and future leaders. But this pass at the data don’t suggest a strong growth argument.

What do you think?

Quick take: “Education Quality and Teaching Practices” in Chile

On this morning’s commute, I caught up on a new NBER Working Paper: “Education Quality and Teaching Practices,” by Marina Bassi, Costas Meghir, and Ana Reynoso.

Here is the abstract: “This paper uses a RCT to estimate the effectiveness of guided instruction methods as implemented in under-performing schools in Chile. The intervention improved performance substantially for the first cohort of students, but not the second. The effect is mainly accounted for by children from relatively higher income backgrounds. Based on the CLASS instrument we document that quality of teacher-student interactions is positively correlated with the performance of low income students; however, the intervention did not affect these interactions. Guided instruction can improve outcomes, but it is a challenge to sustain the impacts and to reach the most deprived children.”

Why no effect for lower-income students? To expand a bit on the abstract: “The most striking result from the table is the association between better student teacher interactions (reflected in a higher CLASS score) and the performance of low income students. In effect, one additional standard deviation in the principal component of CLASS scores is associated with a higher SIMCE test score for low income students of between 15% and 20% of sd units. These results are robust to adjustments in p-values to control for the FWE rate. For higher income students, effects are smaller and in some cases insignificant.”

So if the quality of interactions is particularly important for lower-income students, and the intervention isn’t affecting those interactions, then that could explain the differential effects. It’s an interesting hypothesis, and it points to the ongoing need to better understand what’s happening in the classroom.

Here’s a little more on the intervention: “The main intervention of the program was to support teachers through a modifed method of instruction by adopting a more prescriptive model. Teachers in treated schools received detailed classroom guides and scripted material to follow in their lectures.”