Nothing new under the sun — Education edition

Some years ago, I was evaluating an education program in The Gambia (read the evaluation here), soon after the government had outlawed corporal punishment in schools. We included a question about it in the evaluation and learned that there was a gap between legislation and practice, which the government then sought to resolve. 

Of course, controversy over corporal punishment in schools isn’t new, but I was surprised to see it debated in Noli me tangere, the 1887 novel by Filipino writer José Rizal. A frustrated schoolteacher recounts that after reading several books, his views changed: 

Lashings, for example, which since time immemorial had been the province of schools and which before I had seen as the only effective way to make children learn (that is how they have accustomed us to believe), began to seem far removed from contributing to a child’s progress, completely useless. I became convinced that when one keeps the switch or the rod in view reasoning is impossible… I began to think that the best thing I could do for these children was to develop confidence, security, and self-esteem.

So he eliminates corporal punishment. 

Little by little I held back the switch. I took the whips home and replaced them with emulation and belief in oneself.

Like any good experimenter, he evaluated short and medium run impacts.

In the beginning it seemed as though my method was impractical: a lot of them stopped studying altogether. But I pressed on, and I noticed that little by little their spirits rose. More students attended class, and more often. And when one day one was praised in front of everyone, the following day he learned twice as much.

But the local priest and the parents didn’t buy it and demanded he return to the traditional system.

I had to renounce a system that after a great deal of effort had begun to bear fruit.

Poor guy, but I imagine there are a number of reformers today who can feel his pain. 

The quotes are from Harold Augenbraum’s translation of the book.

The Promise of Teacher Coaching and Peril of Going to Scale

This, from Matt Barnum’s review of a recent meta-analysis on teacher coaching, by Kraft, Hogan, and Blazar.

First, what is coaching?

Teacher coaching involves “instructional experts work[ing] with teachers to discuss classroom practice in a way that is” individualized, intensive, sustained, context-specific, and focused.

What’s the finding?

“We find large positive effects of coaching on teachers’ instructional practice,” the authors write… Similarly, the 21 papers that looked at student achievement found notable positive results, on average.

But before you get too excited

When the research examines large-scale programs (with more than 100 teachers involved), the benefits, relative to small coaching initiatives, are cut roughly in half. 

Read the whole article or the meta-analysis itself.

What do African schoolteachers think about mathematics?

George Bethell has put out a report via the World Bank, Mathematics Education in Sub-Saharan Africa: Status, Challenges, and Opportunities. It brings together an array of analysis on current mathematics performance and teaching around the continent.

I’d like to highlight two elements. First, a survey in 6 African countries shows teacher attitudes towards math. Now, it’s just 70 teachers per country (roughly 50 primary and 20 secondary), so this is suggestive only. Key findings: Almost all teachers agree that mathematical skills are useful for everyone (except in DRC). But between 24 percent and 62 percent of teachers don’t believe that “everyone has the potential to be good at mathematics,” depending on the country, and a majority of teachers in all countries except Cameroon believe that “you have to have the right sort of brain to be good at mathematics.” It’s easy to imagine that these kinds of beliefs then play into the way teachers teach. (You can read more about it in Appendix A of the report.) Of course, I also don’t know how different these are from other parts of the world. The point is not whether these 6 countries are worse or better than other countries, but that teachers in these country have these particular beliefs about math education which may well affect pedagogy.

what-do-african-teachers-think-about-math

The report also lays out what he sees as the big research questions that remain in this area.

  1. How can countries in SSA monitor trends in mathematical achievement?
  2. How do learners understand mathematical concepts as demonstrated by their teachers? How do they approach mathematical problems?
  3. How effective are the textbooks currently being used to teach basic mathematics in SSA?
  4. How can national assessments of student achievement in mathematics be improved so that they provide policy makers and teachers with the information needed to improve outcomes in mathematics?
  5. Where Open Educational Resources have been used as the basis of, or to supplement, formal teacher education development programmes, have they been effective?
  6. Which of the e-learning and m-learning technologies in the classroom have the greatest potential to raise levels of numeracy and mathematical competence? What are the challenges of introducing e- and m-learning technologies – especially in fragile states?

Calling all researchers! To work!

math-questions-to-be-answered

How Steve Levitt convinced me to give my son extra screen time right now

Alternative title – Quick take: “The Behavioralist Goes to School,” by Levitt et al.

This evening my middle-school son was negotiating for additional screen time, and I proposed that he receive that additional time based on incremental improvement of his grades. But we had just discussed Levitt et al.’s recent paper testing loss aversion with student incentives over dinner, so he opted to start with the extra screen time and then lose it if the grades failed to improve. (His grades are pretty good anyway; just sayin’.) And I couldn’t not go with the evidence, so this had better work.

Back to the paper: Do students respond to incentives if the reward comes right away? Do they respond to non-monetary incentives? Does offering those incentives once crowd out intrinsic motivation?

If those are your questions, then Levitt et al. have some answers in “The Behavioralist Goes to School: Leveraging Behavioral Economics to Improve Educational Performance.” Here’s the abstract:

We explore the power of behavioral economics to influence the level of effort exerted by students in a low stakes testing environment. We find a substantial impact on test scores from incentives when the rewards are delivered immediately. There is suggestive evidence that rewards framed as losses outperform those framed as gains. Nonfinancial incentives can be considerably more cost-effective than financial incentives for younger students, but are less effective with older students. All motivating power of incentives vanishes when rewards are handed out with a delay. Our results suggest that the current set of incentives may lead to underinvestment.

Here’s some detail on the motivation from the authors: “One of the biggest puzzles in education is why investment among many students is so low given the high returns. One explanation is that the current set of long-run returns does not sufficiently motivate some students to invest effort in school.”

And here’s a little more detail on the study and results.

They focus on three features of past incentive programs:

  1. There is a time gap between when students exert effort and when they receive the reward.
  2. Rewards are offered as gains (not losses).
  3. Rewards are monetary.

After some proof-of-concept testing, they ultimately run their field experiments with 5,000+ students in Chicago public schools.

On point 1 (the time gap), they use a test where possibility of rewards are announced to students immediately before the test (so it’s a test of immediate effort, not preparation) and the rewards are given immediately after, as compared to a treatment where the reward is delivered a month later. Receipt of the reward is determined by improvement relative to a baseline test several months before. Depending on the setting, the test is a low-stakes diagnostic reading or math assessment.

“We find that large incentives delivered immediately, whether financial [$20 cash] or nonfinancial [trophy worth $3], have a significant impact on test performance of about a tenth of a standard deviation. In stark contrast, rewards delivered with a one month delay have no impact, nor do small financial rewards [$10 cash].” “As far as we know, ours is the first study to demonstrate that student responsiveness to incentives is sensitive to the size of the reward.”

To test point 2 (gains versus losses), they vary whether students receive the reward before the test and then have to return it immediately after testing or they receive the reward after the test. “In the pooled estimates, the coefficients on losses are roughly twice the magnitude of the analogous ‘gain’ treatments, but are not statistically different from those treatments.”

On point 2 (monetary rewards), they test non-monetary rewards – a trophy worth $3 – against the large and small monetary rewards. “In the pooled results, the point estimates for non-pecuniary rewards (framed either as a gain or a loss) are somewhat smaller than those for the $20 treatment and much larger than those from the $10 treatment.”

Gender: “Our findings with respect to gender are consistent with a wealth of prior research that shows boys tend to be more sensitive to short-term incentives than girls, which may be due in part to gender differences in time preferences.”

Age: “In general, we see similar results across young and old students, with the exception of nonfinancial incentives framed as losses, where we find large positive effects on young students and small negative impacts on older students.”

Do these incentives affect subsequent test performance? The low financial incentives (which had no impact in the short run) lead to negative impacts on tests a few months later. The other incentives have no statistically significant impact and have a mix of positive and negative point estimates.

My short take away: Nuance, nuance, nuance. Student motivation is probably largely overlooked, and offering incentives can have a positive effect. But if you’re doing student incentives, test them out before committing at scale. Although List et al. don’t find pervasive evidence of problems after the incentives are removed, they do find a little, and a couple of other studies (here and here) have as well.

Bonus reading – a few other papers on financial incentives for students

Quick take: “I failed, no matter how hard I tried”: A mixed-methods study of the role of achievement in primary school dropout in rural Kenya, by Zuilkowski et al.

In Kenya, virtually every child enrolls in primary school, but many don’t complete it. Stephanie Simmons Zuilkowski, Matthew Jukes, and Peggy Dubeck use mixed methods to explore why.

Three findings stood out to me:

  1. In interviews with both youth and with parents, the youth (age 14-15, mostly, but some older) were “universally” characterized as the main education decision makers. In many cases, parents encouraged them to stay in school but the youth opted to drop out.
  2. Lower performing youth were more likely to drop out of school. This isn’t surprising but it’s useful to see it quantified. It comes out in both the quantitative and the qualitative work here.
  3. Free primary school isn’t free (and I’m not even talking about pure opportunity cost; I’m talking about simple out-of-pocket costs).

Okay, to the study! They point out why cross-sectional studies may miss the point in understanding dropout rates:

A cross-sectional study may identify proximal factors affecting dropout risk—perhaps pregnancy or the need to work for pay (Ball 2012)—but not the earlier factors that put the child on the trajectory toward dropout. In interviews with parents and teachers, proximal reasons for dropout may become the post-hoc rationale for a child’s dropout obscuring the underlying trigger factors.

Finding 1: Who decides on dropouts?  Admittedly, it’s a small sample for this part: They spoke with 21 youth and 20 parents. In most cases, the interviews were conducted separately. Of the youth, half had dropped out. Here is the key finding: “In our interviews with the dropouts in this sample, the youth were described universally as the principal educational decision-makers, both by the parents and by the youth themselves.” Notably, both youth and parents talked about the importance of education. “The stories of all 11 children who dropped out began with some variation of: ‘I wasn’t doing well in school.’” Many of the quotes highlight relative performance and the inability to get extra help. To me, this points back to the importance of structuring education systems that help teachers to teach to the right level (see here and here for more on that). Many of these children simply weren’t getting instruction at their current level.

Finding 2: “A student with a literacy composite score one standard deviation above average would have fitted odds of dropout that are 40% lower than those of the average scorer. A student with a numeracy score one standard deviation above average would have fitted odds of dropout that are 17% lower than those of the average scorer.” N=2,500+

Finding 3: “Despite the official abolition of school fees, all 13 schools the sampled youth attended had charged fees for extra teachers, books, or materials. Nine of the 21 interviewees—five students and four dropouts—said they had been sent home to get money for fees or materials. Children who could not gather the required amounts were not generally allowed back in class.”

I recommend the paper.

Quick take: “Will More Higher Education Improve Economic Growth?” by Hanushek

I hear consistent talk of the need to invest in higher education in low- and middle-income countries, so I was interested to see Eric Hanushek’s paper asking if more higher education will improve economic growth, which is forthcoming in the Oxford Review of Economic Policy.

As Hanushek says, “one does not get electrical engineers and computer scientists without investing in higher education.” And yet, his regression results (growth regressed on cognitive skills, non-tertiary schooling levels, and tertiary schooling levels) suggest no significant association between higher tertiary and economic growth. “These  results suggest the possibility that a number of countries are following a misplaced investment strategy if their goal is to improve economic growth. They might be better off spending on the margin to improve basic skills in earlier schooling (where they can be subsequently built upon in university) than simply expanding colleges and universities with existing basic skills.”

One country does show a strong positive relationship between years of tertiary and economic growth: the USA. He argues that the high quality of US universities and their ability to attract high skilled migrants, many of whom stay to work in the US, may explain that result.

Tertiary may be important for many reasons, including the formation of institutions and future leaders. But this pass at the data don’t suggest a strong growth argument.

What do you think?

Quick take: “Education Quality and Teaching Practices” in Chile

On this morning’s commute, I caught up on a new NBER Working Paper: “Education Quality and Teaching Practices,” by Marina Bassi, Costas Meghir, and Ana Reynoso.

Here is the abstract: “This paper uses a RCT to estimate the effectiveness of guided instruction methods as implemented in under-performing schools in Chile. The intervention improved performance substantially for the first cohort of students, but not the second. The effect is mainly accounted for by children from relatively higher income backgrounds. Based on the CLASS instrument we document that quality of teacher-student interactions is positively correlated with the performance of low income students; however, the intervention did not affect these interactions. Guided instruction can improve outcomes, but it is a challenge to sustain the impacts and to reach the most deprived children.”

Why no effect for lower-income students? To expand a bit on the abstract: “The most striking result from the table is the association between better student teacher interactions (reflected in a higher CLASS score) and the performance of low income students. In effect, one additional standard deviation in the principal component of CLASS scores is associated with a higher SIMCE test score for low income students of between 15% and 20% of sd units. These results are robust to adjustments in p-values to control for the FWE rate. For higher income students, effects are smaller and in some cases insignificant.”

So if the quality of interactions is particularly important for lower-income students, and the intervention isn’t affecting those interactions, then that could explain the differential effects. It’s an interesting hypothesis, and it points to the ongoing need to better understand what’s happening in the classroom.

Here’s a little more on the intervention: “The main intervention of the program was to support teachers through a modifed method of instruction by adopting a more prescriptive model. Teachers in treated schools received detailed classroom guides and scripted material to follow in their lectures.”