Category: Economics

Has the war over RCTs been won?

David Evans1 Comment

Esther Duflo and Abhijit Banerjee of MIT and JPAL weigh in during an interview in Tim Ogden’s forthcoming book, Experimental Conversations, which I am enjoying thoroughly:

Esther: “I think it’s been completely won in that I think it’s just happening. A lot of people are doing it without us. It’s being used. I think it is now understood to be one of the tools. The argument within the economics profession [over the value of RCTs] had two main consequences, both good. First, it raised the profile. If something was debated, people began to believe it must be significant. Second, it did force us to answer the challenges. There were a lot of valid points that were raised and it forced us to react. We’ve become more intelligent as a result.”

Abhijit: “I am less certain that it has been won. The acid test of whether an idea has come to stay is that it becomes something that no one needs to justify using. … RCTs aren’t there yet: it is true almost everyone is doing them, but many of them are taking the trouble to explain that what they do is better than a ‘mere RCT.’ We need to get to the point where people take RCTs to be the obvious tool to use when possible to answer a particular class of empirical questions.”

Beyond incentives: The economist as plumber

David Evans2 Comments

Earlier this month, Esther Duflo of MIT gave a talk at the IMF, and the slides are available here. I found the framing insightful.


She goes on to give three examples from impact evaluations in India that seek to improve “the rules of the game,” creating systems for better governance: (1) “fixing the pipes” — eplatform for workfare payments, (2) “changing the faucet” — biometric identification for welfare payments, and (3) “replacing the meter” — inspections on polluting compliance.

Michael Kremer on how RCTs lead to innovation

David EvansLeave a comment

The modern movement for RCTs in development economics…is about innovation, as well as evaluation. It’s a dynamic process of learning about a context through painstaking on-the-ground work, trying out different approaches, collecting good data with good causal identification, finding out that results do not fit pre-conceived theoretical ideas, working on a better theoretical understanding that fits the facts on the ground, and developing new ideas and approaches based on theory and then testing the new approaches.

This is from an insightful interview with Michael Kremer, Harvard economics professor “generally given credit for launching the RCT movement in development economics with two experiments he led in Kenya in the early 1990s,” and my graduate school advisor.

The interview is in Tim Ogden’s book Experimental Conversations: Perspectives on Randomized Trials in Development Economics.

 

How Steve Levitt convinced me to give my son extra screen time right now

David EvansLeave a comment

Alternative title – Quick take: “The Behavioralist Goes to School,” by Levitt et al.

This evening my middle-school son was negotiating for additional screen time, and I proposed that he receive that additional time based on incremental improvement of his grades. But we had just discussed Levitt et al.’s recent paper testing loss aversion with student incentives over dinner, so he opted to start with the extra screen time and then lose it if the grades failed to improve. (His grades are pretty good anyway; just sayin’.) And I couldn’t not go with the evidence, so this had better work.

Back to the paper: Do students respond to incentives if the reward comes right away? Do they respond to non-monetary incentives? Does offering those incentives once crowd out intrinsic motivation?

If those are your questions, then Levitt et al. have some answers in “The Behavioralist Goes to School: Leveraging Behavioral Economics to Improve Educational Performance.” Here’s the abstract:

We explore the power of behavioral economics to influence the level of effort exerted by students in a low stakes testing environment. We find a substantial impact on test scores from incentives when the rewards are delivered immediately. There is suggestive evidence that rewards framed as losses outperform those framed as gains. Nonfinancial incentives can be considerably more cost-effective than financial incentives for younger students, but are less effective with older students. All motivating power of incentives vanishes when rewards are handed out with a delay. Our results suggest that the current set of incentives may lead to underinvestment.

Here’s some detail on the motivation from the authors: “One of the biggest puzzles in education is why investment among many students is so low given the high returns. One explanation is that the current set of long-run returns does not sufficiently motivate some students to invest effort in school.”

And here’s a little more detail on the study and results.

They focus on three features of past incentive programs:

  1. There is a time gap between when students exert effort and when they receive the reward.
  2. Rewards are offered as gains (not losses).
  3. Rewards are monetary.

After some proof-of-concept testing, they ultimately run their field experiments with 5,000+ students in Chicago public schools.

On point 1 (the time gap), they use a test where possibility of rewards are announced to students immediately before the test (so it’s a test of immediate effort, not preparation) and the rewards are given immediately after, as compared to a treatment where the reward is delivered a month later. Receipt of the reward is determined by improvement relative to a baseline test several months before. Depending on the setting, the test is a low-stakes diagnostic reading or math assessment.

“We find that large incentives delivered immediately, whether financial [$20 cash] or nonfinancial [trophy worth $3], have a significant impact on test performance of about a tenth of a standard deviation. In stark contrast, rewards delivered with a one month delay have no impact, nor do small financial rewards [$10 cash].” “As far as we know, ours is the first study to demonstrate that student responsiveness to incentives is sensitive to the size of the reward.”

To test point 2 (gains versus losses), they vary whether students receive the reward before the test and then have to return it immediately after testing or they receive the reward after the test. “In the pooled estimates, the coefficients on losses are roughly twice the magnitude of the analogous ‘gain’ treatments, but are not statistically different from those treatments.”

On point 2 (monetary rewards), they test non-monetary rewards – a trophy worth $3 – against the large and small monetary rewards. “In the pooled results, the point estimates for non-pecuniary rewards (framed either as a gain or a loss) are somewhat smaller than those for the $20 treatment and much larger than those from the $10 treatment.”

Gender: “Our findings with respect to gender are consistent with a wealth of prior research that shows boys tend to be more sensitive to short-term incentives than girls, which may be due in part to gender differences in time preferences.”

Age: “In general, we see similar results across young and old students, with the exception of nonfinancial incentives framed as losses, where we find large positive effects on young students and small negative impacts on older students.”

Do these incentives affect subsequent test performance? The low financial incentives (which had no impact in the short run) lead to negative impacts on tests a few months later. The other incentives have no statistically significant impact and have a mix of positive and negative point estimates.

My short take away: Nuance, nuance, nuance. Student motivation is probably largely overlooked, and offering incentives can have a positive effect. But if you’re doing student incentives, test them out before committing at scale. Although List et al. don’t find pervasive evidence of problems after the incentives are removed, they do find a little, and a couple of other studies (here and here) have as well.

Bonus reading – a few other papers on financial incentives for students

Quick take: “I failed, no matter how hard I tried”: A mixed-methods study of the role of achievement in primary school dropout in rural Kenya, by Zuilkowski et al.

David EvansLeave a comment

In Kenya, virtually every child enrolls in primary school, but many don’t complete it. Stephanie Simmons Zuilkowski, Matthew Jukes, and Peggy Dubeck use mixed methods to explore why.

Three findings stood out to me:

  1. In interviews with both youth and with parents, the youth (age 14-15, mostly, but some older) were “universally” characterized as the main education decision makers. In many cases, parents encouraged them to stay in school but the youth opted to drop out.
  2. Lower performing youth were more likely to drop out of school. This isn’t surprising but it’s useful to see it quantified. It comes out in both the quantitative and the qualitative work here.
  3. Free primary school isn’t free (and I’m not even talking about pure opportunity cost; I’m talking about simple out-of-pocket costs).

Okay, to the study! They point out why cross-sectional studies may miss the point in understanding dropout rates:

A cross-sectional study may identify proximal factors affecting dropout risk—perhaps pregnancy or the need to work for pay (Ball 2012)—but not the earlier factors that put the child on the trajectory toward dropout. In interviews with parents and teachers, proximal reasons for dropout may become the post-hoc rationale for a child’s dropout obscuring the underlying trigger factors.

Finding 1: Who decides on dropouts?  Admittedly, it’s a small sample for this part: They spoke with 21 youth and 20 parents. In most cases, the interviews were conducted separately. Of the youth, half had dropped out. Here is the key finding: “In our interviews with the dropouts in this sample, the youth were described universally as the principal educational decision-makers, both by the parents and by the youth themselves.” Notably, both youth and parents talked about the importance of education. “The stories of all 11 children who dropped out began with some variation of: ‘I wasn’t doing well in school.’” Many of the quotes highlight relative performance and the inability to get extra help. To me, this points back to the importance of structuring education systems that help teachers to teach to the right level (see here and here for more on that). Many of these children simply weren’t getting instruction at their current level.

Finding 2: “A student with a literacy composite score one standard deviation above average would have fitted odds of dropout that are 40% lower than those of the average scorer. A student with a numeracy score one standard deviation above average would have fitted odds of dropout that are 17% lower than those of the average scorer.” N=2,500+

Finding 3: “Despite the official abolition of school fees, all 13 schools the sampled youth attended had charged fees for extra teachers, books, or materials. Nine of the 21 interviewees—five students and four dropouts—said they had been sent home to get money for fees or materials. Children who could not gather the required amounts were not generally allowed back in class.”

I recommend the paper.

Quick take: “Will More Higher Education Improve Economic Growth?” by Hanushek

David EvansLeave a comment

I hear consistent talk of the need to invest in higher education in low- and middle-income countries, so I was interested to see Eric Hanushek’s paper asking if more higher education will improve economic growth, which is forthcoming in the Oxford Review of Economic Policy.

As Hanushek says, “one does not get electrical engineers and computer scientists without investing in higher education.” And yet, his regression results (growth regressed on cognitive skills, non-tertiary schooling levels, and tertiary schooling levels) suggest no significant association between higher tertiary and economic growth. “These  results suggest the possibility that a number of countries are following a misplaced investment strategy if their goal is to improve economic growth. They might be better off spending on the margin to improve basic skills in earlier schooling (where they can be subsequently built upon in university) than simply expanding colleges and universities with existing basic skills.”

One country does show a strong positive relationship between years of tertiary and economic growth: the USA. He argues that the high quality of US universities and their ability to attract high skilled migrants, many of whom stay to work in the US, may explain that result.

Tertiary may be important for many reasons, including the formation of institutions and future leaders. But this pass at the data don’t suggest a strong growth argument.

What do you think?

do you think living on under $2 a day is only a problem of developing countries? think again.

David Evans1 Comment

a review of Kathryn J. Edin and H. Luke Shaefer’s $2.00 a Day: Living on Almost Nothing in America

“1.5 million households with roughly 3 million children were surviving on cash incomes of no more than $2 per person, per day in any given month” as of 2011. In $2.00 a Day: Living on Almost Nothing in America, Kathryn Edin and Luke Shaefer compellingly demonstrate how families in extreme poverty in the U.S. survive. They use survey data to characterize the magnitude of the problem, then they follow a wide range of families over time to see how they got where they are and how they are getting by. They provide a quick review of the policies that brought us here (with the extreme poor increasing 4 percent over 15 years), as well as ideas for policies to give families a hand up.

One truth comes through in each family’s story: Every parent here would far prefer a steady job over a welfare payment. But guess what? There aren’t enough jobs at the bottom of the ladder, and the jobs that exist are terrible. As Jared Bernstein writes in his review of the book, “I cannot overemphasize the importance of this fundamental flaw in poverty policy, i.e., the assumption that there is an ample supply of perfectly good jobs out there that poor people could tap if they just wanted to do so.” One woman applied for over 100 jobs before getting a terrible one with irregular hours that ultimately caused enough health problems (from working in unheated spaces in Chicago winters) that she lost almost all of her hours. She is not the exception; story after story confirms her experience. (Anecdote plus anecdote plus anecdote eventually add up to data.)

Even when you include various non-cash benefit programs, the number of extremely poor is far too high, as the Huffington Post figure below shows. And part of what Edin and Shaefer demonstrate in the lives of the families they profile is how difficult it is to survive in the United States without cash: How do you ride the bus for a job interview? How do you obtain appropriate clothing to apply for jobs?

two dollars a day

This book is eye-opening and important. As Bernstein writes, “Doing nothing should not be an option.” And Edin and Shaefer don’t recommend a return to cash welfare. Every person they interview wants a job, so they want the government to help create those, via workfare and public-private partnerships. Regardless of whether you agree with that solution, read this book. I hope that it becomes part of our national conversation. Every other review I’ve encountered agrees, from the New York Times and the Los Angeles Times to the Boston Globe, Kirkus, and Publishers Weekly. Jonathan Cohn has an insightful interview with one of the authors in the Huffington Post.

I listened to the unabridged audiobook. It was well narrated by Allyson Johnson.

Excerpts from other reviews

William Julius Wilson, New York Times: “This essential book is a call to action, and one hopes it will [arouse] both the nation’s consciousness and conscience about the plight of a growing number of invisible citizens. The rise of such absolute poverty since the passage of welfare reform belies all the categorical talk about opportunity and the American dream.”

Stephanie Mencimer, Mother Jones: “$2.00 a Day is an intimate chronicle of the ‘cashless economy'”.

Julia Klein, Los Angeles Times: “History and analysis aside, what will likely remain with readers of ‘$2.00 a Day’ are its indelible pseudonymous portraits of families struggling to survive.”

Nick Romeo, Boston Globe: “Edin and Shaefer’s book is an important exposé on what they describe as ‘a poverty so deep that most Americans don’t believe it even exists.'”

Kirkus Reviews: “The authors share deeply human stories of the regular people trapped in poverty, typically through no fault of their own. Some are victims of abuse, others are forced to quit their low-paying jobs due to health concerns, and some simply cannot catch a break despite playing by the rules. An eye-opening account of the lives ensnared in the new poverty cycle.”

Jared Bernstein, Atlantic Monthly: “Given the lives to which Edin and Shaefer have introduced us, doing nothing should not be an option.”

Publishers Weekly: “This slim, searing look at extreme poverty deftly mixes policy research and heartrending narratives from a swath of the 1.5 million American households eking out an existence on cash incomes of $2 per person per day.”

the making of behavioral economics and a behavioral economist

David EvansLeave a comment

misbehaving audioA REVIEW OF RICHARD THALER’S MISBEHAVING: THE MAKING OF BEHAVIORAL ECONOMICS

There have been many books written recently on behavioral economics: Ariely’s Predictably Irrational and The Upside of Irrationality, Mullainathan and Shafir’s Scarcity: The New Science of Having Less and How It Defines Our Lives, Akerlof and Shiller’s Phishing for Phools: The Economics of Manipulation and Deception, Thaler and Sunstein’s Nudge: Improving Decisions About Health, Wealth, and Happiness, etc.

Each of these books demonstrates how people behave irrationally, in contrast to traditional economic thinking, and then discusses either public policies or personal policies that we can use — as Thaler puts it in this book — “to help people achieve their own goals,” whether that’s to save more, procrastinate less, or get out of poverty. I’ve read or listened to several of them.

This book is different and wonderfully refreshing. Thaler has been at the center of much of the development of behavioral economics, and this is a memoir of the field as he has observed and experienced it. It’s full of sharp wit and insightful, entertaining anecdotes, reporting on Thaler’s academic adventures as well as his application of behavioral economics in business consulting, for everyone from ski resorts to car manufacturers.

Because it’s a memoir, areas of behavioral economics that Thaler has touched less get less attention, such as the development of behavioral economics in international development work. That’s fine; you can read the World Bank’s Mind, Society, and Behavior or Mullainathan & Shafir’s Scarcity if that’s exclusively what you’re after.

I thoroughly enjoyed the unabridged audiobook and recommend it.

Here are a few lines that I noted, first on content:
  • On discussions with psychologist Danny Kahneman: “One aspect of these mutual training sessions involved understanding how members of the other profession think, and what it takes to convince them of some finding.”
  • “For those who are at least living comfortably, negative transaction utility can prevent our consuming special experiences that will provide a lifetime of happy memories, and the amount by which the item was overpriced will long be forgotten.”
  • “Interdisciplinary meetings, especially those with high-level agendas (reduce poverty, solve climate change) tend to be disappointing, even when the attendees are luminaries, because academics don’t like to talk about research in the abstract — they want to see actual scientific results.”
  • “As my Chicago colleague Linda Ginzel always tells her students: ‘If you don’t write it down, it doesn’t exist.'” (In other words, collect data in tangible form.)

And then a few pithy observations:

  • “To this day, Orley [Ashenfelter] insists on calling what I do ‘wackonomics,’ a term he finds hysterically funny.”
  • “At some point people reach an age at which they can no longer be considered ‘promising.’ I think it is about the time they turn forty.”
  • “‘Dumb stuff people do’ is not a satisfactory title for an academic paper.”

Should you read Dani Rodrik’s Economics Rules? Yes.

David Evans2 Comments

 

Dani Rodrik reminds us that economic models are good, and what they’re good for

 

Tyler Cowen writes that “Economics Rules is the single best source for explaining the strengths and weaknesses of economics to an outside audience.” I just listened to the audiobook, narrated by James Conlan, and I thoroughly enjoyed it. Over the course of six short chapters (272 pages, or 5.5 hours on audio), Rodrik lays out what economic models do, how to use them correctly, where economists go wrong, and finally answers a series of critiques of economics. One central message of the book is that the source of economics’ richness is its multiplicity of models, and that economists veer wrong when they seek to apply a single universal model to all situations (i.e., they “mistake a model for the model”). A second message is that economics has a lot to say about efficiency, but we shouldn’t pretend that’s the only worth value: “Any economist who makes a broader argument about the fairness, justice, or moral worth of markets that is based on economics proper is simply engaged in malpractice.”

 

I found Rodrik’s discussion of economic models to be engaging, deep, and accessible. He draws on pioneers in the field, like Smith, Marshall, and Keynes. He uses a wonderful one-paragraph short story by Jorge Luis Borges (“Del rigor en la ciencia”) to illustrate that the simplifications in models are a crucial element of their value. He uses an essay by philosopher Isaiah Berlin to distinguish between hedgehogs, who see the world as making sense through a single idea (such as free markets), and foxes, who draw on a wide range of models, always seeking the right model to understand a specific situation. “Economics needs fewer hedgehogs and more foxes engaged in public debates.” He mounts a spirited defense of the field, together with a few critiques of his own. He ends with ten commandments for economists (“Economics is a collection of models; cherish their diversity” and “Do no confuse agreement among economists for certainty about how the world works”) and ten for noneconomists (“When an economist makes a recomendation, ask what makes him/her sure the underlying model applies to the case at hand” and my favorite, “If you think economists are especially rude to noneconomists, attend one of their seminars”). Rodrik also has a healthy discussion of the benefits (and the limitations) of the rise of randomized controlled trials in applied microeconomics.

 

Here are a few quotes:

 

  • “When models are selected judiciously, they are a source of illumination. When used dogmatically, they lead to hubris and errors in policy.”
  • “Economists use math not because they’re smart but because they’re not smart enough.”
  • “If you think economists are especially rude to noneconomists, attend one of their seminars.”
  • Quoting Keynes: “If economists could manage to get themselves thought of as humble, competent people, on a level with dentists, that would be splendid!”
  • “Models are never true, but there is truth in models.”
  • “Economic science is merely disciplined intuition — intuition rendered transparent by logic and hardened by plausible evidence.”

 

I recommend the book for economists, economics students, and interested non-economists.

 

What have others said about the book?

 

Reviews by other people

 

Mixed: 

 

Clive Crook, Bloomberg: “It’s full of good insights, and has some sage advice for producers and consumers of economic knowledge. … But I think is mischaracterizes one important aspect of the problem.”

 

Positive: 

 

Kirkus: “A hopeful contribution to the reconstitution of a profession whose reputation has been seriously damaged, both fairly and unfairly.”

 

Martin Sandbu, Financial Times: “Rodrik convinces with his patient insistence on just how rich and versatile a library of models economists possess. … Rodrik’s book cannot fully dispel the scepticism.”

 

David Leonhardt, New York Times: Dani Rodrik “sets out to explain the discipline to outsiders (and does a nice job). Yet in surveying the larger “rights and wrongs” of economics, to quote his subtitle, Rodrik has diagnosed the central mistake that contemporary libertarians have made: They have conflated ideas that often make sense with those that always make sense.”
 

 

Highly Positive:
 

 

Publishers Weekly: “Excellent little primer”

 

N. Emrah Aydinonat: “Economics Rules is an excellent book; a must read for economists, philosophers of economics, and policy makers, and of course for economics students. And if you consider yourself as a critical economist, a heterodox economist, or a social scientist who is critical of the way in which economists work, behave, etc., the book has a lot to offer to you too (probably a lot to disagree about, but also a new perspective on economic models).”

 

It’s also Tyler Cowen’s list of the best non-fiction books of 2015 on Marginal Revolution.

 

Should I read Akerloff & Shiller’s Phishing for Phools?

David EvansLeave a comment

Here are eight on-line reviews. Three broadly negative:

  • Arnold Kling’s review: “Akerlof and Shiller are Nobel Laureates, which they earned with previous research. That is what makes this book so disappointing. People may enjoy reading Phishing for Phools, but it is lacking in real intellectual nutrition. It is the literary equivalent of a Cinnabon.”
  • The Economist: “Readers are merely left with the impression that there are lots of nasty people about—and perhaps that they may themselves have been phished.”
  • Alex Tabarrok in The New Rambler: “a disappointing foray into behavioral economics from two recent Nobel Prize winners. … Akerlof and Shiller have both made enormous contributions to economics but one will find in this book little of the analytical rigor or attention to evidence that earned them their laurels.”

Three more mixed reviews:

  • Tyler Cowen at Marginal Revolution: “My main complaint about the book is that its chooses easy targets and doesn’t puncture enough sacred cows.  … I wonder to what extent what the authors call “The Resistance and its Heroes” is in fact another example of…phishing for phools.”
  • Cass Sunstein in the New York Review of Books: “Their extraordinary book tells us something true, and profoundly important, about the operations of the invisible hand. But the largest views can lose focus.
  • Siddharth Singh at com: “They leave one question unanswered: how can the preferences of a regulator, or even a set of regulators, be superior to those of the ones being regulated?

With one broadly positive review in the Financial Times and one super positive one in the Times of Higher Education:

  • Robin Harding in the Financial Times: “The style of Phishing for Phools will be familiar to fans of Shiller’s work: light on jargon and pacy enough not to outstay its welcome. The authors tell some engaging tales, although usually at the remove of a fellow academic’s research. There is not much grime or anguish, dialogue or doubt.
    The brilliant, catchy title will sell Phishing for Phools. Indeed, it is almost as if Bob and George want to tempt the monkey on the shoulder of the book-buying public. They give their readers a breezy ride through some modern behavioural economics — and if they leave them hungry for a little more nutrition, well, what clever marketer does not?”
  • Victoria Bateman in the Times of Higher Education: “George Akerlof and Robert Shiller’s brilliant new book”