The effects of financial incentives on standardised testing

Can financial incentives improve outcomes on standardised tests? In an unpublished paper, John List, Jeffrey Livingston and Susanne Neckermann examined whether motivating pupils to gain the knowledge needed to succeed on the International Student Admissions Test (ISAT) would result in higher test scores.

Subjects were 3rd–8th grade (Year 4–Year 9) pupils (n=226E, 226C) who were judged to be at risk of not passing the ISAT and were receiving tutoring to improve their scores in nine elementary and middle schools in a suburb outside of Chicago. Using a system designed by the ISAT’s developers, authors created short benchmark tests designed to measure the knowledge needed to be successful on the ISAT. Pupils, their parents and their tutors were informed that they would receive up to $90 if pupils’ performance improved on these benchmark tests and if they met other academic and behavioural goals.

Results showed that pupils demonstrated significant gains on the incentivised benchmark testing compared to control pupils (effect size =+0.29), indicating they had the knowledge to pass the ISAT. However, they did not demonstrate significant gains compared to controls for the ISAT itself (effect size=+0.05), for which they did not receive any financial incentive. This was true regardless of whether incentives were provided immediately or were delayed. The authors conclude that pupils may not be motivated to show what they know on standardised testing that holds no personal stake for them.

Similar results were found in a review of international experiments evaluating financial incentives in education by Robert Slavin, posted on The Best Evidence Encyclopedia. The emerging consensus is that financial incentives can have an impact on easily counted outcomes for which the incentives were directly given (such as attendance), but not on more general outcomes that should flow from the incentivised outcomes (such as achievement).

Source:  Do students show what they know on standardized tests? working papers (2016),  from the selected works of Jeffrey A Livingston, Bentley University

Behaviour incentives improve exam results for low-achieving students

Low-achieving students respond to incentives to increase their effort and engagement at school and do better than predicted on GCSE exams as a consequence. That is the main finding of a discussion paper published by the University of Bristol.

The project, led by Simon Burgess, Director of the Centre for Market and Public Organisation (CMPO), included more than 10,000 Year 11 students in 63 schools. The schools were recruited in the poorest parts of neighbourhoods in England and were randomised to one of the following treatment groups: financial incentives, non-financial incentives, or control. Students in the incentive treatment groups earned rewards every half-term based on inputs such as attendance, conduct, homework, and classwork, rather than for outputs such as assessment results. The financial incentive rewarded students with cash up to the value of £80 per half-term, while the non-financial incentive offered students the chance to qualify for a high-value event determined jointly by the school and students, such as a sporting event or trip to a theme park.

The researchers hoped to find that the incentives would improve effort and engagement and ultimately lead to improved GCSE performance even though the results themselves carried no rewards. The analysis showed that overall the impact of either financial or non-financial incentives on achievement was low, with small, positive but statistically insignificant effects on exam performance. However, among students with low predicted GCSE grades, those in the intervention groups got better marks than students in the control group, with treatment effects stronger for the financial incentives than the non-financial incentives (particularly in science). For students who were expected to do well, and already making an effort at school, the incentives made little difference.

Source: Understanding the response to financial and non-financial incentives in education: Field experimental evidence using high-stakes assessments (2016), Discussion Paper 16 / 678, University of Bristol

It’s hard work to motivate students

The results of an experiment to use incentives to increase student effort have shown little evidence of a significant positive impact.

The trial was carried out in 63 relatively deprived schools in England in the first two terms of the 2012/13 academic year. The 7,730 Year 11 students who took part were allocated to one of three groups:

  • In the control group schools (n=33), students received no incentives for student effort, but student effort was monitored in the same way as in the two treatment groups.
  • Students at schools in one treatment group (n=15) received financial rewards twice a term (every eight weeks) depending on their effort.
  • Students at schools in the other treatment group (n=15) were able to attend an event at the end of each term (at Christmas and Easter) if their effort met a certain threshold.

The trial aimed to test loss aversion (the idea that individuals dislike losses more than they like gains of the same value), so, for example, the students were told they had £80 in incentives, but money was deducted if they did not reach the threshold in four measures of effort: attendance, behaviour, classwork, and homework.

The results showed no significant improvement in attainment, for either type of incentive, in maths and English standardised tests. For students with a lower level of prior attainment, there was a small but significant improvement in maths scores (effect size +0.13). For the financial incentive there was a positive and statistically significant increase in classwork for English, maths, and science, and a similar (but not significant) improvement with the event incentive. There was no improvement in any of the other measures of effort.

The report is one of seven studies recently published by the Education Endowment Foundation.

Source: Increasing Pupil Motivation (2014), Education Endowment Foundation (EEF).

Text messages change perceptions, but not behaviour

An experiment in Oklahoma examined the effect of giving pupils mobile phones, and then texting them messages to encourage them to participate at school. Almost 2,000 young people (aged 11-13) took part, all in schools with high percentages of children eligible for free or reduced-price lunches. The pupils were randomly allocated to four different groups – two groups received daily text messages, another received credits for reading books, and the last was a control group. The text messages exhorted the value of schooling, for example, “Each year high school students make $21,023. College graduates make $58,613. Do the math.”

Students changed their views about the value of education, and reported that they put in more effort at school. However, it made no difference to their actual attendance or behaviour at school, nor to their achievement. The authors suggest that this may be because the benefits are so far in the future that it is not worth the pupils expending effort now, or because they lack the self-control to commit to studying or going to class.

Source: Information and Student Achievement: Evidence from a Cellular Phone Experiment (2013), Harvard.

Improving education in the developing world

New research, published in Science, explores what improves education in the developing world. The authors reviewed evidence from randomised evaluations. The research showed that making school more financially attractive can increase participation, for example, by providing financial support for poor mothers and through scholarships. Improving the health of children and providing information on how earnings would rise with education can increase schooling even more cost-effectively. Programmes designed to improve teacher performance and school accountability were more context-dependent. Pedagogical reforms that match teaching to pupils’ learning levels were found to be highly cost effective, as were reforms to improve accountability and incentives. “More of the same” solutions, such as more textbooks, had little effect on achievement.

Source: The Challenge of Education and Learning in the Developing World (2013), Science, 340(6130).

Supporting students around the world

This systematic review from the Campbell Collaboration explores the question: “What are the effects of interventions implemented in developing countries on measures of students’ enrolment, attendance, graduation, and progression?” To be included in the review, studies had to meet certain research criteria; for example, studies had to use a randomised controlled trial (with or without baseline control), or a quasi-experimental approach in which baseline controls on main outcomes were included. The final sample included 73 experiments and quasi-experiments. The most common interventions were conditional cash transfers, funding or grants to communities, school breakfasts or lunches, or remedial education or tutoring.

Results showed that the average effects across four main outcome areas (school enrolment, attendance, dropout, or progression) were all positive and statistically significant, although effects on enrolment, attendance, and progression were larger than those on dropout. Results also indicated positive and statistically significant effects for maths achievement and language achievement, but there was no evidence of effects on standardised test scores or other achievement outcomes.

Overall, the authors note that the effects were relatively small in magnitude. They say that “despite the statistical significance of the findings for the main outcomes, most of the effects are equivalent to about 3-9% improvements in the intervention versus control groups.”

Source: Interventions in developing nations for improving primary and secondary school enrollment of children: A systematic review (2012), Campbell Systematic Reviews