Surprise rewards for good attendance had a surprising consequence

A working paper by Carly Robinson and colleagues, published by the Harvard Kennedy School, reports on an experiment to measure the impact of attendance rewards on pupils.

The trial included 15,629 sixth to twelfth grade pupils (Year 7-13) from 14 school districts in California. All the pupils had previously had perfect attendance in at least one month in the autumn. The pupils were randomly allocated to one of three groups:

  • “Prospective Award” pupils received a letter telling them they would receive a certificate if they achieved perfect attendance in February (the following month).
  • “Retrospective Award” pupils received a letter and certificate telling them they had earned an award for perfect attendance during one month in the autumn term.
  • Control pupils received no communication.

The researchers collected data on the pupils’ attendance in the following month (February). They found there was no impact of offering the prospective reward on subsequent attendance. They also found that offering the retrospective award resulted in pupils attending less school in February. Absences among this group increased by 8% (an average of 0.06 days per pupil). The researchers suggest that the retrospective awards may have sent unintended signals to the pupils, telling them that they were performing better than the descriptive social norm of their peers, and exceeding the institutional expectations for the awarded behaviour.

Source: The Demotivating Effect (and Unintended Message) of Retrospective Awards (July 2018) HKS Faculty Research Working Paper Series RWP18-020.

[Edit: The paper was subsequently published in Organizational Behavior and Human Decision Processes (Available online 29 May 2019)]

Video playback pays back dividends

Researchers at Harvard University are conducting a study called The Best Foot Forward Project to determine the accuracy and usefulness of teacher observation using video rather than in person. A new report describes the first-year results of a randomised controlled trial of the project.
As part of the study, 162 teachers were randomly assigned to an experimental group instructed to video their classroom performance for a year. They were then asked to send five clips of their choice for feedback to 51 randomly assigned administrators, who had received training in video observation. The administrators were each assigned three teachers to evaluate. The results were then compared to 50 administrators and 185 teachers assigned to a control group who underwent in-person observation, as they had done in the past. Teachers were matched on years of experience, race/ethnicity, gender, and their schools’ test scores.
Results of the first year of implementation included:
  • Videoed teachers were more likely than controls to report that the post-observation feedback from administrators was fair.
  • Videoed teachers were more likely than controls to change classroom practice as a result of post-observation feedback.
  • Videoed teachers rated their performance as lower than control teachers. They commented that they noticed behaviours when watching themselves on video that they wouldn’t have noticed otherwise.
A smaller study was completed to determine if using videos chosen by a teacher could mask their true performance. A group of external observers used the Classroom Assessment Scoring System to compare videos that teachers did not submit to videos that they did. Results showed that teachers’ strengths and weaknesses were consistent among submitted and not-submitted videos.
Researchers concluded that video observations offer several advantages over in-person observation: it reduces teacher anxiety and increases their perceptions of fairness; it promotes more congenial post-observation meetings between administrators and teachers than in-person observation; videoed teachers are more likely to make behavioural changes; and it allows administrators to perform evaluations on their schedule. The study is continuing and will examine the effects on pupil achievement of teacher behavioural changes following video observations and feedback.
Source: The Best Foot Forward Project – Substituting Teacher-Collected Video for In-person Classroom Observations: First Year Implementation Report (2015), Harvard University.

Parental involvement: Including fathers in the picture

A new meta-analysis from Harvard University explores the relative strength of the association between educational involvement of fathers versus mothers and the achievement of their children. The research suggested that parents have an equal academic impact on children regardless of their gender, although fathers’ mean levels of involvement were lower.

In general, research on parental involvement in education does not distinguish between fathers and mothers, and where the focus is on one parent this is most likely to be the mother. In contrast, this meta-analysis sought to put fathers in the picture. The authors included 52 empirical studies representing 390 correlations for the relation between parental involvement (mothers or fathers) and achievement. They found that parental involvement was positively associated with pupil achievement, and the relation between involvement and achievement was equally strong for fathers and mothers. Child gender did not moderate this relation.

The authors do note some limitations to their analysis, namely a lack of longitudinal studies and wide variability in the way parental involvement and achievement had been measured across the studies.

Source: Including Fathers in the Picture: A Meta-Analysis of Parental Involvement and Students’ Academic Achievement (2015), Journal of Educational Psychology.

US pupils lagging behind

A new report, sponsored by Harvard’s Program on Education Policy and Governance, explores the “educational shortcomings” of US pupils compared to their international counterparts. In particular, the authors wanted to know whether the picture is skewed by poor performance among children from disadvantaged backgrounds. They conclude that it is not just disadvantaged children who are lagging behind; it’s advantaged children as well.

The analysis used state-by-state data from the 2011 eighth grade National Assessment of Educational Progress (NAEP) tests, as well as international data from PISA 2012. PISA (the Programme for International Student Assessment) is a survey conducted every three years by the OECD. It aims to compare the performance of schools and education systems worldwide by assessing 15/16 year olds in three main subjects – mathematics, science, and reading – with a special focus on one subject per survey. PISA 2012 focused on mathematics.

The authors found that, when viewed from a global perspective, US schools seem to do as badly teaching those from advantaged families as they do teaching pupils from disadvantaged families. Overall, the US proficiency rate in maths (35%) places the country 27th among the 34 OECD countries that participated in PISA. The ranking was actually slightly lower for pupils from advantaged backgrounds (28th) than for those from disadvantaged backgrounds (20th). It is important to note that there are significant variations across states.

Although the focus of the report is on maths, the authors show similar results for proficiency in science and literacy. They conclude that the US has two achievement gaps to be bridged – the well-known gap between its advantaged and disadvantaged pupils, but also the distance between itself and its peers abroad. This research suggests that the latter is not a socioeconomic issue.

The report has also been published in Education Next.

Source: Not Just the Problems of Other People’s Children: U.S. Student Performance in Global Perspective (2014), Harvard’s Program on Education Policy.

Text messages change perceptions, but not behaviour

An experiment in Oklahoma examined the effect of giving pupils mobile phones, and then texting them messages to encourage them to participate at school. Almost 2,000 young people (aged 11-13) took part, all in schools with high percentages of children eligible for free or reduced-price lunches. The pupils were randomly allocated to four different groups – two groups received daily text messages, another received credits for reading books, and the last was a control group. The text messages exhorted the value of schooling, for example, “Each year high school students make $21,023. College graduates make $58,613. Do the math.”

Students changed their views about the value of education, and reported that they put in more effort at school. However, it made no difference to their actual attendance or behaviour at school, nor to their achievement. The authors suggest that this may be because the benefits are so far in the future that it is not worth the pupils expending effort now, or because they lack the self-control to commit to studying or going to class.

Source: Information and Student Achievement: Evidence from a Cellular Phone Experiment (2013), Harvard.

Good teachers get results

Everyone knows that a good teacher makes a difference, but establishing who the good teachers are, and what difference they make, has long been a problem. A new study by economists at Harvard University attempts to answer these questions. They analysed the school records and earnings information for 2.5 million children, and found that, when a high “value added” teacher joins a new school, results for their class improve.

Having a high value-added teacher (in the top 5%) for one year raises a child’s cumulative lifetime income by $50,000. How this information is used is clearly a matter of policy, but any system that aims to reward good performance while supporting or punishing poor performance would need to be carefully designed and tested. An interesting article about the study can be found here.

Source: The long-term impacts of teachers: teacher value-added and student outcomes in adulthood (2011), American Economic Review