What is the impact of pay-for-performance?

Teacher Incentive Fund (TIF) grants were awarded in 2010 by the US Department of Education to support performance-based compensation systems for teachers and principals in high-need schools.

In order to assess the impacts of pay-for-performance on educator (teachers and principals) and pupil outcomes, an experimental study design was used in ten US school districts to randomly assign elementary and middle schools to treatment and control groups. Both groups implemented the same performance-based compensation system, but in the control schools, the pay-for-performance element was replaced by a one percent bonus paid to all teachers and principals regardless of performance. A fourth and final report from this evaluation has now been published, covering all four years of the programme (between 2011 and 2015).

Among the key findings are that pay-for-performance had small, positive impacts on pupil achievement by the second year of implementation. From that year onward, reading and maths achievement was higher by 1 to 2 percentile points in schools that offered performance bonuses than in schools that did not. However, it was not entirely clear how this improvement was achieved. The impacts of pay-for-performance on classroom observation ratings did not appear to explain the impacts on pupil achievement, and in treatment schools as many as 40% of teachers were unaware that they could earn a performance bonus.

Source: Evaluation of the Teacher Incentive Fund: Final report on implementation and impacts of pay-for-performance across four years (NCEE 2017-4004),(December 2017), National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education.

A thirty-year look at studies on computer-assisted maths

During the past 30 years, thousands of articles have been written about technology’s effects on pupil achievement. In order to quantify technology’s effects on maths achievement, Jamaal Young at the University of Texas conducted a meta-analysis of all of the meta-analyses on the topic during the last three decades. His second-order meta-analysis was comprised of 19 meta-analyses representing 663 primary studies, more than 141,000 pupils and 1,263 effect sizes. Each meta-analysis that was included had to address the use of technology as a supplement to instruction, use pupil maths achievement as an outcome measure, report an effect size or enough data to calculate one, have been published after 1985 and be accessible to the public.

The author found that all technology enhancements positively affected pupil achievement, regardless of the technology’s purpose. However, technology that helped pupils perform computational functions had the greatest effects on pupil achievement, while combinations of enhancements demonstrated the least effects on pupil achievement. The author found that study quality and the type of technology used in the classroom were the main influencers on effect sizes. The highest-quality studies had the lowest effect sizes, which he attributes to their more rigorous analysis procedures. The high-quality reviews gave an overall effect size for the use of technology of +0.16 (compared with +0.38 for low- and +0.46 for medium-quality reviews).

Source: Technology-enhanced mathematics instruction: A second-order meta-analysis of 30 years of research (November 2017), Educational Research Review, Volume 22

The long-term impact of effective teaching

Peter Tymms and colleagues at Durham University’s Centre for Evaluation and Monitoring conducted a study of 40,000 children in England to examine what impact effective teaching in the first year of school has on achievement at the end of compulsory teaching at age 16.

Children’s early reading and maths development were measured at the start of school, at age four, using the Performance Indicators in Primary Schools (PIPS) assessments. They were assessed again at the end of their first school year and at ages 7, 11 and 16. By assessing children at the beginning and end of their first year, the researchers were able to identify effective classes – defined defined as a class where children made much larger than average gains from ages 4 to 5, controlling for pretests and deprivation.

The study, published in School Effectiveness and School Improvement, found  that children who were taught well in their first year of school went on to achieve better GCSE results in English and maths at age 16 (effect size = +0.2). Long-term benefits in achievement were also reported for those children who were in effective classes in Key Stages 1 and 2, however, these were not as large as those seen in the first year of school.

The study concludes that the first year of school presents an important opportunity to have a positive impact on children’s long-term academic outcomes.

Source: The long-term impact of effective teaching (December 2017), School Effectiveness and School Improvement DOI: 10.1080/09243453.2017.1404478

Preschool language skills a predictor of later reading comprehension

A systematic review published by the Campbell Collaboration summarises the research on the correlation between reading-related preschool predictors, such as code-related skills  and linguistic comprehension, and later reading comprehension skills.

Sixty-four longitudinal studies met the eligibility criteria for the review. These studies spanned 1986 to 2016 and were mostly carried out in the US, Europe and Australia. Overall, the findings of the review found that code-related skills (rhyme awareness, phoneme awareness, letter knowledge and rapid automatised naming) are most important for reading comprehension in beginning readers, but linguistic comprehension (grammar and vocabulary) gradually takes over as children become older. All predictors, except for non-word repetition, were moderately to strongly correlated with later reading comprehension. Non-word repetition had only a weak to moderate correlation to later reading comprehension ability.

These results suggest a need for a broad focus on language skills in preschool-age children in order to establish a strong foundation for reading comprehension.

Source: Preschool predictors of later reading comprehension ability: a systematic review (December 2017), A Campbell Systematic Review 2017:4, Campbell Collaboration

Impact of Research Learning Communities on pupil achievement in reading

An evaluation led by Jo Rose at the University of Bristol and published by the Education Endowment Foundation (EEF) looks at the impact of Research Learning Communities (RLC) on pupil achievement in reading at Key Stage 2 and teachers’ awareness, understanding, and use of research.

As part of a randomised controlled trial involving 199 schools, 60 primary schools were allocated to the treatment condition for the RLC intervention delivered by a team of academics from the Institute of Education at University College London. Two teachers from each of the schools involved in the trial were designated “Evidence Champions”. They attended four RLC workshops in which they discussed research with academic experts and colleagues from other schools. The Evidence Champions were then required to develop school improvement strategies using their learnings from the workshops, and to support other teachers in their schools to engage with research.

While the results of the evaluation showed some evidence that being in an RLC increased teachers’ engagement with research, there appeared to be no evidence that the RLC intervention led to improvements in reading outcomes for 10- and 11-year-olds, compared with the control group (effect size =+0.02). However, there was evidence that there may be some relationship between how engaged teachers are with research and the achievement of their students, regardless of any involvement in the RLC.

Source: Research Learning Communities (December 2017), Education Endowment Foundation

Disseminating research to make a difference in classrooms

The Education Endowment Foundation (EEF) has published the results of its Literacy Octopus trials – named after their multi-armed design – which looked at the impact of research dissemination on achievement in schools.

More than 13,000 primary schools across England were involved in the trials (823 schools in the first trial and 12,500 in the second trial), which drew on a wide range of evidence-based resources and events designed to support the teaching and learning of literacy at Key Stage 2. These included printed and online research summaries (including this Best Evidence in Brief e-newsletter), evidence-based practice guides, webinars, face-to-face professional development events and access to online tools.

The first trial tested whether sending schools high-quality evidence-based resources in a range of different formats would have an impact on pupil outcomes. The second trial tested whether combining the provision of resources with “light-touch” support on how to use them would have greater impact. Some schools were simply sent evidence-based resources, while others received the resources along with simple additional support, such as invitations to seminars on applying the resources in the classroom. As well as pupil outcomes, this trial also measured teachers’ use of research to measure the impact on teacher behaviour.

Neither of the Literacy Octopus trials found evidence of improved literacy achievement at Key Stage 2 for pupils whose teachers took part in the trials compared with the control group. The second trial found no increase in teachers’ use of, or engagement with, research. The results suggest that, in general, light-touch interventions and resources alone are unlikely to make a difference.

Source: The Literacy Octopus: Communicating and engaging with research (December 2017), The Education Endowment Foundation