$575 million programme has no impact on pupil outcomes

Findings from an evaluation of a $575 million programme to improve teacher performance found that, while sites implemented new measures of teaching effectiveness and modified personnel policies accordingly, the programme had no impact on pupil outcomes.

The Intensive Partnerships for Effective Teaching initiative, designed and funded by the Bill and Melinda Gates Foundation, aimed to dramatically improve pupil outcomes by improving pupils’ access to effective teaching. Three US school districts and four charter management organisations participated in the programme, which ran between 2009 and 2016.

The final evaluation report, published by the RAND Corporation, found that by the end of 2014-15, outcomes for pupils in the settings that took part in the initiative were not better than outcomes for pupils in similar settings that did not take part. There was no evidence that low-income minority (LIM) pupils had greater access than non-LIM pupils to effective teaching. In addition, it found very few instances of improvement in the effectiveness of teaching overall, and no improvement in the effectiveness of newly hired teachers compared to experienced teachers. The evaluation also found no increase in the retention of effective teachers, although there was some decline in the retention of ineffective teachers in most settings that took part in the initiative.

The report states several possible reasons that the initiative failed to achieve its goals for improving pupil outcome:

  • incomplete implementation of the key policies and practices
  • the influence of external factors, such as state-level policy changes during the initiative
  • insufficient time for effects to appear
  • a flawed theory of action
  • a combination of all these factors.

 

Source:  Improving teaching effectiveness: Final report: The Intensive Partnerships for Effective Teaching through 2015–2016 (2018), RAND Corporation.

The effect of teacher coaching on teaching and learning

Matthew A Kraft and colleagues conducted a meta-analysis of the causal evidence on the effect of teacher coaching on teaching and learning. Their paper, published in the Review of Educational Research, reviewed 60 studies on teacher coaching programmes conducted after 2006 that measured the impact of teacher coaching on either teaching (measured using tools such as the Classroom Assessment Scoring System or the Early Language and Literacy Classroom Observation) or pupil academic performance (measured by standardised tests).

Their results found that sustained coaching improves both classroom teaching and pupil achievement, with pooled effect sizes of +0.49 standard deviations for teaching and +0.18 standard deviations for academic achievement.

However, the effectiveness of a teacher coaching programme seems to be determined by the number of participants. When studies were divided into programmes that had fewer than 100 participants and those that had more than 100 participants, the impact on teaching was nearly double for the smaller programmes than for programmes with more than 100 participants. For pupil achievement, the smaller programmes showed an impact of nearly three times that of the larger programmes.

Source:  The effect of teacher coaching on instruction and achievement: A meta-analysis of the causal evidence (February 2018), Review of Educational Research, Vol 88, Issue 4

Numeracy intervention for pupils struggling with maths

The Education Endowment Foundation has published an evaluation of 1stClass@Number, a 10-week numeracy intervention, delivered by teaching assistants, that provides intensive support for pupils struggling with maths.

A randomised controlled trial was conducted in 133 schools in south and west Yorkshire. Schools each nominated four children in Year 2 to participate, and the schools were then randomly assigned to either receive the intervention or to continue with normal teaching. A team from the University of Oxford evaluated the programme, which was delivered three times a week for 10 weeks in addition to normal mathematics instruction. A process evaluation collected additional data through observations, questionnaires and phone interviews.

Results showed that the intervention had a positive effect on Quantitative Reasoning Tests (effect size = +0.18) compared to pupils in the control group. Among pupils eligible for free school meals, those in the intervention group did not make any additional progress in the Quantitative Reasoning Test compared to control group pupils.

1stClass@Number seemed to have no impact on performance in end of Key Stage 1 maths tests compared to pupils in the control group. However, there was some evidence that the intervention widened the gap in Key Stage 1 maths results between pupils eligible for free school meals and their peers.

Source: 1stClass@Number: Evaluation report and executive summary (July 2018), Education Endowment Foundation

Classroom-based intervention for pupils with autism

Findings from a cluster randomised trial published in the Journal of Consulting and Clinical Psychology suggest that classroom teachers can effectively deliver a programme for young pupils with Autistic Spectrum Disorder (ASD) that results in better outcomes relative to usual school-based education.

Lindee Morgan and colleagues conducted a trial of the Social, Communication, Emotional Regulation and Transactional Support (SCERTS) intervention – a classroom-based, teacher-implemented intervention aimed at improving active engagement, adaptive communication, social skills, executive functioning and problem behaviour in elementary (primary) school pupils with ASD – to assess what improvement pupils in the intervention group made across a variety of measures compared to pupils in the control group. Sixty schools from three US states were randomly assigned to either the intervention or control groups. Teachers in the intervention group were trained in how to deliver the SCERTS programme and received coaching throughout the school year.

Results showed better outcomes for the intervention group than the control group on observed measures of classroom active engagement with respect to social interaction. The intervention group also had better outcomes on measures of adaptive communication, social skills and executive functioning (effect sizes ranged from +0.31 to +0.45).

Source: Cluster randomized trial of the classroom SCERTS intervention for elementary students with autism spectrum disorder (July 2018), Journal of Consulting and Clinical Psychology, Volume 86(7)

Good behaviour is no game

An evaluation conducted for the Education Endowment Foundation looked at whether the Good Behaviour Game (GBG) improved pupils’ reading skills and behaviour.

The GBG intervention is a classroom management approach designed to improve pupil behaviour and build confidence and resilience. The game is played in groups and rewards pupils for good behaviour. More than 3,000 Year 3 pupils from 77 UK schools took part in a randomised controlled trial of GBG over two years. Around a quarter of the pupils in the schools were eligible for free school meals, around a fifth were pupils with special educational needs, and 23% had English as an additional language.

The analysis indicated that, on average, GBG had no significant impact on pupils’ reading skills (effect size = +0.03) or their behaviour (concentration, disruptive behaviour and pro-social behaviour) when compared to the control group pupils. However, there was some tentative evidence that boys at risk of developing conduct problems showed improvements in behaviour.

Source: Good Behaviour Game: Evaluation report and executive summary (July 2018), Education Endowment Foundation

Evidence supports Foundations: Establishing Positive Discipline Policies

Safe and Civil Schools’ Foundations: Establishing Positive Discipline Policies is a programme designed to create a safer climate within a school and its surrounding grounds. It focuses on using proactive techniques instead of punishment to facilitate change, a technique referred to as school-wide positive behavioural intervention and support (SWPBIS). Foundations helps schools adopt and maintain these techniques by providing school teams with training and data-gathering materials, multiyear training, coaching support and school visits. The programme requires a team at each school to teach the staff these techniques, and data related to safety and behaviour is used to gauge what is working and what is not.

A randomised controlled trial in 32 US elementary (primary) schools in a large urban school district demonstrated that Foundations yielded improvements in school discipline, pupil safety policy and training, staff perceptions of pupils behaviour and suspension and tardiness rates over a two-year period. To determine if positive results of the programme would generalise into a “real world” setting, a second study continued to evaluate these elementary schools for two more years while also scaling up to add all remaining elementary and secondary schools in the district, when these schools adopted the programme without support from research funding.

In total, 74 regular public schools participated in the scale-up study. All schools received two years of Foundations training where school teams learned how to implement improvements related to safety, behaviour and discipline, and to collect and analyse related data to set goals for the next year.

As in the randomised study, schools in the second study showed gains in all areas. Specifically, after Foundations training, staff reported improved pupil behaviour, with fewer suspensions, absenteeism and tardiness, with a positive relationship evidenced between years of implementation and rates of effectiveness. The authors concluded that the improvements from the Foundations training were evidence that the programme generalises well into the real world setting.

Source: A randomized evaluation of the Safe and Civil Schools model for positive behavioral interventions and supports at elementary schools in a large urban school district, School Psychology Review, Volume 42, Issue 3

Scale-up of a Safe & Civil Schools’ model for school-wide positive behavioral interventions and supports (April 2016), Psychology in the Schools, Volume 53, Issue 4