$575 million programme has no impact on pupil outcomes

Findings from an evaluation of a $575 million programme to improve teacher performance found that, while sites implemented new measures of teaching effectiveness and modified personnel policies accordingly, the programme had no impact on pupil outcomes.

The Intensive Partnerships for Effective Teaching initiative, designed and funded by the Bill and Melinda Gates Foundation, aimed to dramatically improve pupil outcomes by improving pupils’ access to effective teaching. Three US school districts and four charter management organisations participated in the programme, which ran between 2009 and 2016.

The final evaluation report, published by the RAND Corporation, found that by the end of 2014-15, outcomes for pupils in the settings that took part in the initiative were not better than outcomes for pupils in similar settings that did not take part. There was no evidence that low-income minority (LIM) pupils had greater access than non-LIM pupils to effective teaching. In addition, it found very few instances of improvement in the effectiveness of teaching overall, and no improvement in the effectiveness of newly hired teachers compared to experienced teachers. The evaluation also found no increase in the retention of effective teachers, although there was some decline in the retention of ineffective teachers in most settings that took part in the initiative.

The report states several possible reasons that the initiative failed to achieve its goals for improving pupil outcome:

  • incomplete implementation of the key policies and practices
  • the influence of external factors, such as state-level policy changes during the initiative
  • insufficient time for effects to appear
  • a flawed theory of action
  • a combination of all these factors.

 

Source:  Improving teaching effectiveness: Final report: The Intensive Partnerships for Effective Teaching through 2015–2016 (2018), RAND Corporation.

Strategies to promote teacher effectiveness

The Institute of Education Sciences has released a new evaluation brief that synthesises findings from two impact studies conducted by the National Center for Education Evaluation (NCEE). One study focused on a strategy of providing teachers  with feedback on their performance for two years (performance feedback), and the other study focused on a strategy of providing teachers with bonuses for four years based on their performance (pay-for-performance). Both strategies were supported by the Teacher Incentive Fund, which provided competitive grants to help US states and districts implement a multi-strategy approach to enhancing teacher effectiveness.

In each study, elementary and middle schools (primary schools) were randomly assigned to implement the strategy (the treatment group) or not (the control group). The performance feedback study included approximately 29,000 pupils and 1,000 teachers in grades 4–8, while the pay-for-performance study included approximately 38,000 pupils and 3,500 teachers in grades 3–8. Pupil outcomes were measured using end-of-year reading and maths scores.

Key findings were as follows:

  • Providing teachers with feedback on their performance for two years improved pupils’ maths achievement after the first year with a difference in scores that corresponds to an effect size of +0.05. The cumulative effect after two years of implementation was similar in magnitude but not statistically significant. The effect on reading in both years was positive but not statistically significant.
  • Providing teachers with bonuses based on their performance for four years improved pupils’ reading achievement after one, two and three years of implementation and pupils’ maths achievement after three years. After each of those periods of implementation, the effect size was +0.04 for reading and +0.06 for maths. However, as noted in the evaluation report, the impacts of pay-for-performance on classroom observation ratings did not appear to explain the impacts on pupil achievement, and in treatment schools, as many as 40% of teachers were unaware that they could earn a performance bonus.

The brief was prepared for NCEE by Andrew Wayne and Michael Garet of American Institutes for Research and Alison Wellington and Hanley Chiang of Mathematica Policy Research.

Source: Promoting educator effectiveness: the effects of two key strategies (March 2018), National Center for Education Evaluation, The Institute of Education Sciences

What is the impact of pay-for-performance?

Teacher Incentive Fund (TIF) grants were awarded in 2010 by the US Department of Education to support performance-based compensation systems for teachers and principals in high-need schools.

In order to assess the impacts of pay-for-performance on educator (teachers and principals) and pupil outcomes, an experimental study design was used in ten US school districts to randomly assign elementary and middle schools to treatment and control groups. Both groups implemented the same performance-based compensation system, but in the control schools, the pay-for-performance element was replaced by a one percent bonus paid to all teachers and principals regardless of performance. A fourth and final report from this evaluation has now been published, covering all four years of the programme (between 2011 and 2015).

Among the key findings are that pay-for-performance had small, positive impacts on pupil achievement by the second year of implementation. From that year onward, reading and maths achievement was higher by 1 to 2 percentile points in schools that offered performance bonuses than in schools that did not. However, it was not entirely clear how this improvement was achieved. The impacts of pay-for-performance on classroom observation ratings did not appear to explain the impacts on pupil achievement, and in treatment schools as many as 40% of teachers were unaware that they could earn a performance bonus.

Source: Evaluation of the Teacher Incentive Fund: Final report on implementation and impacts of pay-for-performance across four years (NCEE 2017-4004),(December 2017), National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education.

The impact of pay-for-performance

A new report from Alison Wellington and colleagues, published by the Institute of Education Sciences, looks at the implementation and impacts in US schools that offered pay-for-performance as part of their 2010 Teacher Incentive Fund (TIF) grants. These grants, now named the Teacher and School Leader Incentive Program, support performance-based compensation systems for teachers and principals in high-need schools.

An experimental study design was used to assess the impacts of pay-for-performance on educator and student outcomes. Elementary and middle schools within the evaluation districts were randomly assigned to treatment and control groups. The treatment schools were to fully implement their performance-based compensation system. The control schools were to implement the same performance-based compensation system with one exception—the pay-for-performance bonus component was replaced with a one percent bonus paid to all educators regardless of performance.

For the 10 evaluation districts that completed three years of TIF implementation (between 2011 and 2014), key findings showed that pay-for-performance had small, significant positive impacts on students’ math and reading achievement. The report notes that after three years of TIF implementation, the average math score was 2 percentile points higher in schools that offered pay-for-performance bonuses than in schools that did not. The average reading score was 1 percentile point higher in schools that offered pay-for-performance bonuses than in schools that did not. This difference was equivalent to a gain of about four additional weeks of learning.

Source: Evaluation of the Teacher Incentive Fund: Implementation and Impacts of Pay-for-Performance After Three Years (2016), Institute of Education Sciences

Teaching in Europe

Eurydice has published a new report about the teaching profession in lower secondary schools (approximately ages 12-15) in Europe. It uses the findings of the the 2013 OECD Teaching and Learning International Survey (TALIS) and Eurydice and Eurostat data on the 28 EU Member States, and seven other European countries. In all, 40 different education systems (including England, Wales, Scotland, and Northern Ireland), and around two million teachers were included.
 
The extensive report examines five areas considered important for policy: (1) demographics and working conditions; (2) initial teacher education and the transition to the teaching profession; (3) continuing professional development; (4) transnational mobility; and (5) attractiveness of the profession.
 
Findings include:
  • A degree is the minimum entry level for teacher training programmes in 15 countries, and 17 countries require a Master’s;
  • The minimum length of initial teacher training is usually between four and six years;
  • Within the EU, 91.2% of teachers have completed an initial teacher training programme;
  • Teaching time is contractually specified in 35 education systems. The majority of countries also centrally regulate the total working time of teachers, which averages 39 hours a week;
  • On average, teaching time constitutes 44% of a teacher’s total working time. England, Estonia, Sweden, Wales, and Northern Ireland are the only education systems that do not contractually specify the number of teaching hours;
  • In 29 education systems CPD is a “professional duty”, although around a third of these (including England, Wales, and Northern Ireland) do not specify how much time should be spent);
  • Incentives to participate in CPD exist in almost two-thirds of the education systems surveyed; and
  • In general, teachers are satisfied or very satisfied with their profession but consider that society does not value it.
 
Source: The Teaching Profession in Europe: Practices, Perceptions, and Policies (2015), Eurydice.

Do higher salaries lead to better teachers?

New research from the Institute for Fiscal Studies explores the question of whether offering higher teacher salaries improves pupil attainment, and finds little evidence that it does.

Estimating the impact of teachers’ pay on pupil attainment is difficult as salaries tend to reflect the experience of the teacher, therefore making it difficult to separate the impact of teacher pay from teacher experience.

However, the authors of this study have dealt with this problem by comparing pupil attainment in primary schools close to the London “fringe boundary” (on the outskirts of the city). Teachers inside this boundary receive a London weighting – around £1,000 extra each year. The researchers compared schools that were broadly comparable in pupil composition, but either side of the boundary.

The results showed little evidence that higher teacher salaries increase pupil attainment in English and maths at the end of primary school. In fact, the difference in pupil attainment between schools on either side of the pay boundary is very close to zero for both English and maths.

The authors conclude that if individual schools offered salary differentials on this scale, they would not necessarily attract more effective teachers. They also argue that there is a remarkable lack of clear evidence about which combination of measures is likely to be most effective in attracting more high quality teachers into the profession or in attracting the best teachers to particular schools.

Source: Does Offering Higher Teacher Salaries Improve Pupil Attainment? (2014), Institute for Fiscal Studies (online ‘observations’ series).