New findings about Investing in Innovation (i3) evaluations

A new report from the National Center for Education Evaluation and Regional Assistance (NCEE) in the US summarises the findings of the first 67 Investing in Innovation (i3) fund evaluations. The i3 fund is a tiered-evidence programme that aligns the amount of funding awarded with the strength of the prior evidence supporting the proposed intervention. The report stated that:

  • Twelve of the i3 evaluations found a statistically significant positive impact on at least one pupil academic outcome.
  • Forty of the i3 evaluations met all of the evaluation quality goals set by i3. In addition to consistency with What Works Clearinghouse (WWC) evidence standards, these goals included: independence, high-quality implementation measurement and including a sample that adequately represents those served under the grant.

Source: The Investing in Innovation Fund:  summary of 67 evaluations – final report (June 2018), National Center for Education Evaluation and Regional Assistance (NCEE), US Department of Education

What is the impact of pay-for-performance?

Teacher Incentive Fund (TIF) grants were awarded in 2010 by the US Department of Education to support performance-based compensation systems for teachers and principals in high-need schools.

In order to assess the impacts of pay-for-performance on educator (teachers and principals) and pupil outcomes, an experimental study design was used in ten US school districts to randomly assign elementary and middle schools to treatment and control groups. Both groups implemented the same performance-based compensation system, but in the control schools, the pay-for-performance element was replaced by a one percent bonus paid to all teachers and principals regardless of performance. A fourth and final report from this evaluation has now been published, covering all four years of the programme (between 2011 and 2015).

Among the key findings are that pay-for-performance had small, positive impacts on pupil achievement by the second year of implementation. From that year onward, reading and maths achievement was higher by 1 to 2 percentile points in schools that offered performance bonuses than in schools that did not. However, it was not entirely clear how this improvement was achieved. The impacts of pay-for-performance on classroom observation ratings did not appear to explain the impacts on pupil achievement, and in treatment schools as many as 40% of teachers were unaware that they could earn a performance bonus.

Source: Evaluation of the Teacher Incentive Fund: Final report on implementation and impacts of pay-for-performance across four years (NCEE 2017-4004),(December 2017), National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education.

Report finds mixed effects for Saxon Math

The Institute of Education Sciences (IES) in the US has released a new research report on Saxon Math, and findings show mixed results for the programme.

Saxon Math is a curriculum for pupils in grades K-12 (Years 1-13). It uses an incremental structure that distributes content throughout the year. For the IES report, researchers reviewed studies of Saxon Math’s primary courses, which include kindergarten (Year 1) through pre-algebra. Out of 26 studies eligible for review, five studies fell within the scope of the What Work Clearinghouse’s (WWC) primary maths topic area and met WWC design standards. These five studies included 8,855 pupils in grades 1–3 and 6–8 in 149 schools across at least 18 states.

According to the report, the estimated impact of the intervention on outcomes in the mathematics achievement domain was positive and substantively important in two studies and indeterminate in three studies. The authors conclude that Saxon Math has mixed effects on maths test scores of pupils in primary classes.

Source: Saxon Math (May 2017), US Department of Education, Institute of Education Sciences, What Works Clearinghouse

What do pupils believe about learning and intelligence?

This study examined reported attitudes and beliefs about growth mindset (the belief that intelligence and academic ability are not fixed and can be increased through effort and learning) for a sample of 103,066 pupils and 5,721 teachers in grades 4–12 (Years 5–13) in Nevada’s Clark County School District in the US.

Three-quarters of pupils reported having beliefs that are consistent with a growth mindset. The average growth mindset score across all pupils was 4 on a scale of 1 to 5 (where 1 indicates agreement with all statements that suggest a fixed-ability mindset, and 5 indicates disagreement). In addition, reported beliefs were found to differ depending on pupils’ ethnicity, school year, prior achievement and whether pupils were native English speakers or not. For example, the average growth mindset score for pupils with English as an Additional Language (EAL) was lower (3.5) than the average growth mindset score for non-EAL pupils (4.0). Lower-achieving pupils reported lower levels of growth mindset than their higher-achieving peers (a difference of 0.8 points).

Teachers’ average growth mindset score was 0.5 points higher than their pupils’ (4.5 compared with 4.0). For the most part, their beliefs regarding growth mindset did not vary significantly depending on the characteristics of the pupils attending their schools.

Source: Growth mindset, performance avoidance, and academic behaviors in Clark County School District (REL 2017–226) (April 2017), US Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory West

What is the evidence to support reading interventions?

A review from the National Center for Education Evaluation and Regional Assistance in the US assesses the evidence base supporting reading interventions in grades 1–3 (Years 2–4 in the UK) to improve reading outcomes for pupils struggling with typical classroom reading lessons.

The findings are based on studies of 20 interventions conducted in the US that Russell Gersten and colleagues identified that met the What Works Clearinghouse evidence standards. Of these 20 interventions, 19 produced positive or potentially positive effects in at least one area of reading. Interventions in grade 1 (Year 2) produced lower effects in reading comprehension (+0.39) than in word and pseudo-word reading (+0.45), but higher effects than in passage reading fluency (+0.23). For grade 2 and 3 (Years 3 and 4) interventions, the weighted mean effects in reading comprehension (+0.33) were lower than those for both word and pseudo-word reading (+0.46) and passage reading fluency (+0.37). The strongest and most consistent effects were found in word and pseudo-word reading for all three grades.

Although the evidence supports the efficacy of reading interventions, the review points out that the majority of interventions evaluated are interventions for individual pupils, as opposed to small-group interventions which are more typical in school settings. In addition, most of the interventions include high levels of ongoing support for teachers.

Source: What is the evidence base to support reading interventions for improving student outcomes in grades 1–3? (April 2017), US Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southeast (REL 2017–271)

Using data to help ensure equity in suspensions

A new guide is available from the Institute of Education Sciences to help educators in the US to use data to determine if any ethnic groups are being disproportionately suspended or expelled within a school or district, and if so, how to use data to promote equity among all ethnic groups.

The guide is divided into two sections. The first describes how to use multiple data to analyse if a group is being disproportionately suspended or expelled, and how to determine the effectiveness of any interventions that might be in place. It also describes the data that can be used to analyse factors that may be contributing to any disproportion. In cases where a school or district determines there are inequalities that may be unjust, the second section outlines a process that helps promote equitable discipline, called Plan-Do-Study-Act. One district’s successful experience using the Plan-Do-Study-Act process is described in detail. The back of the guide contains websites and resources related to equity in school discipline and quality improvement processes.

Source: School discipline data indicators: A guide for districts and schools (April 2017), US Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Northwest (REL 2017–240)