$575 million programme has no impact on pupil outcomes

Findings from an evaluation of a $575 million programme to improve teacher performance found that, while sites implemented new measures of teaching effectiveness and modified personnel policies accordingly, the programme had no impact on pupil outcomes.

The Intensive Partnerships for Effective Teaching initiative, designed and funded by the Bill and Melinda Gates Foundation, aimed to dramatically improve pupil outcomes by improving pupils’ access to effective teaching. Three US school districts and four charter management organisations participated in the programme, which ran between 2009 and 2016.

The final evaluation report, published by the RAND Corporation, found that by the end of 2014-15, outcomes for pupils in the settings that took part in the initiative were not better than outcomes for pupils in similar settings that did not take part. There was no evidence that low-income minority (LIM) pupils had greater access than non-LIM pupils to effective teaching. In addition, it found very few instances of improvement in the effectiveness of teaching overall, and no improvement in the effectiveness of newly hired teachers compared to experienced teachers. The evaluation also found no increase in the retention of effective teachers, although there was some decline in the retention of ineffective teachers in most settings that took part in the initiative.

The report states several possible reasons that the initiative failed to achieve its goals for improving pupil outcome:

  • incomplete implementation of the key policies and practices
  • the influence of external factors, such as state-level policy changes during the initiative
  • insufficient time for effects to appear
  • a flawed theory of action
  • a combination of all these factors.

 

Source:  Improving teaching effectiveness: Final report: The Intensive Partnerships for Effective Teaching through 2015–2016 (2018), RAND Corporation.

How ESSA supports social and emotional learning

The Every Student Succeeds Act (ESSA) allows US states to use federal funding to adopt research-proven programmes to improve pupil achievement. This includes social-emotional learning (SEL) programmes. To offer some guidance and inform decision makers, RAND has released a report, Social and emotional learning interventions under the Every Student Succeeds Act, which reviews recent evidence on these programmes. The report discusses how ESSA supports SEL programmes and outlines the programmes that meet the ESSA evidence standards.

Specifically, authors found 60 SEL programmes that met strong, moderate, or promising evidence standards in grades K-12 (Years   1 to 13). Most were evaluated at the primary school level in urban communities with minority populations. A second report describes these programmes and the research that supports them in detail.

Source: Social and emotional learning interventions under the Every Student Succeeds Act: evidence review (2017), RAND Corporation, RR-2133-WF

Is personalised learning effective?

A new research brief by John F Pane and colleagues at the RAND Corporation asks the question: “Does personalised learning improve pupil learning more than other educational approaches?” As part of their report, the authors present findings from an evaluation of personalised learning (PL) schools conducted by RAND Corporation researchers for the Bill and Melinda Gates Foundation.

The research team analysed maths and reading scores for approximately 5,500 pupils in 32 US schools that received funding from the Next Generation Learning Challenges (NGLC) initiative to support highly personalised approaches to learning. These schools took the Northwest Evaluation Association (NWEA) Measures of Academic Progress (MAP) mathematics and reading assessments for one academic year: autumn 2014 to spring 2015. The research team compared the achievement of pupils in PL schools with matched peers attending non-PL schools and national norms.

Key findings from the research brief include:

  • Early evidence suggests that PL can improve achievement for pupils, regardless of their starting level of achievement.
  • Benefits of PL may take some time to emerge. Analyses suggest that effects may be more positive after schools have experience implementing PL.
  • To date, the field lacks evidence on which practices are most effective or what policies must be in place to maximise the benefits.

The authors note that additional research is needed using more rigorous experimental studies.

Source: How Does Personalized Learning Affect Student Achievement? RAND Corporation, 2017.

Examining the evidence on out-of-school-time programmes

Out-of-school-time (OST) programmes typically provide children with additional academic lessons outside of school hours and/or recreational and enrichment activities. To examine the evidence base on OST programmes, Jennifer McCombs and colleagues from the RAND Corporation reviewed meta-analyses and large-scale, rigorous experimental and quasi-experimental evaluations of after-school and summer programmes. Their review included specialty programmes (eg, sports or arts programmes); multipurpose programmes (eg, Boys and Girls clubs); and academic programmes (eg, summer learning programmes).

After reviewing the research, the authors compiled the following conclusions:

  • OST programmes provide measurable benefits to children and families on outcomes directly related to programme content.
  • Academic OST programmes with sufficient “dosage” (measured by the hours of content provided) can demonstrably improve pupil achievement.
  • Programme quality and intentionality influence outcomes.
  • Children need to attend regularly to measurably benefit from programming.

The authors provide a complete list of studies reviewed and their key findings.

A previous issue of Best Evidence in Brief included a study by the Nuffield Foundation, which examines the effect of OST study programmes on GCSE performance in England.

Source: The value of out-of-school-time programmes (2017), PE-267-WF, RAND Corporation

New research on dual-language immersion programmes

A new research brief by Jennifer L Steele and colleagues, published by the RAND Corporation, presents new research on dual-language immersion (DLI) programmes. These programmes provide both native English speakers and children learning English as an additional language (EAL) with general academic teaching in two languages from kindergarten (Year 1) onwards.

In partnership with the American Councils on International Education and the Portland Public Schools in Oregon (PPS), the authors conducted a random-assignment study of DLI education. The goal was to estimate the causal effects of the district’s DLI programmes on pupil performance over time in reading, mathematics and science, and on EAL pupils’ reclassification as English proficient.

PBS allocates immersion slots using a random-assignment lottery process for those who apply to the programmes. The study focused on 1,625 DLI lottery applicants in the kindergarten cohorts from 2004–2005 to 2010–2011. Pupil achievement was tracked until 2013–2014.

Key findings of the study were as follows:

  • PPS pupils randomly assigned to dual-language immersion programmes outperformed their peers on state reading tests by 13% of a standard deviation in grade 5 (Year 6) and by 22% of a standard deviation in grade 8 (Year 9).
  • Immersion-assigned pupils did not show statistically significant benefits or deficits in terms of mathematics or science performance.
  • There were no clear differences in the effects of dual-language immersion according to pupils’ native language.
  • EAL pupils assigned to dual-language immersion were more likely than their peers to be classified as English proficient by grade 6 (Year 7). This effect was mostly attributed to EAL pupils whose native language was the same as one of the two languages taught.

Source: Dual-language immersion programs raise student achievement in English (2017), RAND Corporation Research Brief, RB-9903

High marks for high-quality preschool

A new report from the RAND Corporation presents findings from a review of research on preschool programmes in the US. Specifically, the report examines whether high-quality preschool produces favourable effects for participating children and their families, the magnitude of the impacts, and how long the benefits last. The preschool programmes ran for one or two years (equivalent to the Reception year, or the Nursery and Reception years, in the UK).

The researchers analysed evaluation findings from 15 full-scale, publicly funded preschool programmes implemented at the national, state, and local levels. The researchers examined evidence for the specific programmes, as well as results from syntheses across multiple preschool programme evaluations. In cases where children had been followed beyond the preschool years, they also considered research regarding longer-term effects.

Key findings of their review include:

  • Favourable impacts have been demonstrated for part- and full-day preschool programmes, as well as one- and two-year programmes, but the research is not definitive about the comparative effectiveness of these options.
  • High quality is a common element among the preschool programmes with the largest effects on school readiness and with sustained effects at older ages. These effective programmes include such features as well-trained classroom teachers who are provided with ongoing professional development support through coaching and other mechanisms, a learning environment that supports teachers and children, a well-defined curriculum that is implemented with fidelity in the classroom and aligned with the early school years, and ongoing monitoring of programme quality and other metrics that support continuous quality improvement.
  • High-quality preschool programmes show sustained benefits for aspects of school performance other than achievement scores, such as lower rates of special education use, fewer students being held back a year, and higher rates of high school graduation.
  • Children across the income spectrum may benefit from high-quality preschool, but the impacts tend to be larger for more disadvantaged children. The researchers also examined evidence on the investment value of preschool, reporting that estimates of the economic return for full-scale high-quality preschool range from about $2 to $4 for every $1 invested.

Source: Informing Investments in Preschool Quality and Access in Cincinnati (2015), RAND Corporation.