Category: i3

New findings about Investing in Innovation (i3) evaluations

A new report from the National Center for Education Evaluation and Regional Assistance (NCEE) in the US summarises the findings of the first 67 Investing in Innovation (i3) fund evaluations. The i3 fund is a tiered-evidence programme that aligns the amount of funding awarded with the strength of the prior evidence supporting the proposed intervention. The report stated that:

  • Twelve of the i3 evaluations found a statistically significant positive impact on at least one pupil academic outcome.
  • Forty of the i3 evaluations met all of the evaluation quality goals set by i3. In addition to consistency with What Works Clearinghouse (WWC) evidence standards, these goals included: independence, high-quality implementation measurement and including a sample that adequately represents those served under the grant.

Source: The Investing in Innovation Fund:  summary of 67 evaluations – final report (June 2018), National Center for Education Evaluation and Regional Assistance (NCEE), US Department of Education

Successful results for Success for All

In 2010, the Success for All Foundation (SFAF) was awarded a $50 million Investing in Innovation (i3) scale-up grant from the US Department of Education, helping to expand its comprehensive school improvement programme. As part of the grant, MDRC carried out an independent evaluation of SFAF’s scale-up initiative. MDRC’s third and final report from the evaluation examines the impact of the Success for All (SFA) reading programme over three years, its incremental cost, and the scale-up process itself.

 
A total of 37 schools were involved in the study, with 19 randomly chosen to adopt SFA in all year groups, and 18 control schools, which continued to use their existing reading programmes. Key findings included:
  • SFA is an effective vehicle for teaching phonics. In the average SFA school, the programme registered a notable, statistically significant impact on a measure of phonics skills for second-graders (age 7/8) who had been in SFA for all three years, compared with their control group counterparts. Pupils in the average SFA school performed better than the average control group school on tests of reading fluency and comprehension, but not significantly.
  • For a subgroup of special concern to policy makers and practitioners – pupils entering school with low pre-literacy skills – SFA appears to be especially effective. Second-graders (Year 3) in the average SFA school who had started kindergarten (Year 1) in the bottom half of the sample in terms of their knowledge of the alphabet and their ability to sound out words registered significantly higher scores on measures of phonics skills, word recognition, and reading fluency than similar pupils in control group schools. The impact on comprehension for this group was also positive but not statistically significant.
In conclusion, the authors say, “The scale-up findings show that, for a modest investment, SFA reliably improves the decoding skills of students in kindergarten through second grade, and that it is especially beneficial for students who begin in the lower half in these skills.”
 
Source: Scaling Up the Success for All Model of School Reform: Final Report from the Investing in Innovation (i3) Evaluation (2015), MDRC.

Success in evidence-based reform: The importance of failure

The latest blog post from Robert Slavin, a Professor in the IEE and director of the Center for Research and Reform in Education, considers the large number of randomised experiments evaluating educational programmes that find few achievement effects. This is a problem that will take on increasing significance as results from the first cohort of the US Investing in Innovation (i3) grants are released.

At the same time, the Education Endowment Foundation in the UK, much like i3, will also begin to report outcomes. It’s possible that the majority of these projects will fail to produce significant positive effects in rigorous, well-conducted evaluations. However, there is much to be learned in the process. For example, the i3 process is producing a great deal of information about what works and what does not, what gets implemented and what does not, and the match between schools’ needs and programmes’ approaches.

Encouraging findings for Success for All

A recently published report presents the first-year findings of a three-year longitudinal evaluation of Success for All (SFA), a whole-school literacy approach. As part of a US i3 scale-up grant, third-party evaluator MDRC randomly assigned 37 elementary schools in underprivileged areas across the US to SFA (n=19) or control (n=18) conditions. The study will follow children between the ages of 5 and 8.

First-year findings indicated significantly positive effects of SFA on the Woodcock Word Attack (phonics) scale, but no differences on Woodcock Letter-Word Identification. These findings are in line with prior longitudinal studies, which have found positive effects on Word Attack in the earliest school years, followed by Letter-Word Identification, and then Passage Comprehension by Year 3 and beyond.

Achievement effects were similar for all types of pupils, but not all subgroups had significant differences when they were analysed separately.

Later reports will focus on reading gains over time and on observation and treatment fidelity rating data.

Source: The Success for All Model of School Reform: Early Findings from the Investing in Innovation (i3) Scale-Up (2013), MDRC.

Promising results for Reading Recovery

A recent report published by the Consortium for Policy Research in Education (CPRE) evaluates the scale-up of the Reading Recovery programme in US schools. Reading Recovery is a short-term intervention designed to help the lowest-achieving readers in Year 2 reach average levels of literacy.

In 2010 the US Department of Education’s Investing in Innovation (i3) fund awarded a grant to expand the use of Reading Recovery by training 3,675 new Reading Recovery teachers with the capacity to reach an additional 88,200 children. The evaluation is taking place over five years, and this report, the first of three, presents results from the 2010-11 and 2011-12 school years.

The authors found that children randomly assigned to Reading Recovery outperformed those in the control group on each subscale of a standardised assessment of reading achievement (Iowa Tests of Basic Skills – ITBS). The mean of Reading Recovery children’s post-test ITBS Total Reading scores was at the 36th percentile nationally, while those in the control group had post-test scores at the 18th percentile – a difference of +18 percentile points. The authors note that Reading Recovery training and implementation were done with high fidelity in schools participating in the scale-up.

Source: Evaluation of the i3 Scale-up of Reading Recovery | Year One Report, 2011-12 (2013), Consortium for Policy Research in Education.