Re-engaging disconnected youth

This report by Melanie Skemer and colleagues at MDRC presents implementation and early impact results from a random assignment evaluation of the Young Adult Internship Program (YAIP), a subsidised employment programme for young people (ages 16 to 24) in New York City who are disconnected from school and work. YAIP offers participants a 10- to 12-week paid internship along with services such as job training and individual support.

MDRC reports that from July 2013 to March 2014, nearly 2,700 young people were assigned at random to either a programme group, which was offered YAIP services, or to a control group, which was not offered those services. MDRC is measuring outcomes for both groups over time to assess whether YAIP services lead to better outcomes. Data sources include administrative records on wages and postsecondary enrolment, subsidised employment payroll records and surveys conducted approximately 4, 12 and 30 months after participants entered the study.

Key findings from the report include:

  • Participation rates were high: over three-quarters of young people assigned to the programme group worked in a subsidised internship and 86 percent of those young people completed the internship.
  • Programme group members were more likely than control group members to report receiving employment services, as well as advice or support and mentorship from staff members at an agency or organisation. However, substantial numbers of control group members also reported receiving help in these areas.
  • Programme group members were more likely than the control group members to work in the year following random assignment, but the quarterly employment rates of the two groups converged after the YAIP internships ended.

MDRC plans to release a report in 2018 that will present YAIP’s final impact results, with a longer-term follow-up of 30 months, as well as the results of a benefit-cost analysis.

Source: Reengaging New York City’s disconnected youth through work: implementation and early impacts of the young adult internship program. (April 2017), OPRE Report 2017-22, MDRC

Early impact of Diplomas Now

MDRC has published the first results from a randomised controlled trial of Diplomas Now, a whole-school reform initiative. Under the Diplomas Now programme:

  • Schools are reorganised so that small groups of teachers work consistently with the same population of students.
  • There is an intensive peer coaching system for maths and English teachers.
  • Early warning indicators are used to identify students who need different types of support.
  • Additional staff help coordinate the transformation, introduce new practices and structures, provide training and support to school staff members, provide additional services to students, and engage with families and community organisations.

In total, 62 schools (33 middle schools and 29 high schools) from 11 large urban districts were recruited. Thirty-two of the participating schools were randomly assigned to implement the Diplomas Now model (DN schools), and 30 were assigned to continue with “business as usual” (non-DN schools).

So far, the study team has been able to explore early impacts for sixth- and ninth-grade (Year 7 and 10) students moving into DN schools during the first two years of the programme. For this cohort of students, DN schools were more successful than non-DN schools in reducing the number of early warning indicators (a statistically significant 3.6 percentage point reduction). The early warning indicator was a combination of daily attendance of 85% or less, suspensions or expulsions for a total of three or more days, and failing grades in English or maths classes. However, the DN programme made no statistically significant impact on any of these measures separately. The project will continue for several more years.

Source: Addressing Early Warning Indicators: Interim Impact Findings from the Investing in Innovation (i3) Evaluation of Diplomas Now (2016), MDRC.

Examining Response to Intervention practices

MDRC has issued a new report examining Response to Intervention (RTI) practices and evaluating their effects on the reading achievement in Grades 1-3 (Years 2-4) during the 2011-12 school year.

RTI is a process aimed at preventing pupils from falling behind in class and is arranged by hierarchy: Tier 1 refers to general class teaching, Tier 2 to small-group tutoring, and Tier 3 to more intense tutoring with 1-2 children. RTI practices also include assessing all pupils at least twice yearly, the use of data to determine Tier 2 or 3 placement, and progress monitoring for pupils in Tiers 2 and 3.

A total of 146 US schools using RTI in reading for three or more years and implementing recommended practices (the impact sample) were compared to 100 randomly chosen schools in the same states (the reference sample) that may or may not have been implementing RTI at all. Researchers compared RTI practices between schools, the intensity of RTI use, and impacts on reading achievement.

Among the findings, researchers found that “impact schools” were more likely than “reference schools” to perform universal screening assessments twice a year and to provide staff to help with data gathering and teaching reading.

They also performed a regression discontinuity study of pupils who were just above and below the cut-off point of needing intervention. This showed that assignment to Tier 2 or Tier 3 intervention had a negative impact on first grade (Year 2) children on the margin of being at risk for reading delays compared to their peers who were near the cut-off point but remained in Tier 1. Second and third graders in Tiers 2 or 3 demonstrated gains that were not significantly different in the RTI and reference schools.

The authors warn that the effectiveness of the entire RTI system should not be judged based on these outcomes that look solely at a subset of RTI pupils.

Source: Evaluation of Response to Intervention Practices for Elementary School Reading (2015), MDRC.

Successful results for Success for All

In 2010, the Success for All Foundation (SFAF) was awarded a $50 million Investing in Innovation (i3) scale-up grant from the US Department of Education, helping to expand its comprehensive school improvement programme. As part of the grant, MDRC carried out an independent evaluation of SFAF’s scale-up initiative. MDRC’s third and final report from the evaluation examines the impact of the Success for All (SFA) reading programme over three years, its incremental cost, and the scale-up process itself.

A total of 37 schools were involved in the study, with 19 randomly chosen to adopt SFA in all year groups, and 18 control schools, which continued to use their existing reading programmes. Key findings included:
  • SFA is an effective vehicle for teaching phonics. In the average SFA school, the programme registered a notable, statistically significant impact on a measure of phonics skills for second-graders (age 7/8) who had been in SFA for all three years, compared with their control group counterparts. Pupils in the average SFA school performed better than the average control group school on tests of reading fluency and comprehension, but not significantly.
  • For a subgroup of special concern to policy makers and practitioners  pupils entering school with low pre-literacy skills  SFA appears to be especially effective. Second-graders (Year 3) in the average SFA school who had started kindergarten (Year 1) in the bottom half of the sample in terms of their knowledge of the alphabet and their ability to sound out words registered significantly higher scores on measures of phonics skills, word recognition, and reading fluency than similar pupils in control group schools. The impact on comprehension for this group was also positive but not statistically significant.
In conclusion, the authors say, “The scale-up findings show that, for a modest investment, SFA reliably improves the decoding skills of students in kindergarten through second grade, and that it is especially beneficial for students who begin in the lower half in these skills.”
Source: Scaling Up the Success for All Model of School Reform: Final Report from the Investing in Innovation (i3) Evaluation (2015), MDRC.

First-year effects of the Communities in Schools programme

MDRC has released a report describing the first-year results of a randomised study of the Communities in Schools (CIS) programme. This is a programme designed to prevent at-risk middle and high school pupils in the US from dropping out by providing them with academic, behavioural, and emotional supports through an organised, in-school, case-managed system.

The study took place in 28 schools during the 2012-2013 school year. The sample included 2,230 pupils, of which 1,140 were assigned to the CIS group, and 1,090 were assigned to receive the regular support services provided by their schools. Both groups were predominantly ethnic minority and low income, and similar in terms of attendance rate, academic achievement, and EAL status, the only difference being that the experimental group was 2.8% more likely to receive free- or reduced-price lunches.

Following one year of services, CIS pupils were more likely than the controls to report having positive relationships with adults outside the home or school setting, to report positive peer relations, and to view education as valuable. However, the case-managed group did not demonstrate more gains in attendance, academics, or discipline than the control group.

The authors discuss areas for improving the programme and will examine the second year of data to continue to assess findings.

Source: Case Management for Students at Risk of Dropping Out: Implementation and Interim Impact Findings from the Communities In Schools Evaluation (2015), MDRC.

The varying impact of an earlier years programme

A paper from MDRC analyses variation in the effects of the Head Start programme in the United States using data from the Head Start Impact Study.

Head Start is the largest US federal programme for early years development of disadvantaged children and has served more than 30 million children since 1965.

The MDRC paper confirms previous studies that suggested substantial variation in the effects of Head Start in relation to the individual, subgroup, and between Head Start Centers.

The main findings were:

  • Head Start improved cognitive outcomes in children with the lowest cognitive skills and tended to reduce disparities between children in key cognitive outcomes.
  • Dual-language and Spanish-speaking children with low pretest scores gained the most from Head Start.
  • Much of the positive effect of Head Start came from mitigating for limited prior English; the positive effect on children with limited English persisted for at least three years.

The added value of Head Start compared with local alternatives varied substantially between Centers and reflected differences in provision (such as hours of care, teacher education, and classroom quality).

Some Head Start Centers were much more effective than alternatives (including parental care) and others were much less effective than alternatives.

Source: Quantifying Variation in Head Start Effects on Young Children’s Cognitive and Socio-Emotional Skills Using Data from the National Head Start Impact Study (2015), MDRC.