How promising are college promise programmes?

The Detroit Promise is a US school programme administered by the Detroit Regional Chamber  that provides the city’s high school graduates with scholarships for state-funded universities and community colleges. To encourage pupils to stay in school once enrolled and to improve their academic outcomes, the Chamber and MDRC created the Detroit Promise Path. This initiative adds four components to the existing scholarship programme: campus coaches who help pupils navigate academic and personal issues, monthly financial support contingent on meeting with coaches, enhanced summer engagement and monitoring and messages informed by behavioural science through a management information system created by MDRC.

MDRC is evaluating the Detroit Promise Path using a randomised control trial design. In a new report, Alyssa Ratledge presents early findings from a pilot cohort of pupils who enrolled in autumn 2016. According to the report:

  • The Detroit Promise Path was implemented with fidelity to the model and participation was high. More than 95 percent of pupils responded to coaches’ outreach and two-thirds of enrolled pupils met with coaches as directed.
  • Pupils appreciate the programme. Ninety-six percent of surveyed pupils who had been in contact with a coach said the programme was “valuable” or “very valuable” to them.
  • The programme had a sizeable impact on enrolment in the second semester and on full-time enrolment in the first and second semesters.

Source: Enhancing promise programs to improve college access and success (July 2017), MDRC

Tiers without tears

Response to Intervention (RTI) is a process aimed at preventing pupils from falling behind in class and is arranged by hierarchy: Tier 1 referring to general class instruction, Tier 2 to small-group tutoring and Tier 3 to more individualised help. However, there are few guidelines for schools who are interested in implementing RTI, no quality standards that schools must meet and no monitoring systems in place. Based on their experience with researching RTI, MDRC has released a research brief offering some practical guidance for schools who are thinking of adopting a tiered system. These include:

  • Scheduling. Should extra support be during or after school? Will extra staff time be required? MDRC looked at this situation in depth in a 2016 report.
  • Duration and intensity. The number of pupils in an intervention and the amount of time they receive it affect both academic achievement and staffing requirements.
  • If a school adopts a curriculum to help struggling pupils, it should align with the current curriculum while being different enough to meet the needs of the pupils who are struggling. Proper use of the supplemental curriculum might require teacher training and new materials.
  • Staffing. Some programmes use only certified teachers, whereas others use paraprofessionals or volunteers. Some train before the programme starts, while others provide training before implementation followed by coaching as the programme continues. This latter method is most effective but requires more resources.
  • Intervention content. Pupil screening can help identify pupils who need extra help, although no specific curriculum is recommended for RTI.
  • Balancing Tier 2/Tier 3. Tier 2 pupils need less than Tier 3 pupils, but scheduling demands can push both sets of pupils into the same intervention at the same time of day. Tier 2 should be seen as an opportunity to prevent the pupils from needing to enter Tier 3. For Tier 2 to be effective, it must clearly be seen as separate from Tier 3. Pupils must be assigned to the proper tier, otherwise they will receive too little or too many services.

Source: Tiered systems of support: practical considerations for school districts (May 2017), MDRC

Re-engaging disconnected youth

This report by Melanie Skemer and colleagues at MDRC presents implementation and early impact results from a random assignment evaluation of the Young Adult Internship Program (YAIP), a subsidised employment programme for young people (ages 16 to 24) in New York City who are disconnected from school and work. YAIP offers participants a 10- to 12-week paid internship along with services such as job training and individual support.

MDRC reports that from July 2013 to March 2014, nearly 2,700 young people were assigned at random to either a programme group, which was offered YAIP services, or to a control group, which was not offered those services. MDRC is measuring outcomes for both groups over time to assess whether YAIP services lead to better outcomes. Data sources include administrative records on wages and postsecondary enrolment, subsidised employment payroll records and surveys conducted approximately 4, 12 and 30 months after participants entered the study.

Key findings from the report include:

  • Participation rates were high: over three-quarters of young people assigned to the programme group worked in a subsidised internship and 86 percent of those young people completed the internship.
  • Programme group members were more likely than control group members to report receiving employment services, as well as advice or support and mentorship from staff members at an agency or organisation. However, substantial numbers of control group members also reported receiving help in these areas.
  • Programme group members were more likely than the control group members to work in the year following random assignment, but the quarterly employment rates of the two groups converged after the YAIP internships ended.

MDRC plans to release a report in 2018 that will present YAIP’s final impact results, with a longer-term follow-up of 30 months, as well as the results of a benefit-cost analysis.

Source: Reengaging New York City’s disconnected youth through work: implementation and early impacts of the young adult internship program. (April 2017), OPRE Report 2017-22, MDRC

Early impact of Diplomas Now

MDRC has published the first results from a randomised controlled trial of Diplomas Now, a whole-school reform initiative. Under the Diplomas Now programme:

  • Schools are reorganised so that small groups of teachers work consistently with the same population of students.
  • There is an intensive peer coaching system for maths and English teachers.
  • Early warning indicators are used to identify students who need different types of support.
  • Additional staff help coordinate the transformation, introduce new practices and structures, provide training and support to school staff members, provide additional services to students, and engage with families and community organisations.

In total, 62 schools (33 middle schools and 29 high schools) from 11 large urban districts were recruited. Thirty-two of the participating schools were randomly assigned to implement the Diplomas Now model (DN schools), and 30 were assigned to continue with “business as usual” (non-DN schools).

So far, the study team has been able to explore early impacts for sixth- and ninth-grade (Year 7 and 10) students moving into DN schools during the first two years of the programme. For this cohort of students, DN schools were more successful than non-DN schools in reducing the number of early warning indicators (a statistically significant 3.6 percentage point reduction). The early warning indicator was a combination of daily attendance of 85% or less, suspensions or expulsions for a total of three or more days, and failing grades in English or maths classes. However, the DN programme made no statistically significant impact on any of these measures separately. The project will continue for several more years.

Source: Addressing Early Warning Indicators: Interim Impact Findings from the Investing in Innovation (i3) Evaluation of Diplomas Now (2016), MDRC.

Examining Response to Intervention practices

MDRC has issued a new report examining Response to Intervention (RTI) practices and evaluating their effects on the reading achievement in Grades 1-3 (Years 2-4) during the 2011-12 school year.

RTI is a process aimed at preventing pupils from falling behind in class and is arranged by hierarchy: Tier 1 refers to general class teaching, Tier 2 to small-group tutoring, and Tier 3 to more intense tutoring with 1-2 children. RTI practices also include assessing all pupils at least twice yearly, the use of data to determine Tier 2 or 3 placement, and progress monitoring for pupils in Tiers 2 and 3.

A total of 146 US schools using RTI in reading for three or more years and implementing recommended practices (the impact sample) were compared to 100 randomly chosen schools in the same states (the reference sample) that may or may not have been implementing RTI at all. Researchers compared RTI practices between schools, the intensity of RTI use, and impacts on reading achievement.

Among the findings, researchers found that “impact schools” were more likely than “reference schools” to perform universal screening assessments twice a year and to provide staff to help with data gathering and teaching reading.

They also performed a regression discontinuity study of pupils who were just above and below the cut-off point of needing intervention. This showed that assignment to Tier 2 or Tier 3 intervention had a negative impact on first grade (Year 2) children on the margin of being at risk for reading delays compared to their peers who were near the cut-off point but remained in Tier 1. Second and third graders in Tiers 2 or 3 demonstrated gains that were not significantly different in the RTI and reference schools.

The authors warn that the effectiveness of the entire RTI system should not be judged based on these outcomes that look solely at a subset of RTI pupils.

Source: Evaluation of Response to Intervention Practices for Elementary School Reading (2015), MDRC.

Successful results for Success for All

In 2010, the Success for All Foundation (SFAF) was awarded a $50 million Investing in Innovation (i3) scale-up grant from the US Department of Education, helping to expand its comprehensive school improvement programme. As part of the grant, MDRC carried out an independent evaluation of SFAF’s scale-up initiative. MDRC’s third and final report from the evaluation examines the impact of the Success for All (SFA) reading programme over three years, its incremental cost, and the scale-up process itself.

 
A total of 37 schools were involved in the study, with 19 randomly chosen to adopt SFA in all year groups, and 18 control schools, which continued to use their existing reading programmes. Key findings included:
  • SFA is an effective vehicle for teaching phonics. In the average SFA school, the programme registered a notable, statistically significant impact on a measure of phonics skills for second-graders (age 7/8) who had been in SFA for all three years, compared with their control group counterparts. Pupils in the average SFA school performed better than the average control group school on tests of reading fluency and comprehension, but not significantly.
  • For a subgroup of special concern to policy makers and practitioners  pupils entering school with low pre-literacy skills  SFA appears to be especially effective. Second-graders (Year 3) in the average SFA school who had started kindergarten (Year 1) in the bottom half of the sample in terms of their knowledge of the alphabet and their ability to sound out words registered significantly higher scores on measures of phonics skills, word recognition, and reading fluency than similar pupils in control group schools. The impact on comprehension for this group was also positive but not statistically significant.
In conclusion, the authors say, “The scale-up findings show that, for a modest investment, SFA reliably improves the decoding skills of students in kindergarten through second grade, and that it is especially beneficial for students who begin in the lower half in these skills.”
 
Source: Scaling Up the Success for All Model of School Reform: Final Report from the Investing in Innovation (i3) Evaluation (2015), MDRC.