Evidence and policy

In a review of important 2017 releases, MDRC recently referenced a memo to policymakers with recommendations for increasing research use and applying evidence to all policy decisions, both educational and otherwise.

Recommendations included:

  • Programmes and policies should be independently evaluated. To ensure high-quality evaluations, they should be directly relevant to policy, free of political or other influences and credible to subjects and consumers.
  • The government should provide incentives for programmes to apply evidence results to improve their performance.
  • Utilise a tiered evidence strategy, such as is used in the Every Student Succeeds Act, to set clear guidelines for standards.
  • Existing funding sources should be applied to generate evidence. A 1% set-aside was recommended.
  • Federal and state agencies should be allowed to access and share their data for evaluation purposes.

Source: Putting evidence at the heart of making policy (February 2017), MDRC

Evidence on the long-term effects of home visiting programmes

Children from low-income families are more likely than those from higher-income families to have poor social, emotional, cognitive, behavioural and health outcomes. One approach that has helped parents and their young children is home visiting, which provides information, resources and support to expectant parents and families with young children.

This MDRC brief summarises prior evidence on the effects of four evidence-based models of home visiting using information from seven studies of families with children aged 5- to 21-years-old. Specifically, the brief looks at what the effects of home visiting are for families as children get older, and how the monetary benefits of home visiting compare with their costs.

  • The key findings of the report include:
    Evidence-based home visiting has improved outcomes for parents and children across a wide range of child ages, outcome areas, and national models.
  • Evidence-based home visiting appears to be cost-effective in the long term.
  • The largest benefits from evidence-based home visiting come through reduced spending on government programmes and increased individual earnings.

The information in this brief will inform the design of a study to assess the long-term effects of home visiting. It will suggest where this long-term follow-up study can seek to replicate prior results, where it can try to fill gaps in current knowledge, and which outcomes are important to measure in order to assess the benefits and costs of home visiting.

Source: Evidence on the long-term effects of home visiting programs: Laying the groundwork for long- term follow-up in the mother and infant home visiting program evaluation (MIHOPE). (September 2017), OPRE Report 2017-73, MDRC

Project-based learning

A working paper from MDRC builds on and updates a literature review of project-based learning (PBL) published in 2000. Focused primarily on articles and studies that have emerged in the last 17 years, the working paper discusses the principles of PBL, how PBL has been used in K–12 (Year 1–13) settings, the challenges teachers face in implementing it, how school and local factors influence its implementation and what is known about its effectiveness in improving learning outcomes.

The report suggests that the evidence for PBL’s effectiveness in improving pupil outcomes is “promising, but not proven”.  The biggest challenge to evaluating the effectiveness of PBL, the researchers suggest, is a lack of consensus about the design of PBL and how it fits in with other teaching methods. Some studies have found positive effects associated with the use of PBL. However, without a clear vision of what a PBL approach should look like, it is difficult for teachers and schools to assess the quality of their own implementation and know how to improve their approach. They also suggest that PBL implementation is particularly challenging because it changes pupil–teacher interactions and requires a shift from teacher-directed to pupil-directed inquiry and requires non-traditional methods of assessment.

The paper concludes with recommendations for advancing the PBL research literature in ways that will improve PBL knowledge and practice.

Source: Project-Based Learning: a literature review (October 2017), A MDRC Working Paper

How promising are college promise programmes?

The Detroit Promise is a US school programme administered by the Detroit Regional Chamber  that provides the city’s high school graduates with scholarships for state-funded universities and community colleges. To encourage pupils to stay in school once enrolled and to improve their academic outcomes, the Chamber and MDRC created the Detroit Promise Path. This initiative adds four components to the existing scholarship programme: campus coaches who help pupils navigate academic and personal issues, monthly financial support contingent on meeting with coaches, enhanced summer engagement and monitoring and messages informed by behavioural science through a management information system created by MDRC.

MDRC is evaluating the Detroit Promise Path using a randomised control trial design. In a new report, Alyssa Ratledge presents early findings from a pilot cohort of pupils who enrolled in autumn 2016. According to the report:

  • The Detroit Promise Path was implemented with fidelity to the model and participation was high. More than 95 percent of pupils responded to coaches’ outreach and two-thirds of enrolled pupils met with coaches as directed.
  • Pupils appreciate the programme. Ninety-six percent of surveyed pupils who had been in contact with a coach said the programme was “valuable” or “very valuable” to them.
  • The programme had a sizeable impact on enrolment in the second semester and on full-time enrolment in the first and second semesters.

Source: Enhancing promise programs to improve college access and success (July 2017), MDRC

Tiers without tears

Response to Intervention (RTI) is a process aimed at preventing pupils from falling behind in class and is arranged by hierarchy: Tier 1 referring to general class instruction, Tier 2 to small-group tutoring and Tier 3 to more individualised help. However, there are few guidelines for schools who are interested in implementing RTI, no quality standards that schools must meet and no monitoring systems in place. Based on their experience with researching RTI, MDRC has released a research brief offering some practical guidance for schools who are thinking of adopting a tiered system. These include:

  • Scheduling. Should extra support be during or after school? Will extra staff time be required? MDRC looked at this situation in depth in a 2016 report.
  • Duration and intensity. The number of pupils in an intervention and the amount of time they receive it affect both academic achievement and staffing requirements.
  • If a school adopts a curriculum to help struggling pupils, it should align with the current curriculum while being different enough to meet the needs of the pupils who are struggling. Proper use of the supplemental curriculum might require teacher training and new materials.
  • Staffing. Some programmes use only certified teachers, whereas others use paraprofessionals or volunteers. Some train before the programme starts, while others provide training before implementation followed by coaching as the programme continues. This latter method is most effective but requires more resources.
  • Intervention content. Pupil screening can help identify pupils who need extra help, although no specific curriculum is recommended for RTI.
  • Balancing Tier 2/Tier 3. Tier 2 pupils need less than Tier 3 pupils, but scheduling demands can push both sets of pupils into the same intervention at the same time of day. Tier 2 should be seen as an opportunity to prevent the pupils from needing to enter Tier 3. For Tier 2 to be effective, it must clearly be seen as separate from Tier 3. Pupils must be assigned to the proper tier, otherwise they will receive too little or too many services.

Source: Tiered systems of support: practical considerations for school districts (May 2017), MDRC

Re-engaging disconnected youth

This report by Melanie Skemer and colleagues at MDRC presents implementation and early impact results from a random assignment evaluation of the Young Adult Internship Program (YAIP), a subsidised employment programme for young people (ages 16 to 24) in New York City who are disconnected from school and work. YAIP offers participants a 10- to 12-week paid internship along with services such as job training and individual support.

MDRC reports that from July 2013 to March 2014, nearly 2,700 young people were assigned at random to either a programme group, which was offered YAIP services, or to a control group, which was not offered those services. MDRC is measuring outcomes for both groups over time to assess whether YAIP services lead to better outcomes. Data sources include administrative records on wages and postsecondary enrolment, subsidised employment payroll records and surveys conducted approximately 4, 12 and 30 months after participants entered the study.

Key findings from the report include:

  • Participation rates were high: over three-quarters of young people assigned to the programme group worked in a subsidised internship and 86 percent of those young people completed the internship.
  • Programme group members were more likely than control group members to report receiving employment services, as well as advice or support and mentorship from staff members at an agency or organisation. However, substantial numbers of control group members also reported receiving help in these areas.
  • Programme group members were more likely than the control group members to work in the year following random assignment, but the quarterly employment rates of the two groups converged after the YAIP internships ended.

MDRC plans to release a report in 2018 that will present YAIP’s final impact results, with a longer-term follow-up of 30 months, as well as the results of a benefit-cost analysis.

Source: Reengaging New York City’s disconnected youth through work: implementation and early impacts of the young adult internship program. (April 2017), OPRE Report 2017-22, MDRC