Examining the evidence on out-of-school-time programmes

Out-of-school-time (OST) programmes typically provide children with additional academic lessons outside of school hours and/or recreational and enrichment activities. To examine the evidence base on OST programmes, Jennifer McCombs and colleagues from the RAND Corporation reviewed meta-analyses and large-scale, rigorous experimental and quasi-experimental evaluations of after-school and summer programmes. Their review included specialty programmes (eg, sports or arts programmes); multipurpose programmes (eg, Boys and Girls clubs); and academic programmes (eg, summer learning programmes).

After reviewing the research, the authors compiled the following conclusions:

  • OST programmes provide measurable benefits to children and families on outcomes directly related to programme content.
  • Academic OST programmes with sufficient “dosage” (measured by the hours of content provided) can demonstrably improve pupil achievement.
  • Programme quality and intentionality influence outcomes.
  • Children need to attend regularly to measurably benefit from programming.

The authors provide a complete list of studies reviewed and their key findings.

A previous issue of Best Evidence in Brief included a study by the Nuffield Foundation, which examines the effect of OST study programmes on GCSE performance in England.

Source: The value of out-of-school-time programmes (2017), PE-267-WF, RAND Corporation

New research on dual-language immersion programmes

A new research brief by Jennifer L Steele and colleagues, published by the RAND Corporation, presents new research on dual-language immersion (DLI) programmes. These programmes provide both native English speakers and children learning English as an additional language (EAL) with general academic teaching in two languages from kindergarten (Year 1) onwards.

In partnership with the American Councils on International Education and the Portland Public Schools in Oregon (PPS), the authors conducted a random-assignment study of DLI education. The goal was to estimate the causal effects of the district’s DLI programmes on pupil performance over time in reading, mathematics and science, and on EAL pupils’ reclassification as English proficient.

PBS allocates immersion slots using a random-assignment lottery process for those who apply to the programmes. The study focused on 1,625 DLI lottery applicants in the kindergarten cohorts from 2004–2005 to 2010–2011. Pupil achievement was tracked until 2013–2014.

Key findings of the study were as follows:

  • PPS pupils randomly assigned to dual-language immersion programmes outperformed their peers on state reading tests by 13% of a standard deviation in grade 5 (Year 6) and by 22% of a standard deviation in grade 8 (Year 9).
  • Immersion-assigned pupils did not show statistically significant benefits or deficits in terms of mathematics or science performance.
  • There were no clear differences in the effects of dual-language immersion according to pupils’ native language.
  • EAL pupils assigned to dual-language immersion were more likely than their peers to be classified as English proficient by grade 6 (Year 7). This effect was mostly attributed to EAL pupils whose native language was the same as one of the two languages taught.

Source: Dual-language immersion programs raise student achievement in English (2017), RAND Corporation Research Brief, RB-9903

High marks for high-quality preschool

A new report from the RAND Corporation presents findings from a review of research on preschool programmes in the US. Specifically, the report examines whether high-quality preschool produces favourable effects for participating children and their families, the magnitude of the impacts, and how long the benefits last. The preschool programmes ran for one or two years (equivalent to the Reception year, or the Nursery and Reception years, in the UK).

The researchers analysed evaluation findings from 15 full-scale, publicly funded preschool programmes implemented at the national, state, and local levels. The researchers examined evidence for the specific programmes, as well as results from syntheses across multiple preschool programme evaluations. In cases where children had been followed beyond the preschool years, they also considered research regarding longer-term effects.

Key findings of their review include:

  • Favourable impacts have been demonstrated for part- and full-day preschool programmes, as well as one- and two-year programmes, but the research is not definitive about the comparative effectiveness of these options.
  • High quality is a common element among the preschool programmes with the largest effects on school readiness and with sustained effects at older ages. These effective programmes include such features as well-trained classroom teachers who are provided with ongoing professional development support through coaching and other mechanisms, a learning environment that supports teachers and children, a well-defined curriculum that is implemented with fidelity in the classroom and aligned with the early school years, and ongoing monitoring of programme quality and other metrics that support continuous quality improvement.
  • High-quality preschool programmes show sustained benefits for aspects of school performance other than achievement scores, such as lower rates of special education use, fewer students being held back a year, and higher rates of high school graduation.
  • Children across the income spectrum may benefit from high-quality preschool, but the impacts tend to be larger for more disadvantaged children. The researchers also examined evidence on the investment value of preschool, reporting that estimates of the economic return for full-scale high-quality preschool range from about $2 to $4 for every $1 invested.

Source: Informing Investments in Preschool Quality and Access in Cincinnati (2015), RAND Corporation.

Study of personalised learning shows promising evidence – or does it?

The US National Education Policy Center’s Think Twice Think Tank Review Project recently reviewed a RAND study on personalised learning. The RAND study examined the effects of three school-wide personalised learning initiatives on pupil achievement to try to find evidence linking specific learning strategies to achievement outcomes.

RAND defined “personalised learning” (PL) as incorporating five specific characteristics including data-supported pupil goals accessible to teachers and pupils, and personalised learning of each pupil’s choice with in-school support and learning outside school.

The RAND study compared the MAP reading and maths scores of 11,000 pupils in 62 schools who had been using a personalised learning approach for two years to the scores of pupils matched at baseline to serve as a comparison. Researchers found higher achievement scores for the PL group, especially at primary age. In addition, the study showed that personalised learners’ scores increased at a greater rate than the nation’s scores. Overall, researchers deemed PL promising practice.

However, the Think Twice Think Tank Review Project disagrees. In a review of the study researchers felt that the study’s limitations prevented it from demonstrating true evidence of promising practice. First, reviewers noted that only pupil involvement in analysing their own data and goal setting was associated with consistent gains. They pointed out that two of the attributes ascribed to the success of PL – flexible learning environments and student grouping – were also used in schools not using PL. They noted that the largest departure from usual classroom practice (competency-based progression) was not used in the majority of the experimental schools, casting doubt on its pertinence. In addition, reviewers were dubious about the generalisability of the findings because 90% of the study schools were charter schools.

Think Tank reviewers concluded that the study suggests there may indeed be personalised practices associated with test score gains, but that the practices in the three experimental models weren’t drastically different than practices in untreated schools. The study’s limitations cast doubt on the models’ generalisation.

Source: Continued Progress Promising Evidence on Personalized Learning (2015), RAND Corporation, and Review of Continued Progress: Promising Evidence on Personalized Learning (2016), NEPC.

Early results in for teacher leadership programme

A US programme intended to boost pupil achievement by providing teachers with two years of professional development, including formal training sessions and meetings with a leadership coach, is showing signs of potential, according to a new RAND Corporation report.

The Leading Educators (LE) Fellowship programme, selects mid-career teachers through a competitive application process. To examine the impact of LE, researchers are comparing pupil achievement gains for teachers who participated in the programme as fellows or mentees with the pupil achievement gains of other teachers.According to RAND, early findings of the programme are mixed, but suggest that it shows promise in improving pupil achievement. Specifically, they report:

  • Among fellows, there are both some statistically significant positive and negative programme effects on pupil achievement, with results that vary across states, subject areas, and model specifications.
  • Among mentee teachers, for whom sample sizes are larger, there is some suggestive evidence of impacts on pupil achievement — in particular, marginally significant and significant positive programme effects among mentees who teach maths and social studies, respectively, in Louisiana.
  • The impact of the programme on teacher retention is unclear, with no consistent pattern of retention impacts across cohorts or states.

The authors note that the current results are based on few years of data and on a small sample of teachers, and results may change when there are more fellows and mentored teachers included in future studies.

Source: Examining the Early Impacts of the Leading Educators Fellowship on Student Achievement and Teacher Retention (2015), RAND Corporation.

Tech for tots

This report from the RAND Corporation identifies goals for technology use in early education. The information is based on findings from a literature review and a May forum that RAND hosted on the topic. The authors say that trends in US education suggest that young children may need to achieve basic digital literacy before starting kindergarten (Year 1), and the presence of a digital divide suggests that children from low-income families may need the most support to ensure readiness in digital literacy (see the previous story for research on technology for at-risk pupils). Based on their research, the authors present the following recommendations:

  • Technology is one of many tools: When technology is used as one tool in a larger toolbox, it can provide the greatest benefits while continuing to allow for the use of other learning tools and activities when they are likely to be most effective in supporting skill growth.
  • Support school readiness in digital literacy: With increasing standards for technology use in US elementary schools, forum experts agreed that all children, particularly those from deprived families, could benefit from acquiring basic technology literacy skills in early childhood education (ECE) settings to ensure readiness for technology use in the classroom.
  • Help narrow the digital divide: Technology use in ECE settings has the potential to address both aspects of the digital divide: access and use. In ECE settings, children from low-income families can access technology that is not available in the home, and they can be taught to use technology in ways that are more likely to result in skill growth and learning, thereby addressing disparities in use.
  • Expand resources for providers and families: Goals for technology use in ECE settings need not focus exclusively on use among children, as there are many ways that technology can be used to support providers and families as they, in turn, support the education of young children.

Source: Getting on the Same Page: Identifying Goals for Technology Use in Early Childhood Education (2014), RAND Corporation.