A systematic review of RCTs in education research

The use of randomised controlled trials (RCTs) in education research has increased over the last 15 years. However, the use of RCTs has also been subject to criticism, with four key criticisms being that it is not possible to carry out RCTs in education; the research design of RCTs ignores context and experience; RCTs tend to generate simplistic universal laws of “cause and effect”; and that they are descriptive and contribute little to theory.

To assess these four key criticisms, Paul Connolly and colleagues conducted a systematic review of RCTs in education research between 1980 and 2016 in order to consider the evidence in relation to the use of RCTs in education practice.

The systematic review found a total of 1,017 RCTs completed and reported between 1980 and 2016, of which just over three-quarters have been produced in the last 10 years. Just over half of all RCTs were conducted in North America and just under a third in Europe. This finding addresses the first criticism, and demonstrates that, overall, it is possible to conduct RCTs in education research.

While the researchers also find evidence to oppose the other key criticisms, the review suggests that some progress remains to be made. The article concludes by outlining some key challenges for researchers undertaking RCTs in education.

Source:  The trials of evidence-based practice in education: a systematic review of randomised controlled trials in education research 1980–2016 (July 2018), Educational Research, DOI: 10.1080/00131881.2018.1493353

Numeracy intervention shows promise for pupils struggling with maths

The Education Endowment Foundation has published an evaluation of 1stClass@Number, a 10-week numeracy intervention, delivered by teaching assistants, that provides intensive support for pupils struggling with maths.

A randomised controlled trial was conducted in 133 schools in south and west Yorkshire. Schools each nominated four children in Year 2 to participate, and the schools were then randomly assigned to either receive the intervention or to continue with normal teaching. A team from the University of Oxford evaluated the programme, which was delivered three times a week for 10 weeks in addition to normal mathematics instruction. A process evaluation collected additional data through observations, questionnaires and phone interviews.

Results showed that the intervention had a positive effect on Quantitative Reasoning Tests (effect size = +0.18) compared to pupils in the control group. Among pupils eligible for free school meals, those in the intervention group did not make any additional progress in the Quantitative Reasoning Test compared to control group pupils.

1stClass@Number seemed to have no impact on performance in end of Key Stage 1 maths tests compared to pupils in the control group. However, there was some evidence that the intervention widened the gap in Key Stage 1 maths results between pupils eligible for free school meals and their peers.

Source: 1stClass@Number: Evaluation report and executive summary (July 2018), Education Endowment Foundation

Evidence supports Foundations: Establishing Positive Discipline Policies

Safe and Civil Schools’ Foundations: Establishing Positive Discipline Policies is a programme designed to create a safer climate within a school and its surrounding grounds. It focuses on using proactive techniques instead of punishment to facilitate change, a technique referred to as school-wide positive behavioural intervention and support (SWPBIS). Foundations helps schools adopt and maintain these techniques by providing school teams with training and data-gathering materials, multiyear training, coaching support and school visits. The programme requires a team at each school to teach the staff these techniques, and data related to safety and behaviour is used to gauge what is working and what is not.

A randomised controlled trial in 32 US elementary (primary) schools in a large urban school district demonstrated that Foundations yielded improvements in school discipline, pupil safety policy and training, staff perceptions of pupils behaviour and suspension and tardiness rates over a two-year period. To determine if positive results of the programme would generalise into a “real world” setting, a second study continued to evaluate these elementary schools for two more years while also scaling up to add all remaining elementary and secondary schools in the district, when these schools adopted the programme without support from research funding.

In total, 74 regular public schools participated in the scale-up study. All schools received two years of Foundations training where school teams learned how to implement improvements related to safety, behaviour and discipline, and to collect and analyse related data to set goals for the next year.

As in the randomised study, schools in the second study showed gains in all areas. Specifically, after Foundations training, staff reported improved pupil behaviour, with fewer suspensions, absenteeism and tardiness, with a positive relationship evidenced between years of implementation and rates of effectiveness. The authors concluded that the improvements from the Foundations training were evidence that the programme generalises well into the real world setting.

Source: A randomized evaluation of the Safe and Civil Schools model for positive behavioral interventions and supports at elementary schools in a large urban school district, School Psychology Review, Volume 42, Issue 3

Scale-up of a Safe & Civil Schools’ model for school-wide positive behavioral interventions and supports (April 2016), Psychology in the Schools, Volume 53, Issue 4

Reform needed for early years initiative

As part of their Straight Talk on Evidence initiative, the Laura and John Arnold Foundation (LJAF) has released a report that discusses new findings from a randomised controlled trial (RCT) of Tennessee’s voluntary pre-kindergarten programme for low-income children.

The Voluntary Pre-K (VPK) initiative provides Tennessee’s four-year-old children—with an emphasis on four year olds who are at-risk—an opportunity to develop school readiness skills (pre-academic and social skills). The study randomly assigned 3,131 eligible children who applied for admission at one of 79 oversubscribed VPK programmes across the state to either a programme group that was offered admission or a control group that was not (but could access other available child and family services in the community). Pupil achievement and other outcomes were measured in third grade (Year 4) using state educational records.

According to the LJAF report, the study found positive short-term effects on achievement (at the end of the pre-k year), but these effects dissipated as children entered elementary (primary) school and turned modestly negative by third grade (Year 4). At the third-grade follow-up, the control group scored significantly higher in maths and science achievement than the pre-k group.

The report offers possible reasons for the adverse effects, and suggests that the programme be reformed by incorporating evidence-based funding criteria aimed at improving its effectiveness over time.

Source: Large randomized trial finds state pre-k program has adverse effects on academic achievement. Reform is needed to increase effectiveness. Straight Talk on Evidence, The Laura and John Arnold Foundation

Do physically active lessons improve pupil engagement?

A study published in Health Education and Behavior looks at the effects of introducing physically active lessons into primary school classes. Emma Norris and colleagues used the Virtual Traveller (VT) intervention to evaluate whether physically active lessons had any effect on pupil engagement, physical activity and on-task behaviour.

Virtual Traveller is a programme of pre-prepared physically active lesson sessions delivered using classroom interactive whiteboards during regular lessons. A total of 219 children aged 8- to 9-years-old from 10 schools in Greater London took part in the cluster-randomised controlled trial. Children in the intervention schools received 10-minute VT sessions three times a week, for six weeks, during maths and English lessons. To assess the effectiveness of VT, pupils’ physical activity levels, on-task behaviour and engagement were measured at baseline (T0), at weeks two (T1) and four  (T2) of the six-week intervention, and at one week (T3) and three months (T4) post-intervention.

Pupils in the intervention group showed more on-task behaviour than those in the control at T1 and T2, but this was not maintained post-intervention. No difference in pupil engagement between the control and intervention groups was observed at any time point. VT was found to increase physical activity, but only during lesson time.

Source: Physically active lessons improve lesson activity and on-task behavior: a cluster-randomized controlled trial of the “Virtual Traveller” intervention (March 2018), Health Education & Behavior DOI: 10.1177/1090198118762106

Parenting app has positive impact on children’s development

A new randomised controlled trial of EasyPeasy, conducted by the University of Oxford’s Department of Education and published by the Sutton Trust, suggests that the EasyPeasy app had moderate positive effects on children’s concentration levels, determination and ability to make their own decisions, as well as parents’ sense of control.

EasyPeasy is a smartphone app for the parents and caregivers of children ages 2 – 6 that aims to improve school readiness by encouraging positive play and parent–child interaction. A total of 302 families with children ages 3 – 4 were recruited from eight children’s centres in the London borough of Newham. The eight centres were randomly assigned to either the intervention group or comparison group. All families in the intervention centres were given access to the EasyPeasy app, and games were sent via the app once a week over the three-month duration of the intervention.

Families in the intervention group scored higher than those in the comparison group on two parent-reported outcomes: children’s cognitive self-regulation (effect size = +0.35) and parents’ sense of control (effect size = +0.26). Parents reported that they felt more able to get their child to behave well and respond to boundaries, as well as feeling more able to stay calm when facing difficulties.

However, because of the self-report measures used in the evaluation, the researchers note that caution must be exercised when interpreting the results from the study.

These findings build on similar results from an earlier evaluation of EasyPeasy, which showed some positive benefits for children’s cognitive self-regulation and parents’ sense of control.

Source: EasyPeasy: Evaluation in Newham findings from the Sutton Trust Parental Engagement Fund (PEF) project (April 2018), The Sutton Trust