What difference does it make?

We regularly quote effect sizes in Best Evidence in Brief as a measure of the impact of an intervention or approach. But what is the impact of a normal school year on children, and how much of that impact is due to the school? A study by Hans Lutyen and colleagues, published in School Effectiveness and School Improvement, attempts to find out.

The study analysed 3,500 pupils from 20 mostly independent (private) English primary schools on four different learning outcomes. These measures, part of the Interactive Computer Adaptive System (InCAS), were reading, general maths, mental arithmetic and developed ability, the last of which measures items such as vocabulary and non-verbal pattern recognition.

Children were measured on these outcomes from Years 1 to 6. Using a regression-discontinuity approach that exploited the discontinuity between the youngest pupils in one year and the oldest pupils in the year below, the researchers were able to identify the overall progress of the children, and the extent to which this was a result of the impact of the school.

The results showed a declining impact of a school year as children got older. The effect size of Year 1 ranged from +1.18 for mental arithmetic to +0.8 for general maths. By Year 6, effect sizes varied from +0.88 for general maths to +0.49 for reading and developed ability.

The effect of schooling itself accounted for an average of between 23.5% and 43.4% of this impact across the four measures. Put another way, the effect size of schooling in Year 1 ranged from +0.55 for reading to +0.31 for developed ability. By Year 6, effect sizes had fallen to between +0.27 for general maths and +0.08 for reading and developed ability.

The researchers suggest that, when setting benchmarks for educational interventions, it is not only important to consider the phase of the educational career, but also the specific measure.

Source: The contribution of schooling to learning gains of pupils in Years 1 to 6 (February 2017), School Effectiveness and School Improvement

Evaluation of Challenge the Gap

An evaluation of the Challenge the Gap (CtG) programme for the Education Endowment Foundation found no evidence that the programme increased average achievement for either primary or secondary pupils overall.

Challenge the Gap is a two-year school improvement programme that aims to help schools improve the achievement of their disadvantaged pupils through a professional development programme for staff. The evaluation conducted by The University of Manchester, involved 21,041 pupils from 104 schools (64 primary schools and 39 secondary schools). Around 24% of pupils in the primary schools and 16% in the secondary schools were eligible for free school meals. The evaluation assessed the impact on all participating schools using 2015 Key Stage 2 or Key Stage 4 results. CtG schools were compared to schools with a similar socio-demographic profile.

No evidence was found that CtG increased the average achievement for either primary or secondary school pupils, overall. For children eligible for free school meals (FSM), those in CtG primary schools made two months’ additional progress (average effect size = +0.10) compared to similar children in non-CtG schools. In CtG secondary schools, FSM-eligible pupils made two months’ less progress compared to similar pupils in non-CtG secondary schools (average effect size = -0.10). The smaller number of FSM-eligible pupils in the trial means that these results are less secure than the overall findings.

Source: Challenge the Gap: Evaluation report and executive summary (July 2017), Education Endowment Foundation