No evidence of impact for a modularisation and self-paced computer-assisted approach to college maths

One of the greatest challenges facing community colleges in the US is that most students’ maths skills are below college level. These students are often referred to developmental maths courses, however, most students never complete the course and fail to earn a college degree.

A study published in Journal of Research on Educational Effectiveness looks at whether a modularised, computer-assisted approach that allows students to move at their own pace through the developmental maths course has any impact on students’ likelihood of completing the developmental maths course, compared with more traditional teaching.

The findings of the randomised trial of 1,400 students found that although the programme was well-implemented, there was no evidence that it was any more or less effective than traditional courses at helping students complete the developmental maths course. The researchers comment that although the results are disappointing, they are important because modularisation and self-paced computer-assisted approaches are popular teaching methods.

Source: A randomized controlled trial of a modularized, computer-assisted, self-paced approach to developmental math (September 2019), Journal of Research on Educational Effectiveness

What role do principals play in improving teaching and student achievement?

A new report from the Institute of Education Sciences in the US has found that an intensive approach to helping principals (headteachers) improve their leadership practices did not improve pupil achievement or change principal practices as intended.

The study looked at the effectiveness of a professional development (PD) programme for elementary (primary) school principals that focused on helping them to conduct structured observations of teachers’ classroom teaching and provide targeted feedback. It provided nearly 200 hours of PD over two years, half of it through individualised coaching. One hundred schools from eight districts in five US states took part in the study. Within each district, schools with similar characteristics were paired together, and within each pair, one school was randomly assigned to participate in the programme for two years while the other did not.

To measure the effects on pupil achievement, the researchers compared pupil test scores in grades 3 to 5 (Years 4 to 6) for both years of programme implementation plus one additional school year. They found that, on average, pupils had similar achievement in English or maths whether they were in schools that received the principal PD programme or not.

The results of the study also found that although the programme was implemented as planned, principals did not increase the number of times they observed teachers. In fact, teachers whose principals received the PD reported receiving less frequent teaching support and feedback than teachers whose principals did not receive the PD.

Source: The effects of a principal professional development program focused on instructional leadership (October 2019), Institute of Education Sciences, US Department of Education

Long-term impacts of KIPP middle schools

The Knowledge Is Power Program (KIPP) is the largest network of public charter schools in the US, serving more than 100,000 pupils across a network of more than 240 schools. KIPP schools predominantly educate low-income pupils from underserved communities, with the goal of closing achievement gaps and preparing pupils to succeed in college.

In this Mathematica report, Thomas Coen and colleagues present the results of a long-term tracking study that follows 1,177 pupils who applied to enter 1 of 13 oversubscribed KIPP middle schools through a 5th or 6th grade (Year 6 or 7) admissions lottery ten years ago.

The study found that pupils who won a place at a KIPP middle school through the admission lottery were six percentage points more likely to enrol in a four-year college programme within two years of finishing high school than pupils who lost the lottery. After adjusting for those pupils who actually attended a KIPP school after receiving an offer (only 68% of the lottery recipients actually attended a KIPP school), the impact estimate increased to 12.9 percentage points.

The study also tracked the pupils who enrolled in college immediately after high school, and examined whether they remained in college programmes over the next two years. Pupils who attended KIPP middle schools were more likely to still be enrolled in college after two years (33%) than similar pupils who did not attend KIPP middle schools (24%). However, although rates of entering college immediately and then continuing for two years were higher for KIPP pupils, this difference was not large enough to be statistically significant.

Source:  Long-term impacts of KIPP middle schools on college enrollment and early college persistence (September 2019), Mathematica

Effects of different rewards on spelling scores and prosocial behaviour

A study published in Educational Psychology examines how different approaches to rewarding pupils affected their spelling scores and prosocial behaviour for different ability levels.

A total of 1,005 pupils, ages 9 and 10, in 28 classes were recruited from three primary schools in Singapore. Classes were randomly assigned to one of five reward conditions: competitive, cooperative, individualistic, cooperative-competitive, and cooperative-individualistic. An ABABA (A= implementation, B = withdrawal) design was used for each condition, and pupils’ spelling scores were tracked over a period of 10 weeks. Teachers were asked to rate pupils’ prosocial behaviour before and after the study.

The results showed that the different conditions did affect pupils’ spelling scores and prosocial behaviour, but that these effects depended on ability level, such that different conditions were more effective for different ability levels.  Across all five conditions, only the cooperative-competitive condition resulted in increased spelling scores and prosocial behaviour across all three ability groups, with these improvements maintained when the intervention was withdrawn. In the cooperative-competitive condition, pupils cooperated as a group and the group with the highest average spelling score (compared to other groups) was rewarded.

Source: Effects of reward pedagogy on spelling scores and prosocial behaviors in primary school students in Singapore (October 2019), Journal of Educational Psychology

How to make a systematic review’s meta-analysis high quality

Terri Piggott at Loyola University Chicago and Joshua Polanin at American Institutes for Research have published a Methodological guidance paper: High-quality meta-analysis in a systematic review, now appearing on Review of Educational Research’s Online First website.

A meta-analysis synthesises the quantitative findings of many studies on a given topic. The guidance paper outlines the characteristics that make a meta-analysis in a systematic review high quality, discussing unbiased screening and coding procedures, establishing a protocol for carrying out a review, and then discussing in depth the best practices for computing effect sizes and reporting the data.

The authors conclude that “the role of researchers using systematic review and meta-analysis is to produce both high-quality analyses and to interpret those results in ways accessible to a wide audience. A high-quality systematic review and meta-analysis is difficult and time-consuming to produce; it is worth the effort to ensure that the results inform future research and policymaking through clear discussion of the results. Researchers should consider preparing different summaries of their review tailored to their audience of researchers, policymakers, and practitioners.”

Source: Methodological guidance paper: High-quality meta-analysis in a systematic review (September 2019), Review of Educational Research

Digital feedback in primary maths

The Education Endowment Foundation has published an evaluation of Digital Feedback in Primary Maths, a programme that aims to improve primary school teachers’ feedback to pupils.

The intervention uses a tablet application called Explain Everything, diagnostic assessments, and training on effective feedback. The app allows teachers to provide pupils with digitally recorded feedback on a tablet, rather than written feedback. Pupils have the opportunity to review their feedback and develop their work further. By improving teachers’ diagnostic and feedback skills when teaching maths in primary schools, the intervention aims to ultimately improve pupils’ outcomes in maths.

To estimate the impact of Digital Feedback on maths achievement, the evaluation used a randomised controlled trial involving 2,564 pupils in 108 classes across 34 English primary schools. While the intervention took place in each school, classrooms were randomly assigned to the treatment or control group, which carried on with business-as-usual teaching.

The results of the evaluation found no evidence that pupils taking part in the programme made more progress in maths, on average (effect size = -0.04), than the control group. 

Source: Digital feedback in primary maths (September 2019), Education Endowment Foundation