How to make a systematic review’s meta-analysis high quality

Terri Piggott at Loyola University Chicago and Joshua Polanin at American Institutes for Research have published a Methodological guidance paper: High-quality meta-analysis in a systematic review, now appearing on Review of Educational Research’s Online First website.

A meta-analysis synthesises the quantitative findings of many studies on a given topic. The guidance paper outlines the characteristics that make a meta-analysis in a systematic review high quality, discussing unbiased screening and coding procedures, establishing a protocol for carrying out a review, and then discussing in depth the best practices for computing effect sizes and reporting the data.

The authors conclude that “the role of researchers using systematic review and meta-analysis is to produce both high-quality analyses and to interpret those results in ways accessible to a wide audience. A high-quality systematic review and meta-analysis is difficult and time-consuming to produce; it is worth the effort to ensure that the results inform future research and policymaking through clear discussion of the results. Researchers should consider preparing different summaries of their review tailored to their audience of researchers, policymakers, and practitioners.”

Source: Methodological guidance paper: High-quality meta-analysis in a systematic review (September 2019), Review of Educational Research

What are the best self-regulated learning strategies for Chinese pupils?

Self-regulated learning has been regarded as essential for effective learning. Research suggests that self-regulated learning is associated with academic performance, but different self-regulated learning strategies are not equally effective. Addressing the gap that occurred for Chinese pupils because few studies conducted in Asia were included in a previous meta-analysis, a meta-analysis published in Frontiers in Psychology has investigated what the most effective strategies for Chinese pupils were.

Using Chinese academic databases, Junyi Li and colleagues analysed 264 independent samples that involved 23,497 participants from 59 studies. In order to be included in this meta-analysis, studies had to be conducted in real teaching situations; studies based on online learning environments were excluded. Furthermore, participants had to be primary, middle, or secondary school pupils in China. The effect sizes of self-regulated learning strategies on academic achievement were analysed. The results showed that: 

  • Among the self-regulated learning strategies, self-efficacy (ES = +0.70), self-evaluation (ES = +0.72), and task strategies (ES = +0.60) had relatively large effect sizes on academic achievement.
  • On the other hand, the effect sizes of goal orientation (ES = +0.09) and attributions (ES = +0.27) were relatively small. 
  • The effect sizes of self-regulated learning on science (ES = +0.45) were larger than those on language (ES = +0.29).

The authors suggest that task strategies supported learning by reducing a task to its key parts, that self-evaluation helped learners compare results with their goals, and that self-efficacy helped learners to use their resources. 

Source: What are the effects of self-regulation phases and strategies for Chinese students? A meta-analysis of two decades research of the association between self-regulation and academic performance (December 2018), Frontiers in Psychology

The effect of screen time on academic performance

A meta-analysis examining the evidence between overall screen time, specific screen-based activities, and academic performance found that overall screen time is not related to children’s and teens’ academic achievement, yet the type of screen time is.

Mireia Adelantado-Renau and colleagues in Spain found that TV and video game time greater than two hours a day was associated with poorer academic achievement, while internet and mobile phone time was not. In addition, the negative effects on academic performance were larger for teens than for children.

The meta-analysis included 58 studies from 23 countries that met its inclusion criteria, encompassing the academic achievement of 106,000 4–18 year olds (assessed by school grades, standardised tests, and academic failure). Subgroup analysis was conducted between children and teens. In children (4–12 years old), the length of TV watching negatively affected performance in language (effect size = -0.20) and maths (ES= -0.36); in teens (12–18 years old), longer TV duration affected language (ES= -0.18) and maths (ES= -0.21). Playing video games also negatively impacted teens’ scores (ES= -0.16), but did not affect the scores of younger children (ES=+0.04).

The authors suggest that these findings offer evidence that decreasing TV and video game time might be an effective strategy in improving academic achievement in children and teens.

Source: Association between screen media use and academic performance among children and adolescents: A systematic review and meta-analysis (September 2019), JAMA Pediatrics

Printed vs digital text: A meta-analysis

A meta-analysis in the Journal of Research in Reading has synthesised the findings of studies comparing print and digital text regarding time required to read, reading comprehension and readers’ perceptions of their comprehension. Researcher Virginia Clinton performed a systematic literature review, only including studies using random assignment and that were published between 2008 and 2018, yielding 29 reports of 33 studies for analysis. She found that readers require equal amounts of time to read print and digital text, although screen reading negatively impacted reading comprehension (effect size = -0.25). Readers were more accurately able to judge their comprehension on paper (effect size = +0.20) than on screen.

The negative effect on performance for reading text from screens rather than paper did not vary for readers who were adults or children (under 18). However, the author suggests this finding should be interpreted with caution because there were more studies with adult participants (26) than child participants (7).

Best Evidence in Brief reported on an earlier meta-analysis solely examining reading comprehension, whose results also favoured printed text.

Source: Reading from paper compared to screens: A systematic review and meta‐analysis (May 2019), Journal of Research in Reading, volume 42, issue 2

Is there a gender gap in IT?

Previous studies have revealed gender differences in attitudes towards information technology (IT) literacy, with boys generally considering their IT literacy to be higher than that of girls. A new meta-analysis, published in Educational Research Review, tests whether the same gender differences can be seen in pupils’ actual performance on IT literacy tasks as measured by performance-based assessments.

In total, 46 effect sizes were extracted from 23 studies using a random-effects model. The main findings suggest that:

  • Girls perform better than boys on performance-based IT literacy assessments (ES= +0.13).
  • Gender differences in favour of girls are larger in primary schools (ES= +0.20) than in secondary schools (ES= +0.11).
  • The overall effect size is robust across several analysis conditions.
  • Overall, the gender differences in IT literacy are significant but small.

As these findings seem to contrast those obtained from previous meta-analyses that were based on self-reported IT literacy, researchers Fazilat Siddiq and Ronny Scherer conclude that the IT gender gap may not be as severe as it had been claimed to be.

Source: Is there a gender gap? A meta-analysis of the gender differences in students’ ICT literacy (June 2019), Educational Research Review, Volume 27

Effects of youth mentoring programmes

Mentoring programmes that pair young people with non-parental adults are a popular strategy for early intervention with at-risk youth. To examine the extent to which these types of interventions improve outcomes for young people, Elizabeth B Raposa and colleagues conducted a meta-analysis of outcome studies of one-to-one youth mentoring programmes written in English between 1975 and 2017.

Their analysis included 70 studies with a sample size of 25,286 children and young people (average age = 12 years), and considered five broad outcome categories: school, social, health, cognitive and psychological outcomes.

The findings from their meta-analysis suggest no significant difference in effect sizes across these five types of outcomes. Overall, they found an average effect size of +0.21 across all studies and outcomes, which is consistent with past meta-analyses that have shown overall effect sizes ranging from +0.18 to +0.21.

Programmes that had a larger proportion of young males who were being mentored in the sample, a greater percentage of male mentors, or mentors who worked within the helping profession showed larger effect sizes, as did evaluations that relied on questionnaires and youth self-report.

Source: The effects of youth mentoring programs: A meta-analysis of outcome studies (January 2019), Journal of Youth and Adolescence