A meta-analysis of writing in social studies, science and maths

Is writing about classroom content an effective way to learn? Arizona State University’s Steven Graham and colleagues at the University of Utah recently performed a meta-analysis on the effects of writing about classroom content in social studies, science and maths. Specifically, they examined if writing increased pupil achievement, if the results differed among subjects, and if any relationships existed by year level, activity type, or any other factors.

To be included, studies had to meet quality criteria including true or quasi-experimental research design, reliability of measures, controlling for teacher effects, multiple classes in the experimental and control conditions, experimental and control group pre-test equivalence, and both groups experiencing equal amounts of time learning the same topics.

This search yielded 56 studies in 53 documents meeting criteria for inclusion, involving 6,235 pupils in grades 1-11 (Years 2-12). Pupils in experimental groups wrote about classroom content, while most controls did not write at all. Forty-six percent of the studies assessed the impact of writing on science, 38% on maths and 14% on social studies. Thirty-four percent examined elementary (primary) pupils, and 32% each examined middle and high school (secondary) pupils. The types of writing activities for the experimental groups included writing informational text, such as summarising information or writing a report (34%); journal writing (32%); argumentative writing (13%); and narrative writing, such as creating a word problem in maths lessons (5%). These were coded to determine which, if any, were more effective than others.

Results showed that writing about content increased pupil achievement when compared to equivalent peers in non-writing control groups. Average weighted effect sizes were statistically significant in science (+0.31), social studies (+0.31) and maths (+0.32), as they were when broken down by elementary (+0.29), middle (+0.30) and high school (+0.30) levels. No correlation was found with number of treatment days, type of writing task, or type of assessment.

Source: The effects of writing on learning in science, social studies, and mathematics: A meta-analysis (March 2020), Review of Educational Research

How to make a systematic review’s meta-analysis high quality

Terri Piggott at Loyola University Chicago and Joshua Polanin at American Institutes for Research have published a Methodological guidance paper: High-quality meta-analysis in a systematic review, now appearing on Review of Educational Research’s Online First website.

A meta-analysis synthesises the quantitative findings of many studies on a given topic. The guidance paper outlines the characteristics that make a meta-analysis in a systematic review high quality, discussing unbiased screening and coding procedures, establishing a protocol for carrying out a review, and then discussing in depth the best practices for computing effect sizes and reporting the data.

The authors conclude that “the role of researchers using systematic review and meta-analysis is to produce both high-quality analyses and to interpret those results in ways accessible to a wide audience. A high-quality systematic review and meta-analysis is difficult and time-consuming to produce; it is worth the effort to ensure that the results inform future research and policymaking through clear discussion of the results. Researchers should consider preparing different summaries of their review tailored to their audience of researchers, policymakers, and practitioners.”

Source: Methodological guidance paper: High-quality meta-analysis in a systematic review (September 2019), Review of Educational Research

Computer-supported collaborative learning

Juanjuan Chen and colleagues recently performed a meta-analysis on the effects of computer-supported collaborative learning (CSCL).

Using 425 empirical studies (all of which used a controlled experimental or quasi-experimental design) published between 2000 and 2016, researchers found several main characteristics to examine: the effects of the collaboration itself; the effects of computer use during collaboration; the effects of extra technology-related learning tools used in CSCL, such as videoconferencing and sharing visuals with team partners; and strategies such as role assignment and peer feedback.

Collaborative learning itself positively affected:

  • Knowledge gain (+0.42)
  • Skill acquisition (+0.62)
  • Pupil perceptions of the experience (+0.38)

The use of computers, when combined with collaborative learning, positively affected:

  • Knowledge gain (+0.45)
  • Skill acquisition (+0.53)
  • Pupil perceptions (+0.51)
  • Group task performance (+0.89)
  • Social interaction (+0.57)

Lastly, extra technology-related learning tools during CSCL positively affected knowledge gain (+0.55), as did the use of strategies (+0.38).

Source: The role of collaboration, computer use, learning environments, and supporting strategies in CSCL: A meta-analysis (December 2018), Review of Educational Research, 88(6).

Professional development and early childhood education and care

A meta-analysis published in Review of Educational Research summarises findings from studies that evaluated the effects of in-service training for early childhood teachers on the quality of early childhood education and care (ECEC) and child outcomes. Overall, data from 36 studies with 2,891 teachers was included in the analysis. For studies to qualify, child care quality had to be measured externally with certified raters at the classroom level.

The analysis, carried out by Franziska Egert and colleagues, revealed that at the teacher level, in-service training had a positive effect on the quality of ECEC, with an effect size of +0.68. Furthermore, a subset of nine studies (including 486 teachers and 4,504 children) that provided data on both quality ratings and child development were analysed, and they showed a small effect at the child level (effect size = + 0.14) and a medium effect at the corresponding classroom level (effect size = +0.45).

Source: Impact of In-Service Professional Development Programs for Early Childhood Teachers on Quality Ratings and Child Outcomes: A Meta-Analysis, Review of Educational Research, 88:3 401 – 433.

The effect of teacher coaching on teaching and learning

Matthew A Kraft and colleagues conducted a meta-analysis of the causal evidence on the effect of teacher coaching on teaching and learning. Their paper, published in the Review of Educational Research, reviewed 60 studies on teacher coaching programmes conducted after 2006 that measured the impact of teacher coaching on either teaching (measured using tools such as the Classroom Assessment Scoring System or the Early Language and Literacy Classroom Observation) or pupil academic performance (measured by standardised tests).

Their results found that sustained coaching improves both classroom teaching and pupil achievement, with pooled effect sizes of +0.49 standard deviations for teaching and +0.18 standard deviations for academic achievement.

However, the effectiveness of a teacher coaching programme seems to be determined by the number of participants. When studies were divided into programmes that had fewer than 100 participants and those that had more than 100 participants, the impact on teaching was nearly double for the smaller programmes than for programmes with more than 100 participants. For pupil achievement, the smaller programmes showed an impact of nearly three times that of the larger programmes.

Source:  The effect of teacher coaching on instruction and achievement: A meta-analysis of the causal evidence (February 2018), Review of Educational Research, Vol 88, Issue 4

The impact of professional development in early childhood education

Franziska Egert and colleagues in Germany and Amsterdam have conducted a review of the effects of professional development (PD) for early childhood educators on programme quality and children’s educational outcomes.

Studies were only included if they addressed quality of child care or child development, included early childhood teachers (including preschool, kindergarten and centre-based care), were quantitative, were experimental or quasi-experimental, reported effect sizes or data and addressed children 0–7 years old. This yielded 36 studies of 42 programmes evaluating quality ratings, and nine studies of 10 programmes evaluating both quality ratings and pupil outcomes.

Results showed that professional development improved the external quality ratings (as evaluated using the Classroom Assessment Scoring System, Early Language and Literacy Classroom Observation, Environmental Rating Scales and Individualized Classroom Assessment Scoring System) of early childhood education (effect size=+0.68), with programmes providing 45–60 PD hours having the greatest impact on classroom practice as compared to programmes offering fewer or more hours. This was true regardless of whether teachers held a university degree or not. Further, programmes that solely used coaching were almost three times as effective as other programmes. A second meta-analysis of a subset of studies (n=486 teachers, 4,504 children) showed that improvement in the quality of early childhood education programmes was correlated with improvements in child development (effect size=+0.14) as determined by language and literacy scores, maths scores, social-behavioural ratings, and assessment of cognition, knowledge and school readiness.

Source: Impact of in-service professional development programs for early childhood teachers on quality ratings and child outcomes: a meta-analysis (January 2018), Review of Educational Research, Vol 88, Issue 3