The reader-friendliness of effect sizes

Effect size has long been the standard measurement used in educational research. This commonality allows for comparison across studies, between programmes, and so on. It’s a tricky statistic, though, because its implications are not necessarily understood by the typical consumer of research. For example, saying that a programme has an effect size of +0.13 is likely to be less meaningful to the layperson than saying that a programme yielded a gain of one month’s learning.

In an effort to make effect sizes more reader-friendly, writers sometimes translate effect sizes into terms easier to understand, most often into units of time, such as days/years of learning. Yet research statisticians warn that what is gained in understandability may be lost in accuracy.

In an article appearing on Educational Researcher’s Online First site, RAND’s Matthew Baird and John Pane compared the “years of time” translation to three other reader-friendly measures, rating which were most and least accurate reflections of effect sizes: to benchmarking against similar groups in other studies, to percentile growth, and to determining the probability of meeting a certain threshold. Specifically, Baird and Paine used data from a 2017 evaluation of personalised learning that reported detailed assessment procedures, data structure and methods of analysis, applying this information to calculate whether each reader-friendly term incorporated six properties they deemed necessary to promote accuracy between the effect size reported and the more reader-friendly terms in which it was stated.

Results showed that the units of time translation was in fact the least accurate, while the percentile gains option yielded the best results.

Source: Translating standardized effects of education programs into more interpretable metrics (2019), Educational Researcher DOI: 10.3102/0013189X19848729

This study is discussed further in this blogpost by Robert Slavin.

Positive effects of a healthy debate

Johns Hopkins University’s Daniel Shackelford has conducted the first quantitative study examining the effects of participation in an extracurricular debate club during pre-adolescence on pupils’ later academic and engagement outcomes, including entry to selective-entrance high schools.

Dr Shackelford examined a 10-year sample of 2,263 4th to 8th grade pupils (Year 5 to Year 9) participating in Baltimore City’s Baltimore Urban Debate League (BUDL) between the 2004 to 2013 school years, comparing their standardised maths and reading scores, attendance, and entry to selective-entrance high schools to 81,906 peers who did not participate in BUDL. Ninety-one percent of both groups were African American, and 96% of both groups received free and reduced-price lunch.

It is of note that these two groups were not matched at baseline: pupils who became debaters differed from controls prior to their participation in BUDL, with higher standardised test scores and attendance, so no true causal conclusion can be drawn from comparing groups. Yet among the debate pupils themselves, results showed that pre-adolescent debate participation yielded more than a 6% increase in reading scores and a 4% increase in maths scores on standardised testing. While debate inherently involves reading and might be accountable for increased reading achievement, Dr Shackelford observes that debaters were 10% more likely not to be chronically absent than non-debaters, and this increased engagement in school may have yielded the improvements in maths scores. BUDL pupils were also more likely to attend a selective high school (+0.12) or selective career tech high school (+0.01) than to attend a traditional high school.

Source: The BUDL effect: Examining academic achievement and engagement outcomes of preadolescent Baltimore Urban Debate League participants (February 2019), Educational Researcher. 

Is free and reduced-price lunch a valid measure of educational disadvantage?

Almost 60% of American pupils receive lunch through the National School Lunch Program (NSLP), which provides free- and reduced-price lunches (FRPL) to pupils in households who demonstrate economic disadvantage. Income information is based on parent report of household income in the month preceding application to the programme. Because NSLP enrolment has been correlated with lower pupil achievement and is an indicator of how disadvantaged a school’s population is, this classification plays an important role in how funds are allocated to schools and how schools are classified in educational research.

Researchers at the University of North Carolina at Chapel Hill, the US Census Bureau, the University of California at Irvine, and NORC at the University of Chicago recently examined how accurately FRPL enrolment measures actual income and educational disadvantage by comparing IRS income records with pupils’ lunch enrolment and achievement records. Specifically, researchers examined the records of all eighth grade (Year 9) pupils in a California public school district from 2008-2014 (n=14,000) and in Oregon public schools from 2004-2014 (n=363,000), examining the relationship between FRPL enrolment, IRS income records, and eighth grade English language achievement scores on the California Achievement Test and the Oregon Assessment of Knowledge and Skills.

Results showed that school lunch programme enrolment explains the relationship between economic disadvantage and pupil achievement better than IRS-reported annual income. Compared to their non-NSLP peers, California free-lunch pupils scored 0.40 standard deviations (SD) lower on the eighth grade English language test, and the reduced-lunch pupils scored 0.20 lower than those not enrolled. In Oregon, the FRPL pupils scored 0.36 SD lower than those not enrolled. In comparison, when using IRS-reported household income to explain English language achievement, economically disadvantaged pupils scored approximately 0.15 SD lower than pupils appearing to be ineligible for FRPL in one California district and 0.26 SD lower across Oregon public schools. In other words, FRPL enrolment accounted for 16% of the variance in English language scores, whereas IRS data only accounted for 13% of the variance. This indicates that FRLP appears to capture aspects of disadvantage that IRS data do not.

Source: Is free and reduced-price lunch a valid measure of educational disadvantage? (December 2018), Educational Researcher, Volume: 47 issue: 9

The benefits of theatre field trips

Field trips to the theatre provide a number of educational benefits to pupils, according to research published in Educational Researcher. Jay P Greene and colleagues found that giving pupils the opportunity to take part in a field trip to see a live theatre performance produced an increase in tolerance as well as a greater understanding of the plot and vocabulary of those plays.

Schools in Arkansas in the US were assigned by lottery to receive free tickets to attend one of five live theatre performances over a two-year period. Grade 9 (Year 10) classes from participating schools were then randomly assigned to take part in theatre field trip or to serve as a control group and not take part in the field trips. In addition, for two of the five experiments, a second treatment group was added in which pupils were randomly assigned to watch a film version of the theatre play. The average age of pupils in the treatment and control groups was 14 years old.

The impact to pupils of the theatre field trip was measured on five outcomes: tolerance, social perspective taking (the ability to understand others’ feelings and perspectives), content knowledge, theatre consumption and theatre participation. Pupils in the theatre field trip treatment groups scored higher for levels of tolerance and social perspective taking (+0.14 and 0.16 of a standard deviation higher than the control group). Pupils’ content knowledge of the plot and vocabulary in the plays was also greater (by 0.15 of a standard deviation) than pupils in the control group.

However, watching a film did not produce benefits, and as the film-viewing group also left school for a field trip, the results suggest that the educational benefits to pupils come from the experience of watching live theatre, and not simply from leaving school for a field trip. Results also indicate that theatre field trips may encourage pupils to visit the theatre more often.

Source: The play’s the thing: experimentally examining the social and cognitive effects of school field trips to live theater performances (March 2018), Educational Researcher, Vol 47, Issue 4, pp. 246 – 254

Decades of evidence supports early childhood education

A recent meta-analysis of almost 60 years’ worth of high-quality early childhood education (ECE) studies in the US found that participating in ECE programmes significantly reduced special education placement and grade retention (pupils having to repeat a year), and lead to increased graduation rates from secondary school.

Dana Charles McCoy and colleagues examined data from studies spanning 1960-2016. All had to meet strict inclusion criteria and address ECE’s effects on special education placement, grade retention, or dropout rates, yielding 22 studies. Seven were randomised controlled studies, four were quasi-experimental, and eleven used non-randomised assignment and compared groups who were equivalent at baseline.

Results showed statistically significant effects of ECE. Compared to pupils who did not attend ECE, participants were 8.1% less likely to be placed in special education, 8.3% less likely to be held back a year and 11.4% more likely to graduate from secondary school.

Source: Impacts of early childhood education on medium- and long-term educational outcomes (November 2017), Educational Researcher Volume 46, issue 8

Trends in early literacy

The transition from kindergarten to first grade (Year 1 to Year 2 in the UK) is considered to be a critical period for children’s academic and social development. Expectation about children’s early literacy learning has risen over time, but has their achievement – and how?

Jerome V D’Agostino and Emily Rodgers analysed achievement data obtained from a US database for Reading Recovery (a literacy intervention for first grade pupils) for more than 364,000 children entering first grade in the same schools. From this data they created a literacy profile for pupils at entry to the first grade over a 12-year period, beginning in the 2002–03 school year.

Their research, published in Educational Researcher, found that overall, reading for all pupils in the first grade improved measurably between 2002 and 2013. Literacy scores on entry increased over time. The effect size change in achievement gaps narrowed (-0.10) on basic skills like letter identification, but widened on advanced skills like text reading level (+0.08) over 12 years.

Source: Literacy achievement trends at entry to first grade (March 2017), Educational Researcher, Vol 46, Issue 2.