Do teacher pay incentives improve pupil test scores?

A meta-analysis published in the American Educational Research Journal looks at the association between teacher pay incentives and pupils’ test scores, and suggests that teacher pay incentives have the potential to improve pupil test scores in some contexts.

Lam D Pham and colleagues analysed effect sizes across 37 studies, 26 of which were conducted in the US. To be included in the meta-analysis, studies had to include a sample comprising teachers and pupils in K-12 education (Year 1 to Year 13) located in a school district or area that had a teacher pay incentive programme. Studies also had to use a randomised controlled trial with a business-as-usual comparison group, and report on pupil outcomes on standardised tests.

Overall, among the US-based studies, the effect of teacher pay incentives on pupil test scores was positive (effect size +0.04), however, this varied across subjects and settings. The average effect size of pay incentives on pupils’ maths test scores (+0.05) was larger than the effect on English test scores (+0.03). Pay incentives for elementary (primary) school teachers were associated with larger effects (+0.10) than middle school teachers (+0.01). In addition, larger pay incentives, and pay incentives that are based on multiple measures of teacher effectiveness, were associated with larger effect sizes.

Source: Teacher merit pay: A meta-analysis (February 2020), American Educational Research Journal

Digital vs. paper reading

A study published in the American Educational Research Journal compares reading processes and outcomes for pupils when reading a text from paper with the same text delivered on a touchscreen laptop.

Amanda P Goodwin and colleagues conducted the study with 371 pupils in grades 5–8 (Years 6–9) from three schools in an urban district in the southeastern US. Pupils were randomly assigned to two conditions: Condition A read the first section of a text on paper, and the second half digitally, whereas pupils in Condition B read the first part digitally and the second part on paper. The content in both conditions was identical. When reading on paper, pupils had access to highlighters, pens and sticky notes; when reading digitally, they had access to digital highlighters, annotating and dictionaries.

Results suggest that pupils highlight and annotate more when reading on paper vs. digital text. Also, reading on paper vs. digitally was slightly supportive of reading comprehension for the longer sections of text, although effect sizes were very small (odds ratio of 1.077).

Source: Digital versus paper reading processes and links to comprehension for middle school students (December 209), American Educational Research Journal

Targeting EALs with science

A recent large-scale randomised controlled trial, published in the American Educational Research Journal, has examined the impact of a science curriculum with a focus on pupils with English as an Additional Language (EALs).

The study was implemented in 66 schools (33 treatment and 33 control) across three school districts in one south-eastern US state. During the 2012–2013 school year, the project involved 258 teachers (123 treatment and 135 control) and a total of 6,673 students. The trial evaluated P-SELL, a science curricular and professional development intervention for fifth-grade students with a focus on EALs.

The P-SELL curriculum’s approach aligns with state science standards and high-stakes science assessments administered at fifth grade. It is based on an inquiry-oriented approach and addresses the learning needs of EALs by providing guidance and scaffolding for English language development. Teachers are supported with a teacher’s guide and professional development workshops. The workshops incorporated critical features of effective professional development: content focus, active learning, coherence, sufficient duration, and collective participation.

The study used both the high-stakes state science assessment as an outcome measure and a researcher-developed science assessment that was administered at the beginning and end of the year and allowed for a pre-measure of science achievement. The study examined the effect of the intervention on science achievement for all students and for students of varying levels of English proficiency (EAL, recently reclassified EAL, former EAL, and non-EAL).

The results found significant and meaningfully sized average intervention effects on the researcher-developed science assessment scores (effect size = +0.25) and the state science assessment scale scores (+0.15). The P-SELL intervention had significant and meaningfully sized effects for EALs (+0.35) on the researcher-developed assessment. The intervention effects were positive but not statistically significant for EALs (+0.12) on the state science assessment, although other subcategories (non-EALs and former EALs) were positive and significant. This is the first year of a three year study, and future years will provide information on the long-term impact of the teachers’ professional development.

Source: Impact of a Large-Scale Science Intervention Focused on English Language Learners (2016), American Educational Research Journal.

The effects of data-based decision making

The use of data to inform educational decisions is becoming increasingly popular worldwide. An article in the most recent American Educational Research Journal describes the effect of a two-year schoolwide data-based decision-making intervention, called Focus, on student achievement.

Focus trains schoolwide teams of teachers and administrators to use data to guide their teaching using a protocol developed at the University of Twente in the Netherlands. Staff receive seven training meetings in year 1 and four training meetings in year 2, and are provided with documents and planning aids to help them track student data and progress.

Fifty-three primary schools (1,193 staff members) in the Netherlands used Focus to apply student achievement data to guide instruction during a two-year study. All schools (n=53) were trained to use data-based decision-making in mathematics during years 1 and 2, and had the option to also use it in spelling lessons in year 2 (n=38). Student achievement data from standardised maths tests given twice a year were collected for children aged 6-12 for two years before implementing Focus and then for two years during the intervention. Results showed benefits of the intervention equal to an extra month of schooling and were most statistically significant for students from low socio-economic backgrounds.

Source: Assessing the Effects of a School-Wide Data-Based Decision-Making Intervention on Student Achievement Growth in Primary Schools (2016), American Educational Research Journal.

Second success launches Number Rockets

A recent article in the American Educational Research Journal describes a scaled-up replication study of a randomised controlled trial evaluating the effects of Number Rockets, a small-group intervention for children aged 6/7 at risk of mathematical difficulties.

A replication study repeats the procedures of an earlier study to see if the effects of the intervention are consistent. If similarly positive results are reproduced, this lends credence to that programme’s efficacy. Replication rates for education research are often low.

In this case, the original study found a positive impact for pupils using Number Rockets in a single US state district under ideal conditions. It was conducted in ten schools and involved 139 pupils (70 in the experimental group, 69 in control) identified as at-risk because of their performance in the lowest fifth on a screening assessment. The current, scaled-up study involved 76 schools across four districts in four US states, with 994 pupils (615 experimental, 379 control) identified by a screening score as being in the sample’s lowest third.

In an effort to more closely replicate real-world implementation, tutors were provided with less support and monitoring than in the original study. The experimental group received the Number Rockets curriculum in groups of 2-3 pupils three or more times a week for 17 weeks in addition to their regular maths curriculum, while the control pupils continued with their current curriculum. The Number Rockets curriculum is scripted, and focuses on concepts and operations with whole numbers, such as addition/subtraction, equality, comparing quantities, number placement on a number line, and also includes 10 minutes of fact practice.

The Number Rockets group outperformed the control group on the TEMA-3 standardised maths test, replicating the findings of the original study. Coincidentally, effect sizes for the experimental group were +0.34 for both the original and current studies.

Source: Intervention for First Graders With Limited Number Knowledge: Large-Scale Replication of a Randomized Controlled Trial (2015), American Educational Research Journal, 52(3).

Teaching academic vocabulary knowledge

Children whose first language is not English have to cope with learning English and academic content in English at the same time. A new study has shown the benefit of specifically teaching academic vocabulary.

The randomised controlled trial (RCT) was carried out in 14 middle schools in California, where 50 teachers were assigned to treatment or control conditions. A total of 2,082 sixth-grade (Year 7) students participated, 71% of whom spoke another language (mostly Spanish) at home. They followed Academic Language Instructions for All Students (ALIAS), a 20-week programme teaching academic vocabulary – words that are not subject-specific but often appear in sixth-grade textbooks (such as expanse, integrated, generate, according to). The programme was supported with teaching materials and professional development.

Students improved their vocabulary knowledge, morphological awareness skills, and comprehension of expository texts that used the academic language that was taught. They also improved their performance on a standardised measure of written language skills (effect size=+0.19). The effects were generally larger for children whose home language was not English and for those who started the intervention with underdeveloped vocabulary knowledge.

Source: Effects of Academic Vocabulary Instruction for Linguistically Diverse Adolescents (2014), American Educational Research Journal 51(6)