Nudging proves difficult to scale up

Small-scale trials of “nudge” approaches, where, for example, students are encouraged via a series of text messages to apply for financial support for college, have shown positive results. These trials have involved a few thousand students, but could the approach be scaled up to state or national level?

A working paper from the National Bureau of Economic Research reports the results of two large randomised controlled trials that collectively reached over 800,000 students in the US. Kelli A. Bird and colleagues tested the impact of a national and state-level campaign to encourage students (average age 18.6 years) to apply for financial aid for college, with multiple treatment arms to investigate different potential mechanisms and approaches.

The trials found no impacts on financial aid receipt or college enrollment overall or for any student subgroups. There was no evidence that different approaches to message framing, delivery, or timing, or offers of one-on-one advice, affected the efficacy of any of the campaigns.

The researchers suggest three reasons why the scaled-up approach may not have been effective:

  • Most previous studies involved a local partner with closer connections to and knowledge of the students. Local partners may know something important about their students and students may react differently to messages from organisations in their communities.
  • A global scale-up results in messaging content that is more generic and less personalised to students.
  • The students in this study may have had better information about the financial support available to them than previous cohorts, so the intervention made less impact.

Source: Nudging at Scale: Experimental Evidence from FAFSA Completion Campaigns (August 2019) NBER Working Paper No. 26158

Do EAL pupils need an extra year to learn English?

Since 2002, all third grade (Year 4) pupils in Florida are required to obtain specific state-wide reading test scores in order to progress to the fourth grade (Year 5). A new NBER working paper considers whether this third grade retention policy, which includes additional teaching and support in reading, might be particularly beneficial for pupils with English as an additional language (EAL).

David N Figlio and Umut Özek used longitudinal data for all pupils between grades three and ten (Years 4 to 11) from 12 US school districts in Florida in order to examine the short-, medium- and long-term effects of repeating the third grade on EAL pupils’ English skills, as measured by their reading test scores, the length of time needed for them to reach required levels of English proficiency, and their course choice in middle and high school.

The results find that repeating the third grade (Year 4) can help to improve the English skills of EAL pupils, and that the benefits are even greater for EAL pupils born outside of the US, pupils whose first language is Spanish, and pupils in lower-poverty elementary schools.

Specifically, they suggest that EAL pupils who repeat the third grade:

  • do better on reading test scores in elementary and middle school
  • reach the required levels of English proficiency in half the time
  • are less likely to take a remedial English course in middle school
  • are more likely to take an advanced course in maths and science in middle school
  • are more likely to take college credit-bearing courses in high school.

Source: An extra year to learn English? Early grade retention and the human capital development of English learners (January 2019), NBER Working Paper No. 25472, National Bureau of Economics Research

Can teaching be simplified?

Technology that simplifies teaching by providing teachers with “off-the-shelf” lessons may increase pupil achievement, particularly if the teachers are supported in using them, according to a working paper published by the National Bureau of Economic Research (NBER).

The study, conducted by Kirabo Jackson and Alexey Makarin, provided middle school maths teachers with online lessons from the Mathalicious curriculum – an inquiry-based maths curriculum for grades 6 to 12 (Years 7 to 13) grounded in real-world topics and situations. Maths teachers from 170 schools across Virginia, US, took part and were randomly assigned to one of three conditions: full treatment (online access to Mathalicious lessons along with supports to promote their use); lesson-only (online access to Mathalicious lessons only); or control (business-as-usual).

While positive effects on pupil achievement in maths were seen in both the full treatment and lesson-only conditions, results were only significant for the full-treatment group. Providing teachers with online access to the lessons along with supports to promote their use increased pupil maths achievement by an effect size of +0.09 (p<.05). Test scores for pupils in the lesson-only group were non-significantly higher than those of the control group (effect size = +0.04).

Source: Can Online Off-The-Shelf Lessons Improve Student Outcomes? Evidence from A Field Experiment (January 2017), National Bureau of Economic Research (NBER), NBER Working Paper No. 22398

What does the evidence say about technology use?

New educational technology programmes are being released faster than researchers can evaluate them. The National Bureau of Economic Research in the US has written a working paper, Education Technology: An Evidence-Based Review, which discusses the evidence to date on the use of technology in the classroom, with the goal of finding decision-relevant patterns.

Maya Escueta and colleagues compiled publicly available quantitative research that used either randomised controlled trials or regression discontinuity designs (where pupils qualify for inclusion in a programme based on a cut-off score at pre-test). All studies had to examine the effects of an ed-tech intervention on any education-related outcome. Therefore, the paper included not only the areas of technology access, computer-assisted learning and online courses, but also the less-often-studied technology-based behavioural interventions.

Authors found that:

  • Access to technology may or may not improve academic achievement at the K-12 level (Years 1–13), but does have a positive impact on the academic achievement of higher education students (ES=+0.14).
  • Computer-assisted learning, when equipped with personalisation features, was an effective strategy, especially in maths.
  • Behavioural intervention software, such as text-message reminders or e-messages instructing parents how to practise reading with their children, showed positive effects at all levels of education, and was also a cost-effective approach. Four main uses for behavioural intervention software emerged: encouraging parental involvement in early learning activities, communication between the school and parents, successfully transitioning into and through higher education, and creating mindset interventions. Research is recommended to determine the areas where behavioural intervention software is most impactful.
  • Online learning courses had the least amount of research to examine and showed the least promise of the four areas. However, when online courses were accompanied by in-person teaching, the effect sizes increased to scores comparable to fully in-person courses.

 Source: Education technology: an evidence-based review (August 2017), National Bureau of Economic Research (NBER) Working Paper No. 23744

The effects of head teacher training on pupil achievement

An NBER Working Paper examines the impact of implementing management training for head teachers on pupil achievement. The management training focused on lesson planning, data-driven teaching and teacher observation and coaching (approximately 300 hours over two years). Using a school-level randomised experiment, 58 schools in Houston, Texas, were randomised to receive either the training intervention or to serve as a business-as-usual control group.

The study found that offering management training to head teachers led to increased test scores across low-stakes tests in a range of subjects in year one (effect size = +0.19). For high-stakes test scores in maths and reading, the effect size was lower (+0.10). However, the training intervention had no impact on high-stakes tests in year two.

The training was most beneficial for head teachers who were less experienced, had better maths skills, had more internal locus of control, had higher levels of “grit” and remained in the school for both years of the study.

The intervention showed most impact on teachers in the schools who were more experienced and more educated. The intervention showed most impact for pupils who were new to the school, white or Hispanic and economically well-off.

Source: Management and student achievement: Evidence from a randomized field experiment (May 2017), NBER Working Paper No. 23437, National Bureau of Economic Research

 

Unhealthy school lunches nibble away academic performance

A National Bureau of Economic Research (NBER) Working Paper looks at the effect of offering healthier school lunches on end-of-year test scores for pupils in California.

Michael L Anderson and colleagues analysed data collected over a five-year period (academic years 2008/2009 to 2012/2013) from around 9,700 schools that reported test scores on California’s Standardized Testing and Reporting exam, a state-wide test given to all pupils in grades 2 to 11 (Years 3 to 13). In order to determine the link between food quality and pupil achievement, they also collected data from the California Department of Education on school districts’ meal vendors for the same time span. Over that five-year period, about 12% of California schools contracted an outside company to provide lunch for at least one school year. The nutritional value of the school lunch menus was analysed by nutritionists at the Nutrition Policy Institute using the Healthy Eating Index.

The results of the study found that in years when schools offered a healthy lunch menu, pupil test scores were on average higher (effect size +0.03 to +0.04). In addition, test scores for pupils who qualified for reduced-price or free school lunches, (and therefore more likely to eat the healthy lunches), increased by about 40% in comparison to pupils who didn’t receive free school lunches.  The positive effect of healthy lunches on academic achievement persisted for the duration of a long-term contract.

Source: School lunch quality and academic performance (March 2017), NBER Working Paper No. 23218, National Bureau of Economic Research (NBER)