New findings about Investing in Innovation (i3) evaluations

A new report from the National Center for Education Evaluation and Regional Assistance (NCEE) in the US summarises the findings of the first 67 Investing in Innovation (i3) fund evaluations. The i3 fund is a tiered-evidence programme that aligns the amount of funding awarded with the strength of the prior evidence supporting the proposed intervention. The report stated that:

  • Twelve of the i3 evaluations found a statistically significant positive impact on at least one pupil academic outcome.
  • Forty of the i3 evaluations met all of the evaluation quality goals set by i3. In addition to consistency with What Works Clearinghouse (WWC) evidence standards, these goals included: independence, high-quality implementation measurement and including a sample that adequately represents those served under the grant.

Source: The Investing in Innovation Fund:  summary of 67 evaluations – final report (June 2018), National Center for Education Evaluation and Regional Assistance (NCEE), US Department of Education

Urban violence and pupil performance

A new study published in Sociology of Education finds that children who attend school with many pupils from violent neighbourhoods can earn significantly lower test scores than peers with classmates from safer areas. The lead author of the study was sociologist Julia Burdick-Will from Johns Hopkins University.

Burdick-Will studied pupils who attended Chicago Public Schools from 2002 to 2010, analysing administrative data from the school system, crime statistics from the Chicago Police Department and school surveys from the University of Chicago Consortium on Chicago School Research. She looked at five cohorts of pupils who were freshmen (Year 10) between the autumn of 2002 and 2006, and followed each pupil for up to four years. Results indicated that in schools where more pupils have a high exposure to violence, their classmates score as much as 10 percent lower on annual standardised maths and reading tests.

According to the report, the study shows that when pupils experience higher levels of neighbourhood violence, the whole school reports feeling less safe, having more disciplinary problems and feeling less trust in their teachers.

Source: Neighborhood violence, peer effects, and academic achievement in Chicago (June 2018), Sociology of Education DOI: 10.1177/0038040718779063

 

Free glasses improve reading achievement

In the first US school-based study to link reading achievement with the provision of free glasses, Robert Slavin, director of the Center for Research and Reform in Education and colleagues at Johns Hopkins’ Wilmer Eye Institute, examined the effects on reading performance of providing free glasses to disadvantaged pupils.

A total of 317 second and third grade pupils (Years 3 and 4) in 12 disadvantaged Baltimore City schools had their vision tested in the autumn and winter of 2014-2015. They also completed reading pre- and post-tests from the Woodcock Language Proficiency Battery at those times. Sixty-nine percent (n=182) of the pupils’ vision tests showed they needed glasses. Pupils who needed glasses were given two pairs, one for home and one for school. Lost or broken glasses were replaced, and school staff were enlisted to help children remember to wear their glasses. Results showed that the reading scores for the children provided with glasses improved more than those for pupils who did not need glasses (effect size=+0.16).

The study points to a new strategy for improving reading performance in high-poverty schools.

Source: In plain sight: reading outcomes of providing eyeglasses to disadvantaged children (May 2018) Journal of Education for Students Placed at Risk (JESPAR) DOI: 10.1080/10824669.2018.1477602

Do physically active lessons improve pupil engagement?

A study published in Health Education and Behavior looks at the effects of introducing physically active lessons into primary school classes. Emma Norris and colleagues used the Virtual Traveller (VT) intervention to evaluate whether physically active lessons had any effect on pupil engagement, physical activity and on-task behaviour.

Virtual Traveller is a programme of pre-prepared physically active lesson sessions delivered using classroom interactive whiteboards during regular lessons. A total of 219 children aged 8- to 9-years-old from 10 schools in Greater London took part in the cluster-randomised controlled trial. Children in the intervention schools received 10-minute VT sessions three times a week, for six weeks, during maths and English lessons. To assess the effectiveness of VT, pupils’ physical activity levels, on-task behaviour and engagement were measured at baseline (T0), at weeks two (T1) and four  (T2) of the six-week intervention, and at one week (T3) and three months (T4) post-intervention.

Pupils in the intervention group showed more on-task behaviour than those in the control at T1 and T2, but this was not maintained post-intervention. No difference in pupil engagement between the control and intervention groups was observed at any time point. VT was found to increase physical activity, but only during lesson time.

Source: Physically active lessons improve lesson activity and on-task behavior: a cluster-randomized controlled trial of the “Virtual Traveller” intervention (March 2018), Health Education & Behavior DOI: 10.1177/1090198118762106

Is the pen mightier than the mouse?

Ben Backes and James Cowan from CALDER have published a working paper on the differences between computer- and paper-based tests.

In 2015, Massachusetts introduced the new PARCC assessment. School districts could choose whether to use the computer or paper versions of the test, and in 2015 and 2016, districts were divided fairly evenly between the two. The authors use this division to compare results for pupils in Grades 3–8 (Years 4–9).

Pupils who took the online version of PARCC scored about 0.10 standard deviations lower in maths and about 0.25 standard deviations lower in English than pupils taking the paper version of the test. When pupils took computer tests again the following year, these differences reduced by about a third for maths and by half for English.

The study also looked at whether the change to computer tests affected some pupils disproportionately. There were no differences for maths, but for English there was more of an effect on pupils at the bottom of the achievement distribution, pupils with English as an additional language and special education pupils.

The authors point out that these differences not only have consequences for individual pupils, but for other decisions based on the data, including teacher and school performance measures and the analysis of schoolwide programmes.

Source: Is the pen mightier than the keyboard? The effect of online testing on measured student achievement (April 2018), National Center for Analysis of Longitudinal Data in Education Research, Working Paper 190

Study shows benefits of Healthy Harlem programme

Mathematica Policy Research posted a new research brief that summarises findings from a study of Healthy Harlem, an after-school programme aimed at promoting healthy lifestyles. The study, by James Mabli, Martha Bleeker and Mary Kay Fox, showed that participation in the Healthy Harlem programme led to increased physical activity and improved weight status for overweight and obese pupils.

Key components of Healthy Harlem include physical activity, healthy snacks, nutrition education lessons and parent workshops. To assess Healthy Harlem’s effectiveness, the authors monitored pupils at 21 after-school sites during an initial baseline year and then measured programme impacts after two and three years of participation. They collected data through a pupil survey, a fitness test and direct measurements of height and weight. Key findings were as follows:

  • A 5.5 percent decrease in mean BMI z-scores after two years of participation and a 9.0 percent decrease after three years of participation. According to the report, a BMI z-score reflects the number of standard deviations a pupil’s BMI is from the mean BMI for a reference population.
  • A decrease of 12.2 percentage points in the percentage of pupils who were overweight or obese after two years, and a decrease of 18.4 percentage points after three years.
  • An increase in the percentage of pupils considered to be within the Harlem Fitness Zone, a measure of fitness based on a pupil’s ability to complete a minimum number of laps, defined for age-and-gender subgroups.

Source: The impact of Healthy Harlem on the body mass index and weight status of adolescents after two and three Years (March 2018), Mathematica Policy Research