What difference does it make?

We regularly quote effect sizes in Best Evidence in Brief as a measure of the impact of an intervention or approach. But what is the impact of a normal school year on children, and how much of that impact is due to the school? A study by Hans Lutyen and colleagues, published in School Effectiveness and School Improvement, attempts to find out.

The study analysed 3,500 pupils from 20 mostly independent (private) English primary schools on four different learning outcomes. These measures, part of the Interactive Computer Adaptive System (InCAS), were reading, general maths, mental arithmetic and developed ability, the last of which measures items such as vocabulary and non-verbal pattern recognition.

Children were measured on these outcomes from Years 1 to 6. Using a regression-discontinuity approach that exploited the discontinuity between the youngest pupils in one year and the oldest pupils in the year below, the researchers were able to identify the overall progress of the children, and the extent to which this was a result of the impact of the school.

The results showed a declining impact of a school year as children got older. The effect size of Year 1 ranged from +1.18 for mental arithmetic to +0.8 for general maths. By Year 6, effect sizes varied from +0.88 for general maths to +0.49 for reading and developed ability.

The effect of schooling itself accounted for an average of between 23.5% and 43.4% of this impact across the four measures. Put another way, the effect size of schooling in Year 1 ranged from +0.55 for reading to +0.31 for developed ability. By Year 6, effect sizes had fallen to between +0.27 for general maths and +0.08 for reading and developed ability.

The researchers suggest that, when setting benchmarks for educational interventions, it is not only important to consider the phase of the educational career, but also the specific measure.

Source: The contribution of schooling to learning gains of pupils in Years 1 to 6 (February 2017), School Effectiveness and School Improvement

Impact of secondary school mindfulness programmes

Catherine Johnson and colleagues carried out a randomised controlled evaluation of a secondary school mindfulness programme (called “.b mindfulness” for “Stop, Breathe and Be!”) to measure impact on self-reported measures of anxiety, depression, weight/shape concerns, well-being and mindfulness.

Five hundred and fifty-five pupils in four secondary schools in South Australia participated (mean age = 13.44 years). Pupils were assigned using a cluster (class-based) randomised controlled design to one of three conditions: the nine-week mindfulness curriculum, the nine-week mindfulness curriculum with parental involvement, or a control (business-as-usual) curriculum.

The evaluation found no differences between the mindfulness groups with or without parental involvement and the control group at post-intervention or at the six- and twelve-month follow-up. The researchers conclude that further research is required to identify the optimal age, content, and length of programmes delivering mindfulness to teenagers.

Source: A randomized controlled evaluation of a secondary school mindfulness program for early adolescents: Do we have the recipe right yet? (September 2017), Behaviour Research and Therapy, Vol 99

Children with reading difficulties may have undiagnosed hearing problems

A report published by the Nuffield Foundation, which compared a group of children with dyslexia to children with a history of repeated ear infections (otitis media with effusion, OME) to see if there were any similarities in their phonological and literacy difficulties, has found that their difficulties may be due in part to undiagnosed hearing problems.

Julia Carroll and Helen Breadmore compared a group of 36 children with dyslexia to 29 children with OME, and also to control groups of typically developing children of the same age and groups of younger children at the same reading level. This made a total sample size of 195 children, ages 8 to 10, from 20 schools in the UK. All of the children completed a series of tests to establish their reading and writing skills and also their phonological skills (ability to manipulate speech sounds) and morphological skills (knowledge of grammatical word structure). Eighteen months later, the children’s reading, spelling and phonological awareness was re-tested, and a hearing screening conducted.

The results showed that the children with dyslexia had different patterns of literacy difficulties than children with OME, although there were some overlaps. The children with dyslexia showed difficulties with both phonological and morphological skills, whereas children with OME had difficulties only on phonological tasks. Both dyslexic children and children with OME had lower levels of reading than the age-matched control children.

The results from the hearing screening eighteen months later found that 9 of the 36 children with dyslexia had mild or moderate hearing impairments, of which their parents and teachers were unaware. The researchers suggest, therefore, that children with reading difficulties should be screened for hearing problems so that they are able to receive more structured support that could help them improve their literacy skills.

Source: Morphological processing in children with phonological difficulties (October 2017), Coventry University and University of Warwick Briefing Paper for The Nuffield Foundation

The positive effects of vocational schools

More than 90% of high schools in the US offer a vocational training option for pupils, and more than one in five pupils participate in these career and technical education (CTE) programmes. Traditionally in CTE, career-training courses are offered during each school day, and non-career classes contain a mix of CTE and non-CTE pupils. Massachusetts has taken CTE a step further by offering 32 regional vocational and technical high schools (RVTS), where all pupils participate in CTE, alternating one week of schooling with one week of career training.

Shaun Dougherty of the University of Connecticut recently studied the effects of attending an RVTS. In this study, pupils had to apply to attend, with admission based on middle school (Key Stage 3) record of attendance, standardised test scores in maths and disciplinary record. Using state data and admissions applications, Dougherty compared 4,000 ninth grade pupils (Year 10) in three RVTS schools, 2,000 of whom were just above and below the cutoff points for admission, inferring that RVTS participation would be the main variable affecting their performance. Following these ninth grade pupils through to twelfth grade (Year 13), he found that, as compared to pupils who barely missed the cutoff, pupils who had participated in RVTS:

  • Were 7-10 percentage points more likely to graduate from high school
  • Were more likely to earn industry-recognised credentials while in high school
  • Were likely to receive comparable state test scores that qualified them for graduation

Given that state test scores in core subjects were similar for the pupils who did and didn’t participate in the RVTS, Dougherty concluded that RVTS schools positively influence graduation rates without sacrificing knowledge in core subjects.

Unlike traditional CTE programmes, where pupils may study one course one year and switch the next year, RVTS pupils stay within the same course of study from tenth to twelfth grade (Years 11–13), often with the same teachers and peers. They offer a larger selection of courses, and classes are taught within the same building, allowing vocational-training teachers to confer with traditional-course teachers and carry over career training into the traditional realms of maths, English and social studies.

Source: The effect of career and technical education on human capital accumulation: causal evidence from Massachusetts (October 2017), Education Finance and Policy doi: 10.1162/EDFP_a_00224

What does the evidence say about technology use?

New educational technology programmes are being released faster than researchers can evaluate them. The National Bureau of Economic Research in the US has written a working paper, Education Technology: An Evidence-Based Review, which discusses the evidence to date on the use of technology in the classroom, with the goal of finding decision-relevant patterns.

Maya Escueta and colleagues compiled publicly available quantitative research that used either randomised controlled trials or regression discontinuity designs (where pupils qualify for inclusion in a programme based on a cut-off score at pre-test). All studies had to examine the effects of an ed-tech intervention on any education-related outcome. Therefore, the paper included not only the areas of technology access, computer-assisted learning and online courses, but also the less-often-studied technology-based behavioural interventions.

Authors found that:

  • Access to technology may or may not improve academic achievement at the K-12 level (Years 1–13), but does have a positive impact on the academic achievement of higher education students (ES=+0.14).
  • Computer-assisted learning, when equipped with personalisation features, was an effective strategy, especially in maths.
  • Behavioural intervention software, such as text-message reminders or e-messages instructing parents how to practise reading with their children, showed positive effects at all levels of education, and was also a cost-effective approach. Four main uses for behavioural intervention software emerged: encouraging parental involvement in early learning activities, communication between the school and parents, successfully transitioning into and through higher education, and creating mindset interventions. Research is recommended to determine the areas where behavioural intervention software is most impactful.
  • Online learning courses had the least amount of research to examine and showed the least promise of the four areas. However, when online courses were accompanied by in-person teaching, the effect sizes increased to scores comparable to fully in-person courses.

 Source: Education technology: an evidence-based review (August 2017), National Bureau of Economic Research (NBER) Working Paper No. 23744

Evaluation of a parent-delivered early language enrichment programme

A study, published in The Journal of Child Psychology and Psychiatry, evaluates the effectiveness of a parent-delivered language programme on pre-school children’s language and emerging reading skills.

Kelly Burgoyne and colleagues conducted a randomised controlled trial with 208 pre-school children (mean age 3 years, 1 month) and their parents living in socially diverse areas of the UK. Children and parents received either an oral language programme or an active control programme targeting motor skills. Parents delivered the 20-minute sessions to their child at home every day over 30 weeks. Children were assessed at pre-test, post-test, and 6 months after post-test on measures of language and motor skills. Early literacy skills (letter-sound knowledge, phoneme awareness and regular and irregular word reading) were assessed at 6 months after post-test only, as children were non-readers at pre- and post-test.

Children who received the language programme made larger gains in language skills (effect size = +0.21) and narrative skills (effect size = +0.36) at post-test than those children who received the active control programme, and these results were maintained six months later. Improvements were also seen in letter-sound knowledge (effect size = +0.42) and regular word reading (effect size = +0.35). No evidence was found that the control programme improved motor skills.

Source: Evaluation of a parent-delivered early language enrichment programme: evidence from a randomised controlled trial (September 2017), Journal of Child Psychology and Psychiatry. doi:10.1111/jcpp.12819