Test results don’t show how effective teachers are

A new study has looked at the link between instructional alignment (how teaching is aligned with standards and assessments), value-added measures of teacher effectiveness, and composite measures of teacher effectiveness using multiple measures.

The issue is important as, in the US and around the world, there is more emphasis on measuring teacher effectiveness and rewarding effective teachers. The study looked at 324 teachers of fourth and eighth grade (Year 5 and Year 9) mathematics and English language arts in five US states. They completed a Survey of Enacted Curriculum to measure their instructional alignment. This was then compared with value-added measures (taken from state assessments and two supplementary assessments) and teacher effectiveness (using Framework for Teaching scores, widely used by states).

The results showed modest evidence of a relationship between instructional alignment and value-added measures, although this disappeared when controlling for pedagogical quality. The one significant relationship they found was that the association between instructional alignment and value-added measures is more positive when pedagogy is high quality. There was no association between instructional alignment and measures of teacher effectiveness.

These results suggest that the tests used for calculating value-added measures are not able to detect differences in the content or quality of classroom teaching.

Source: Instructional Alignment as a Measure of Teaching Quality (2014), Education Evaluation and Policy Analysis, online first, May 2014.

Alleviating parental depression

A new article published in BMC Health Services Research looks at the impact of the Incredible Years (IY) parenting programmes on parental depression and service use. IY aims to reduce conduct disorder in children and depression in their parents, two issues that are often apparent in the same family. Recent trials in the UK and Ireland have focused on the effects of the programme on children, but less was known about the effects on parents.

The authors explored parental depression and service use (and the associated costs) after attending a 12-week, group-based IY Basic Parenting programme. They conducted a secondary analysis of data gathered in a randomised controlled trial (RCT) of the programme. The original RCT sample consisted of 153 (Intervention N=104, Control N=49) parents of children aged 3–4 years old (at baseline) living in 11 disadvantaged Sure Start areas in north and mid Wales.

Depression scores were compared over time for the intervention and control groups. The authors found that parental depression decreased at six months for both the intervention and control groups; however, this decrease was only significant for the intervention group. The differences between intervention and control groups were not significant.

The article also looked at service use costs (primary services such as GPs and health visitors; social services; and hospital services). The parents in the trial accessed a high number of services, particularly in primary health. Total mean costs of service use for the intervention group increased at six and eighteen months post-baseline; however, costs decreased at twelve months post-baseline. Parents who scored above the “clinical level” of self-reported depression in both the intervention and control groups accessed more health and social services than those who were below the clinical level for concern.

Source: Parental Depression and Child Conduct Problems: Evaluation of Parental Service Use and Associated Costs After Attending the Incredible Years Basic Parenting Programme (2014), BMC Health Services Research, 13.

Intensive tutoring and counselling for struggling teenagers

This working paper, published by the US National Bureau of Economic Research, presents findings from a randomised controlled trial of an intervention that aims to provide both academic and non-academic remediation for disadvantaged teenagers who are falling behind and at risk of dropping out of school. The academic portion of the intervention includes intensive, individualised one-to-two maths tutoring provided for an hour every day. The non-academic portion includes social-cognitive skills training such as learning how to evaluate consequences ahead of time.

The study took place in a high school in a deprived area of Chicago with high levels of ethnic minority pupils and where nearly all the pupils are eligible for free or reduced-price lunch. The sample was 106 males aged 14–16 who were identified using an “academic risk index”.

Findings showed that participation in the intervention reduced course failures by about 66% in both maths and non-maths classes, increased rates of being “on track” for graduating high school by 46%, and showed large gains in a broad measure of maths test scores.

These are promising findings for a small-scale pilot, but testing the intervention at scale will be an important next step. Also, the current study measured outcomes only during the programme year, so no conclusions can be drawn yet regarding lasting impacts. Cost is another factor, and the authors do note that intensive small-group tutoring can be expensive. However, they say that the tested intervention (costing roughly $4,400 per pupil) seems to yield larger gains in adolescent outcomes per dollar spent than many other intervention strategies.

Source:The (Surprising) Efficacy of Academic and Behavioral Intervention with Disadvantaged Youth: Results from a Randomized Experiment in Chicago (2014),NBER.

What Works Centres for the UK

The UK government has launched the What Works Network – six independent institutions responsible for gathering, assessing, and sharing the most robust evidence to inform policy and service delivery in health, education, crime, promoting active and independent ageing, effective early intervention, and fostering local economic growth. The network includes a number of existing organisations:

  • The What Works Centre for Improving Education Outcomes for School-aged Children will be the Education Endowment Foundation.
  • The What Works Centre for Early Intervention will be provided by the recently formed Early Intervention Foundation.
  • The National Institute for Health and Clinical Excellence (NICE) will provide the What Works Centre for health.

Other centres will be set up in the coming months.

Evaluation of children’s centres in England

The Evaluation of Children’s Centres in England (ECCE) is a six year study, commissioned by the Department for Education and undertaken by NatCen Social Research, the University of Oxford and Frontier Economics, that aims to provide an in-depth understanding of children’s centre services, their effectiveness and cost efficiency in delivering different types of services. In this first report from the study, children’s centres in the most deprived areas are examined using the responses from a survey of children’s centre leaders conducted in July and September 2011.

The report shows the changing environment in which children’s centres operate with 40 per cent experiencing recent cuts in services or staffing, and many leaders managing two or more centres. The subsequent outputs from this study will examine children’s centres’ service delivery, multiagency working and reach, impact analysis, cost benefit analysis, and the families using them.

Source: Evaluation of Children’s Centres in England (2012), Department for Education

What makes for an effective student reward?

A new working paper from the National Bureau of Economic Research in the US explores the short-term effects of incentives on student effort and performance, varying the size and type of the rewards as well as how they are presented. As part of the study, field experiments were conducted across multiple years in over 7,000 US elementary and high schools. Findings were as follows:

  • Incentives framed as losses (ie, a reward that is given before an assessment begins that the pupil can keep if they meet the goal, or will have to give back if they don’t) have more robust effects than comparable incentives framed as gains (ie, receiving a reward only after the goal is met).
  • Non-financial incentives (eg, a trophy) are considerably more cost-effective than financial incentives for younger pupils, but were not effective with older pupils.
  • All motivating power of the incentives vanishes when rewards are handed out with a delay rather than immediately. For this study, the delay was one month.

Source: The Behavioralist goes to school: Leveraging behavioral economics to improve educational performance (2012), National Bureau of Economic Research