Test results don’t show how effective teachers are

A new study has looked at the link between instructional alignment (how teaching is aligned with standards and assessments), value-added measures of teacher effectiveness, and composite measures of teacher effectiveness using multiple measures.

The issue is important as, in the US and around the world, there is more emphasis on measuring teacher effectiveness and rewarding effective teachers. The study looked at 324 teachers of fourth and eighth grade (Year 5 and Year 9) mathematics and English language arts in five US states. They completed a Survey of Enacted Curriculum to measure their instructional alignment. This was then compared with value-added measures (taken from state assessments and two supplementary assessments) and teacher effectiveness (using Framework for Teaching scores, widely used by states).

The results showed modest evidence of a relationship between instructional alignment and value-added measures, although this disappeared when controlling for pedagogical quality. The one significant relationship they found was that the association between instructional alignment and value-added measures is more positive when pedagogy is high quality. There was no association between instructional alignment and measures of teacher effectiveness.

These results suggest that the tests used for calculating value-added measures are not able to detect differences in the content or quality of classroom teaching.

Source: Instructional Alignment as a Measure of Teaching Quality (2014), Education Evaluation and Policy Analysis, online first, May 2014.

Weak links between teacher perception surveys and student outcomes

This report from the Institute of Education Sciences provides information on how perception data from a teacher survey in Idaho correlated with student outcomes that school improvement efforts seek to affect. The survey was the Educational Effectiveness Survey (EES), an annual teacher survey developed and administered by the Center on Educational Effectiveness to gather data on school qualities believed to be the goals, processes, and supports essential for school success.

A total of 75 low-achieving schools received the survey, and 1,745 teachers (91%) responded. The teachers rated their perceptions of their school on topics such as effective school leadership; curriculum, instruction, and assessment aligned with standards; and focused professional development.

Analyses of the survey data and publicly available data from the Idaho State Department of Education revealed that teachers’ perceptions of the presence of essential goals, processes, and supports were generally not related to students’ reading proficiency, math proficiency, or attendance. According to the report, these findings suggest that educators should proceed cautiously when using perceptual survey data to make school improvement decisions.

Source: Connections Between Teacher Perceptions of School Effectiveness and Student Outcomes in Idaho’s Low-Achieving Schools (2014), Institute of Education Sciences.

Improving reading outcomes for pupils with or at risk of reading problems

The US Institute of Education Sciences (IES) has released a new synthesis of research on improving reading outcomes for pupils with or at risk of reading problems. The purpose of the review is to describe what has been learned specifically from IES-funded research grants. A total of 111 peer-reviewed journal articles and book chapters that were products of 48 research projects are examined. A few key findings, which are grouped into four domains, are as follows:

  • Assessment: Screening all children’s reading skills (ie, universal screening) at the beginning of the school year, especially in the early years, can be a valid and efficient process to identify children who are at risk of reading difficulties.
  • Basic cognitive and linguistic processes: Malleable linguistic processes, such as oral language skills and vocabulary, positively predict children’s reading performance.
  • Intervention: Language outcomes for many preschool children at risk for language delays can improve if they are provided extensive opportunities to hear and use complex oral language.
  • Professional development: Developing teachers’ specialised knowledge and supporting consistent long-term implementation of research-based teaching strategies can improve delivery of complex, evidence-based teaching and interventions.

Source: Improving Reading Outcomes for Students with or at Risk for Reading Disabilities: A Synthesis of the Contributions from the Institute of Education Sciences Research Centers (2014), Institute of Education Sciences.

Academies under-performing on GCSEs

NFER have released a new report analysing the progress between Key Stage 2 and GCSE of pupils attending academy schools and non-academy schools. The authors found that in 2011 and 2012 pupils at academy schools achieved, on average, higher attainment outcomes and made more progress between KS2 and KS4 than those at non-academy schools when taking into account both GCSEs and non-GCSE qualifications (eg, NVQs).

However, analysis of the 2012 data excluding non-GCSE qualifications show that academy schools (that had held that status for more than two years) had average GCSE scores that were significantly lower than non-academy schools. The authors say that this may indicate alternative entry policies into GCSE and non-GCSE qualifications, or that pupils in academies perform particularly well in non-GCSE subjects.

Longitudinal analysis of GCSE outcomes from 2007 to 2012 showed no significant improvement in the rate of improvement of GCSE results for academy schools over and above the rate of improvement in all schools.

The research took into account other school level factors that may have been associated with a variation in progress, including the proportion of pupils receiving free school meals and with special educational needs, as well as geographical location.

Source: Analysis of Academy School Performance in the 2011 and 2012 GCSEs (2013), NFER.

Examining the effects of assessment

Researchers at the RAND Corporation have conducted a series of literature reviews that focus on topics such as high-stakes testing, performance assessment, and formative evaluation.

Their findings, published in a new report, suggest that there are a wide variety of effects that testing might have on teachers’ activities in the classroom, including changes in curriculum content and emphasis (eg, changes in the sequence of topics, reallocation of emphasis across and within topics); changes in how teachers allocate time and resources across different pedagogical activities (eg, focusing on test preparation); and changes in how teachers interact with individual pupils (eg, using test results to personalise teaching). The report, “New Assessments, Better Instruction? Designing Assessment Systems to Promote Instructional Improvement”, also identifies a number of factors (eg, pupil characteristics and regional policies) that mediate the relationship between assessment and teaching practices.

The authors suggest that the role of tests would be enhanced by policies that ensure tests mirror high-quality teaching, are part of a larger, systemic change effort, and are accompanied by specific supports for teachers.

Source: New Assessments, Better Instruction? Designing Assessment Systems to Promote Instructional Improvement (2013), RAND Corporation.

High-stakes tests may damage teaching quality

A new article published in the American Educational Research Journal has found that the quality of instructional support (ie, teaching methods and classroom organisation) is lower when teachers are under the greatest pressure to increase test performance.

The authors used two years of observation data from a cohort of US pupils who were first graders (Year 2) during the 2007–08 school year. A total of 348 observations took place in 23 classrooms in eight selected schools, when the children were in second grade and third grade (Years 3 and 4).

Using the Classroom Assessment Scoring System (CLASS), the researchers found that in the months leading up to high-stakes testing in Year 4, teachers in these classrooms offered lower levels of instructional support than Year 3 teachers who were not experiencing the same level of accountability pressure. However, observations after the tests revealed the quality of instructional support was indistinguishable between Years 3 and 4.

The authors suggest that accountability policies do not necessarily need to have negative consequences for classroom quality, but could be designed to improve it by including relevant measures.

Source: Pressures of the Season: An Examination of Classroom Quality and High-Stakes Accountability (2013), American Educational Research Journal, 50(5).