In ministerial speeches ‘evidence-based policy’ is now as almost as routine as ‘I care passionately about …’ and is as likely to be greeted with hollow laughter. So it’s to the credit of the House of Commons Education Select Committee that it has undertaken an enquiry into the use of evidence by the Department of Education, asking DfE to list the evidential basis of a number of policies before inviting the public to comment via a web forum. Nine areas of policy were nominated for these ‘evidence checks’: phonics, teaching assistants, professional measurement metrics, summer-born children, the National College of Teaching and Leadership, universal infant free school meals, the impact of raising the school participation age, music, the school starting age. There was a further section on DfE’s use of evidence generally and this prompted the largest number of responses, including the following which we reprint as our final blog of 2014. It’s longer than usual, but then you won’t be hearing from us again until January.
DFE’s use of evidence
Several contributors to this enquiry commend DfE for its commitment to evidence, but surely this is a minimum condition of good governance, not a cause for genuflection. More to the point are the concerns of Dame Julia Higgins that DfE’s use of evidence is inconsistent (or, as Janet Downs puts it, ‘slippery’) and those many other contributors across the Committee’s nine themes who find DfE overly selective in the evidence on which it draws and the methodologies it prefers.
The principal filters appear to be ideological (‘is this researcher one of us’?) and electoral (‘will the findings boost our poll ratings / damage those of the opposition’?) and such scientifically inadmissible criteria are compounded by DfE’s marked preference for research dealing in big numbers, little words and simple solutions.
In the latter context, we should be wary of endorsing without qualification the view of several contributors that the randomised control trial (RCT) is the evidential ‘gold standard’, trumping all other attempts to get at the truth. Education is complex and contested, and its central questions are as much ethical as technical – a challenge which the fashionable but amoral mantra ‘what works’ conveniently ignores. The RCT language of ‘treatment’ and ‘dosage’ is fine for drug trials but is hardly appropriate to an activity which is more craft and art than science, and in untutored hands the effort to make teaching fit this paradigm may reduce to the point of risibility or destruction the very phenomena it claims to test. I should add that I make these observations not as a disappointed research grant applicant but as recipient of substantial funding from the rightly esteemed Educational Endowment Foundation for a ‘what works’ project involving RCT.
Of the nine ‘evidence check’ memoranda submitted to the Committee by DfE, those on phonics, the school starting age and the National College most conspicuously display some of the tendencies I’ve so far identified. Thus the defence and citations in DfE’s phonics statement neatly sidestep the methodological controversies and evidential disputes surrounding what is now the government’s mandated approach to teaching reading, so the contributor who applauds DfE’s grossly biased bibliography as ‘accurate’ is plain wrong.
DfE’s school starting age citations carelessly – or perhaps carefully – attribute a publication of the Cambridge Primary Review (Riggall and Sharp) to NFER, but again avoid any evidence running counter to the official view that children should be packed off to school as soon as possible; or the more nuanced finding of the Cambridge Primary Review that the real issue is not the starting age for formal schooling but the availability and quality of early years provision, wherever it takes place; or indeed the inconvenient truth that some of this country’s more successful PISA competitors start formal schooling one or even two years later than England.
As for the National College of Teaching and Leadership (NCTL), no independent evidence is offered in support of DfE’s insistence that this agency, and the models of teacher training and school improvement it espouses, justify its consumption of public funds. Only two publications are cited in DfE’s ‘evidence check’. One is NCTL’s statement of accounts; the other a DfE press release which is neither evidence nor independent. Proper evaluation of NCTL became all the more essential when DfE abolished the relatively ‘arms length’ bodies that NCTL subsumed and charged it with ‘delivering’ approved policies. Of course NCTL can be shown to be effective in relation to the delivery of policies x and y. But what if those policies are wrong?
The Committee has received many unhappy comments from parents about schools’ draconian responses to term-time absences. These highlight a further problem: there are important areas of educational policy, at both school and national level, where evidence is rarely or never on view and parents and the electorate are expected to comply with what may be little more than unsubstantiated claims. In the case of those blanket bans on term-time absence about which so many parents complain to the Committee, as with the tendency to fill more and more of children’s (and parents’) waking hours with homework (i.e. schoolwork done at home) of variable and in some cases little educational value, there appears to be a deep-seated assumption that schools have a monopoly of useful learning. The Cambridge Primary Review scotched this mistaken and indeed arrogant belief in the comprehensive research review on children’s lives outside school that it commissioned from Professor Berry Mayall. Except that the then government preferred summarily to reject the evidence and abuse the Review team rather than engage with the possibility that schools might do even better if more of them understood and built on what their pupils learn outside school.
So although the Education Committee has applied its ‘evidence check’ to nine areas of policy, it might also consider extending its enquiry in two further directions: first, by examining the evidential basis of policies and initiatives, such as those exemplified above, about which teachers, parents and indeed children themselves express concern; second by adding some of those frontline policies which DfE has justified by reference to evidence but which are conspicuously absent from the Committee’s list.
Examples in the latter category might include: (i) the government’s 2011-13 review of England’s National Curriculum; (ii) the development of new requirements for assessment and accountability in primary schools; (iii) the rapid and comprehensive shift to school-led and school-based initial teacher education; (iv) the replacement of the old TDA teacher professional standards by the current set; (v) the strenuous advocacy and preferential treatment of academies and free schools. Each of these illustrates, sometimes in extreme form, my initial concerns about politico-evidential selectivity and methodological bias.
Thus in the 2011-13 national curriculum review ministers deployed exceptionally reductionist and naive interpretations of the wealth of international evidence with which they were provided by DfE officials and others. They resisted until the last moment overwhelming evidence about the educational centrality of spoken language. They ignored Ofsted warnings, grounded in two decades of school inspection (and indeed evidence going back long before Ofsted) about the damage caused by a two-tier curriculum that elevates a narrow view of educational basics above all else – damage not just to the wider curriculum but also the ‘basics’ themselves. And they declined to publish or act on their own internal enquiry which confirmed the continuing seriousness of the challenge of curriculum expertise in primary schools, an enquiry which – and this much is to ministers’ credit – DfE undertook in response to, and in association with, the Cambridge Primary Review. The report of that enquiry, and the wider evidence that informed it, still awaits proper consideration. A job for the Education Committee perhaps?
Similarly, DfE, like its predecessor DCSF, has stubbornly held to its view – challenged by the Education Committee as well as numerous research studies and the Bew enquiry – that written summative tests are the best way both to assess children’s progress and hold schools and teachers to account, and that they provide a valid proxy for children’s attainment across the full spectrum of their learning.
Then, and in pursuit of what has sometimes looked suspiciously like a vendetta against those in universities who undertake the research that sometimes rocks the policy boat, DfE has ignored international evidence about the need for initial teacher education to be grounded in equal partnership between schools and higher education, preferring the palpable contradiction of locating an avowedly ‘world class’ teacher education system in schools that ministers tell us are failing to deliver ‘world class’ standards. Relatedly, DfE has accepted a report from its own enquiry into professional standards for teachers which showed even less respect for evidence than the earlier and much-criticised framework from TDA, coming up with ‘standards’ which manage to debase or exclude some of the very teacher attributes that research shows are most crucial to the standards of learning towards which these professional standards are supposedly directed.
Finally, in pursuit of its academies drive government has ignored the growing body of evidence from the United States that far from delivering superior standards as claimed, charter schools, academies’ American inspiration, are undermining public provision and tainted by financial and managerial corruption. England may not have gone that far, but new inspection evidence on comparative standards in academies and maintained schools (in HMCI’s Annual Report for 2013-14) should give the Committee considerable pause for thought about the motivation and consequences of this initiative.
In relation to the Committee’s enquiry as a whole, the experience of the Cambridge Primary Review (2006-10) and its successor the Cambridge Primary Review Trust is salutary, depressing and (to others than hardened cynics) disturbing. Here we had the nation’s most comprehensive enquiry into English primary education for half a century, led by an expert team, advised and monitored by a distinguished group of the great and good, supported by consultants in over 20 universities as well as hundreds of professionals, and generating a vast array of data, 31 interim reports and a final report with far-sighted conclusions and recommendations, all of them firmly anchored in evidence, including over 4000 published sources.
Far from welcoming the review as offering, at no cost to the taxpayer, an unrivalled contribution to evidence-based policy and practice in this vital phase of education, DCSF – DfE’s predecessor – systematically sought to traduce and discredit it by misrepresenting its findings in order to dismiss them, and by mounting ad personam attacks against the Review’s principals. Such behaviour in the face of authoritative and useful evidence was unworthy of holders of elected office and, for the teachers and children in our schools, deeply irresponsible.
It is with some relief that we note that DfE’s stance towards the Review and its successor the Cambridge Primary Review Trust has been considerably more positive under the Coalition than under Labour, and we record our appreciation of the many constructive discussions we have had with ministers and officials since 2010. Nevertheless, when evidential push comes to political shove, evidence discussed and endorsed in such meetings capitulates, more often than not, to the overriding imperatives of ideology, expediency and media narrative. This, notwithstanding the enhanced research profile applauded by other contributors, remains the default.
This particular Education Committee enquiry was set up as a web forum, with a deadline of 15 December. The DfE evidence checks and the comments they provoked may be viewed here.
DfE’s use of evidence attracted 154 comments, followed by summer-born children (111), phonics (90) and the school starting age (64).