How do we know what works in education?
Educators need strong evidence to choose the most effective educational interventions
Authors
Partner
Mind, Brain, and Education
Educators routinely put additional practices in place to support their students, on top of usual teaching. This takes resources and workload, but the impact on student outcomes can be unclear. In 10 Welsh schools in 2022, for example, 242 interventions were in use. In a sub-sample of 138 of these, there was evidence that 3% did not work, and 67% had no published evidence on their impact – there was an absence of evidence for most of the interventions.
Why is there no evidence for so many educational interventions?
While researchers see evidence as necessary for responsible practice in education, many products are created by private companies, not researchers. A well-produced flyer or a good review by word of mouth may be enough for a company to get a contract with a school or group of schools. There’s not really an incentive for companies to produce evidence surrounding their solutions.
However, the issues run deeper than that. Producing evidence is not simple, as evaluations are long, time-consuming, and expensive. Many intervention companies are start-ups without the resources to invest in rigorous evaluations. Research and evaluation happen slowly, whereas schools and the world of educational enterprise are fast paced. What’s more, if educators see students struggling, of course they want to try an intervention that might help, even if the evidence is not currently there. All this means that schools tend to pursue immediate results based on ‘best bets’ rather than waiting for certainty.
“Research and evaluation happen slowly, whereas schools and the world of educational enterprise are fast paced.”
What does evidence look like in education?
The randomised controlled trial, or RCT, is a highly trusted method for determining what works. In an RCT, participants (individual students, classrooms, or clusters of schools) are randomly assigned to either receive an educational intervention, or not. This experimental method gives a strong indication that the intervention caused any change to the participants who received it. The use of RCTs in educational evaluations has grown hugely over the past decade, which has helped establish the theoretical evidence base for some interventions.
However, there are some challenges in applying RCTs to education. RCTs were developed for health studies, in which every participant typically has the same disease or symptoms. Participants receive a measured dose of a specific, known active ingredient. Meanwhile, in educational RCTs, participants tend to vary a lot more. Researchers often don’t know what the ‘active ingredient’ is, as many practices are established without a clear theory of exactly what causes the change.
In addition, the ‘dosage’ of the intervention cannot easily be controlled. Schools have different delivery standards, they may miss intervention sessions due to other school commitments, or they may use other interventions at the same time.
Given the sheer diversity of learners, classrooms, and schools, extremely large sample sizes are usually needed for RCTs. All this means that the effects on educational outcomes are very small compared to outcomes in health studies that use the same method.
Could there be alternative methods for evaluating educational interventions?
Other evaluation methods could provide sufficient evidence for educational studies. For example, for a large-scale trial, a non-randomised design could be used alongside a qualitative evaluation of the process. This would provide a good indication of whether the intervention worked, and how. Mixed-methods research like this could give a robust picture of the evidence, allow researchers to develop the theory around the intervention, and give insights into practical implementation. Integrating multiple forms of evidence would also lower the stakes for intervention developers, as they wouldn’t have everything riding on the results of a large, expensive trial.
Likewise, experiments with small groups of participants or even individuals would be cheaper and easier to manage. Educators often want to know which interventions work best for specific student profiles and why. Smaller studies could give that information, which often gets lost in large trials.
“We appear to be heading towards a more nuanced approach to evidence generation in education research.”
We appear to be heading towards a more nuanced approach to evidence generation in education research. But even with better methods, standards are needed to ensure an intervention is suitable for schools. We hope policy and infrastructure changes will develop over time to meet this need, so that interventions have a positive impact on children’s learning.
Footnotes
This article is based on the paper ‘Evaluating What Works in the Classroom: Best Practice and Future Opportunities’ that was recently published in the journal Mind, Brain, and Education by Astrid E. J. Bowen, Lucy R. J. Palmer, Roisin C. Perry, Michael S. C. Thomas, Andrew Tolmie, Grégoire Borst, and Jo Van Herwegen. It is the second paper in the journal’s special new article format, Practitioners’ Perspectives. This format aims to help bridge the research–practice gap by offering a new platform to educators and practitioners to discuss issues linked to implementing and contextualising research findings in practice and policy. The article is also part of the Share Your Science series, a collaboration between BOLD and Mind, Brain, and Education. In the series, authors of articles published in Mind, Brain, and Education write an accompanying BOLD article to share their insights with a wider audience.
Astrid and Jo are members of the University of London Centre for Educational Neuroscience: website, LinkedIn, X.
Authors
Partner
Mind, Brain, and Education