Author: Prof Lisa Holmes (University of Sussex) and Prof Jenny Woodman (UCL)
How do we avoid results we can’t interpret in natural experiment evaluations? Examples from children’s social care
When natural experiment evaluations produce ‘no evidence of effect’, what does that really mean? Does it show an intervention made no difference, or that the evaluation could not detect one?
Drawing on their recent paper in the British Journal of Social Work, Prof Lisa Holmes (University of Sussex) and Prof Jenny Woodman (UCL) explore three common challenges when using administrative data in natural experiment evaluations and share practical lessons for producing findings that are more meaningful and easier to interpret.
The promise and limits of administrative data
Administrative data are increasingly used to evaluate the impact of policies and complex interventions. These approaches offer clear advantages: longitudinal, population level coverage, lower cost, and less burden on children, families and practitioners. But what happens when these evaluations produce results we cannot interpret?
In our recent paper in the British Journal of Social Work, we show this is not uncommon. For example, recent evaluations of Health & Social Care Partnerships in Scotland, of Multi-Agency ‘Front Door Services’ in children’s social care, of health visiting (public health service for under 5s) and of Special Educational Needs and Disability services all found ‘no evidence of effect’. However, they also concluded that their results may all be due to methodological challenges. This leaves a tricky question: has the evaluation found that the intervention does not help children, young people and families at all, or only that no change was detected because of the selected outcomes and/or methodological challenges? We argue that for meaningful and responsible evaluations, we need to think beyond the analysis of routinely collected administrative data.
Three questions every evaluator should ask
Drawing on insights from two webinars hosted by the Children’s Social Care Data User Group, we identified three core challenges that administrative data cannot resolve in isolation. Although our examples come from children’s social care, these challenges and the lessons they raise will resonate with evaluators working across other areas of public policy.
Challenge 1: What is the intervention?
For an evaluation to tell us anything useful about impact, we need to be clear about what is being evaluated—both in theory (to design the evaluation) and in practice (to interpret findings). This information is rarely well captured in administrative datasets. Without such understanding, evaluators risk choosing the wrong populations or outcomes (if theory is not well-developed) or misinterpreting findings (if implementation is not understood).
This is particularly problematic for complex interventions, which are implemented differently across sites, including in ways that potentially undermine effectiveness. For example, research in England found that more than 1 in 4 local areas had implemented Signs of Safety (trademarked approach to working with families in children’s social care) but that areas took a ‘pick and mix’ approach, with the more relational elements less frequently implemented than the structured tools and checklists.
Challenge 2: Which groups are being compared?
An impact evaluation needs an appropriate comparison group to estimate outcomes for those who did not receive the intervention as well as those who did. For a comparison group to be meaningful, it must be similar to the intervention group on all relevant characteristics at baseline, i.e., before the intervention starts. If this condition is not met, differences in outcomes may reflect pre-existing differences rather than intervention effects.
While administrative data are often used to construct comparison groups, they may not contain all the information needed to ensure the comparison and intervention groups are equivalent across crucial characteristics. These limitations are especially pronounced for targeted interventions where referral and eligibility decisions may be shaped by professional judgement rather than clearly recorded criteria. For example, in a study about health visiting, babies who received ‘enhanced’, more frequent health visiting had higher rates of unplanned hospital use, exactly the opposite of what was expected. Qualitative research suggested that one possible explanation for these results was that the administrative data on which the study relied did not measure the most important triggers for enhanced services, which included family structure and networks, siblings with emotional or behavioural difficulties and quality of physical home environment. In other words, the groups may not have been comparable, affecting the outcome of the study.
Challenge 3: Are we measuring what matters?
Administrative datasets tend to prioritise service activity and demand measures, such as referrals rather than outcomes. These measures are often readily available and attractive to evaluators but do not tell us much about more experiential measures, such as practitioner interactions or concerns about a child. They also lack information about the quality of family interactions. Child, family and caregiver views and experiences are also largely absent from administrative returns.
For many evaluations, at least some critical outcomes will therefore need to be collected directly from children, families, and/or practitioners, rather than inferred from administrative data.
Taking the learning forward
We do not dispute that administrative data remain a powerful resource for informing policy and practice, including the evaluation of complex interventions. However, they are not a complete solution.
Instead, we argue for an approach that brings together skilled research teams, practitioners, and people with lived experience, to work collaboratively while utilising a variety of complementary data sources. Such investment is essential if evaluations are to move beyond what is easily measured and towards what is genuinely meaningful.
Our thanks to Lisa Holmes and Jenny Woodman for this guest blog and for contributing their perspective to the UK Evaluation Society’s work.
Acknowledgements
This blog was written in collaboration with the wider authorship group of the British Journal of Social Work journal article: Rick Hood, Louise Mc Grath-Lone, Rocio Méndez Pineda, Jeongeun Park, Jonathan Scourfield and Joanna Soraghan.
How to cite this article
Holmes, L. and Woodman, J. (2026) How do we avoid results we can’t interpret in natural experiment evaluations? Examples from children’s social care. UK Evaluation Society blog. Available at: https://evaluation.org.uk/blog-natural-experiment-evaluations/(Accessed: [insert date]).
Professor Lisa Holmes
Lisa Holmes is Professor of Applied Social Science, within the School of Education and Social Work at the University of Sussex.
She was previously an Associate Professor within the Rees Centre (University of Oxford) and Deputy Director of Research for the Department of Education. Lisa was Director of the Rees Centre from January 2018 until January 2021. She was also formerly the Director of the Centre for Child and Family Research (CCFR), Loughborough University.
Over the past twenty years Lisa has undertaken a body of research funded by government departments, non-government organisations and children’s charities to inform child welfare policy and practice. Publications include academic journal articles and books, as well as government reports and practice guides to inform policy and practice.
Lisa also carries out responsive studies for various government departments in England, these include the Department of Health, Department for Education and Ministry of Justice. The responsive studies are carried out to address specific policy issues.
Lisa first started her career in child welfare as an outreach worker in 1991, working with vulnerable families followed by two years working as a residential social worker in an adolescent children’s home.
Professor Jenny Woodman
Prof Jenny Woodman is Professor of Child and Family Policy at UCL, with a cross-faculty appointment spanning the UCL Social Research Institute and the UCL Great Ormond Street Institute of Child Health.
Her research focuses on evaluating public services for vulnerable children and families, particularly at the intersection of health and children’s social care. Using linked administrative data alongside qualitative research, she explores how children and families interact with health, social care and education services and how these services can be improved.
She is Co-Director of the NIHR Children and Families Policy Research Unit and co-founded the Children’s Social Care Data User Group (CSCDUG), a network that brings together researchers, policymakers and practitioners to improve the use of children’s social care data.
