In his introduction to the latest edition of Evaluative Practice, Professor Murray Saunders, Editor of Evaluative Practice and a co-founder of the UK Evaluation Society, reflects on the diversity of approaches available to evaluators and the importance of adaptability and ethical practice in responding to real-world challenges.
In this issue:
- Message from the President Dr Kirstine Szifris
- Message from the Director Nick Posford
- Editorial Murray Saunders
- The brief before the method: How evaluation problems are built in from the start
- A Tale of Two Clocks: Evaluating in System Time
- Taking a Stronger Ethical Approach to Evaluation: Towards an Empathetic Evaluation Paradigm for Active and Protracted Conflict
- Beyond the Average: Using Statistical Process Control in Evaluation
- What if a method is not enough? Building a bridge to evaluative cultures in place-based systems
- The Nourish school food improvement programme: connecting frontline and evaluation teams through adaptive evaluation
- What evaluating social and emotional learning in Indian public education systems has taught us about standardisation, contextual validity and professional judgement
- Shortening the findings-to-action chain: Creating a collective evaluative practice
- Learning partnerships: reimagining evaluation consulting in education and youth services
To read the full issue you need to log in to the member resources hub.
It is with pleasure that we offer this bumper edition of Evaluative Practice. It embodies the rich diversity of approaches to evaluation and eschews the view that we are bounded by a dominant mode and pre-occupied by metrics alone. The broad message here is that there is the use of a variety of perspectives and methods at our disposal with a focus on adaptability and nimble responses to the challenge of evaluation in real time. This is accompanied by a clear-eyed view of ethical and responsible practice with all stakeholders in the ecology of an evaluative environment.
As in other editions we are fortunate to feature the work of the poster prize-winners in this year’s conference, all of whom have converted their work to short evocative pieces appropriate to the format of the publication.
Kevin Harris and the team from the Universities of Hartpury, Plymouth, York St John, Sheffield Hallam and Sport England build on their outstanding poster award and explore a more distributed and embedded approach to building evaluative culture. The use of a bridge as a metaphor is apt and works excellently as a way of foregrounding the often untidy context of evaluative environments. Continuing with the theme of ‘evaluative culture’ is the innovative approach advocated by Esher Winslow. With her award of ‘best’ presentation she gives us an evocative account of an approach designed to ‘bridge the gap’ between findings and actions. The use of a wide variety of data sources, regular collective sense making and an iterative process of identifying change requires a serious approach to preparatory work on expectations.
The promises and the challenges of this approach are addressed in Esther’s piece. Emily Dawson and Priya Cooper develop their ideas from their prizewinning contribution on the evaluation of the Nourish School Food Improvement Programme in which they explore the idea of an adaptive evaluation. Emily and Priya argue that adaptive evaluation (involving ongoing dialogue between evaluators and delivery teams), using evaluation findings to support continuous improvement, strengthening programme responsiveness, and generating practical evidence for implementation requires trust between the evaluation team and ‘implementers’ and a focus on learning rather than end point judgements. Deepta Sunil Valliyil recasts the positioning of evaluation in her discussion piece based on her poster that won the Best Poster Prize for Evaluation. She uses power and partnership as the principle analytical lenses, arguing for the value of a learning partnerships model as a means of making evaluative relationships more equitable, accessible and useful. She points to a paradox at the heart of many evaluations in which evaluators are expected to be both objective ‘outsiders’ and collaborative codesigners. This piece examines this tension through the lens of power and partnership.
There are synergies between the adaptive learning focus, the collaborative aspirations and permeable boundaries within the evaluative space in all contributors from prize-winners in this edition.
Roy Aulie Jacobsen’s contribution takes us back to the point at which evaluation begins: the brief. His deceptively straightforward question about whether a programme worked, whether it should continue and what should change before the next funding cycle points to a more fundamental problem.
Evaluation is often asked to answer questions for which the conditions for answering them have not necessarily been established. Tight timelines, uneven monitoring data, partial baselines and the absence of a comparison group can all become features of the evaluative landscape, even when the brief implies a much stronger claim about causality and impact. The piece is a useful reminder that the quality of an evaluation is not simply a matter of choosing the right method once the work has begun. The problem itself needs to be properly constructed at the outset, with a realistic appreciation of what the available evidence and design can support.
Georgia Pike continues this concern with context by turning our attention to time. Her ‘tale of two clocks’ makes the important point that evaluation within complex systems cannot easily be detached from the timescales over which change actually occurs. We can become preoccupied with methodological questions about experimental and quasi-experimental designs, indicators and robust evidence, whilst overlooking the fact that health, organisational and system change may unfold according to quite different temporalities. The question of how long it takes to see an outcome is therefore not simply a technical matter. It raises questions about the timing, relevance and utility of evaluation and about whether the evaluation itself is properly aligned with the system in which it is taking place.
place. The ethical dimension of evaluation is brought into particularly sharp relief by Deepti Sastry in her discussion of evaluation during active and protracted conflict. Here the familiar language of rigour takes on a different meaning. How do we generate credible evidence when those from whom we seek knowledge are themselves living with insecurity, trauma and uncertainty, and when the act of evaluation may impose additional demands? Sastry’s proposal for an empathetic evaluation paradigm is an important contribution to this debate. Her five principles of contextual empathy, relational ethics, adaptive rigour, shared duty of care and reflexive objectivity suggest that empathy need not be seen as an alternative to rigour. Indeed, in such circumstances, understanding the conditions under which people are able and willing to contribute knowledge may itself be essential to producing credible evidence.
Chloe Preston, Francisco Frasquilho and Marco Aurelio provide an interesting methodological intervention in their exploration of Statistical Process Control. Their starting point, that all improvement requires change but not all change results in improvement, provides a useful introduction to a method which has considerable potential beyond its origins in manufacturing. Control charts offer a means of seeing variation over time, distinguishing common from special causes and identifying patterns that might otherwise be obscured by more conventional pre- and post-measures. There is an important connection here with the wider emphasis in this edition on evaluation as a process of learning and adaptation. Rather than waiting for an endpoint at which a judgement can be made, SPC offers the possibility of using evidence as a continuing resource for understanding and responding to change.
The contribution from Joseph Thomas Rijo, Apoorva Bhatnagar, Sreehari Ravindranath and Amit V. Kumar takes us into another important territory: the relationship between standardisation and contextual validity. Their experience of evaluating social and emotional learning in Indian public education systems leads them to question an assumption that is deeply embedded in evaluative practice – that consistency of measurement is necessarily synonymous with rigour. The argument is not against standardisation itself, but asks what happens when the same instrument produces results that are comparable in form but not necessarily in meaning. This is particularly significant when attempting to measure outcomes such as trust, belonging, voice, resilience and emotional regulation, where the measures we use are necessarily proxies for complex human experiences. Their proposition that changing an instrument, or limiting the claims made from it, can sometimes represent the more rigorous choice is a valuable challenge to methodological orthodoxy.
There is a common thread running through these contributions. Each, in different ways, asks us to pause before assuming that the established evaluative response is necessarily the appropriate one. Together these pieces reinforce the central message of this edition. Evaluation is not a single technical activity conducted at a safe distance from the world it seeks to understand. It is an adaptive, situated and relational practice. Its credibility depends not only on the technical properties of its methods but also on the quality of the questions we ask, the timing of our interventions, the measures we select, the relationships we establish and the ethical responsibilities we accept. The challenge, as these contributors demonstrate in different ways, is not to abandon rigour but to make our understanding of rigour sufficiently responsive to the complexity of the environments in which evaluation now takes place.
A huge thank you to all our Evaluative Practice contributors for sharing their insights, experiences and examples from across the evaluation community — your work continues to spark learning and reflection across the field.
We welcome contributions from both members and non-members who want to share their ideas, experiences or reflections on evaluation. To discuss a potential piece or submit a short abstract for consideration, please email hello@evaluation.org.uk with the subject line ‘Att. Editor – Evaluative Practice’.
