Through the perspectives of an experienced commissioner, and evaluator, this series explores the realities of evaluation Terms of Reference. Two voices – Jonathan Kuhn-Patrick (commissioner) and Mat Pritchard (evaluator) share their views in an open conversation.
Each article in the series examines the assumptions, tensions and trade-offs that shape evaluation in practice, drawing on decades of experience from both sides of the table.
This instalment looks at the comments matrix and who pays for what it reveals.
The email arrives
Mat and Jonathan: You’ve put in months of work. The data is collected, cleaned, analysed and argued over, first inside the team and then over the wording of the draft. The report goes to the client and you wait. How will they react? In horror at the negative findings, or with a list of typos and nothing else?
Then the email: “Please find attached the consolidated comments matrix.”
Somewhere in that matrix is the answer to a question this series has been asking since the first post. The Terms of Reference asked for more than the money could buy. The bid said it could be done anyway. The inception report trimmed it back. Delivery went the way delivery goes. A surprising amount of what looks like disagreement about the draft report is an earlier risk decision coming home to roost.
Very little of this gets created at report stage. It just shows up there.
And when a risk surfaces in the matrix, somebody has to pay for it. This post is about who pays, and how that gets decided.
Where the comments come from
Jonathan: Let me start by defending the commissioner, because the usual story does us a disservice.
The usual story is that commissioners want positive findings and lean on evaluators until they get them. When we started this series I expected to write that post, complete with the gotcha moment and the smoking gun. I had sat with programme managers who told me, without embarrassment, that they didn’t much mind how an evaluation went as long as it came out positive. A programme’s standing is already on the record in its annual review score, and to a lesser extent the narrative, and an evaluation that contradicts the score is about as welcome as a doo-doo in the bathwater.
Then I went looking for a case from my own years commissioning where a finding actually bent under pressure. I couldn’t find one I would stand behind. Plenty of attitude and some posturing, but nothing I could point to.
And even on my side there are two camps. Programme teams have a stake in what the findings say. Most evaluation advisers don’t much mind, as long as the method stands up.
Most hard questioning is something else. The evaluation team has spent months swimming in the data. I get a compressed report where a sentence resting on sixty interviews and six datasets looks much like one resting on three, or like an opinion masquerading as a finding. I can’t tell which is which, so I query the method, because the method is the only lever I have. That is pressure-testing.
Mat: There’s a version of the usual story on my side too. Nobody has to lean on anybody, because we do it to ourselves first.
Findings do get softened at drafting stage, and it happens through our internal pressure-testing. We know the client is going to challenge the evidence, so we challenge it ourselves before they do. A form of words that implies a more concrete pile of evidence than we’ve actually got will get picked up by someone on the team, or by the external QA reviewer if not. That is exactly what pressure-testing is for, and it makes the report better.
The question I can’t answer is how much of that is commercial worry about the next contract dressed up as sensible pressure-testing. I’m sure it’s there, lurking in the background. But I’ve never once heard anyone say we can’t put that in because we’ll lose the client. James McNulty, writing in 2012, lists the fear of being too frank among the threats to evaluator independence, and I’m sure he’s right. I just can’t tell you when it’s operating and when it isn’t, including in my own work.
And it pulls both ways. A lot of evaluators are driven by fairly deep convictions about what the evidence will and won’t bear, and there’s the same reward in our work as in any research for being the one who says hang on, I don’t think that’s right.
Jonathan: From my side, we can’t tell the difference. Pressure-testing and softening look identical from where the commissioner sits, because all of it happens before the report reaches us and we only ever see what survived the cull. It can cut the other way too, and a critical finding can come out stronger once someone has gone back to the evidence. But from the outside there’s no way of knowing which happened. The worry, on our side, is that pressure-testing becomes pressure white-washing.
And yet the commissioner’s side has a failing here that nobody talks about, and it’s volume.
A draft goes round a group of advisers. Each adds comments because commenting is what being consulted means and few are sufficiently confident and experienced to say that something is good enough. Advisers strive for perfection in an ideal world, not adequacy in complex environments dealing with wicked problems. Nobody filters them – or at least not enough. The programme manager running the evaluation is usually too junior or lacking the confidence borne of years of practical relevant experience to tell a senior adviser that their comment is out of scope or not worth the four days it will cost to fix. What reaches the evaluator is everyone’s comments added together – which is not the same as a position.
So a comment of twenty words arrives in the matrix and means four days of reopening the data. We write the twenty words and never hear about the four days.
Triage
Mat: So the matrix arrives and I sort it, informally, into three buckets.
Every comment gets tested against three things in order: evidence, cost and reputation. There are more factors than that, but those are the main ones. Evidence comes first and does most of the sorting. Does the comment point at a real gap or weakness in our evidence, our analysis or the way we’ve presented it? Then cost, because any change takes time and time is money we have usually already spent and there is no magic money tree to shake. Then reputation, which can overturn the other two.
Bucket 1: the GottaDos. Fair points. We have to do something about them, and we do, sometimes whatever it costs.
Bucket 2: the NoWayJoses. Not supported by our evidence, outside the scope, or a plain misunderstanding. We’re on solid evidential or contractual ground, so we decline.
Bucket 3: the Difficult Middle. Reasonable enough to take seriously, but not fixable by rewording. They mean real extra work, or a real reputational hit if we get them wrong.
The buckets only tell you how we respond. To see where a comment came from, and who ends up paying for it, you have to follow it back.
Three examples of comments, followed back
A NoWayJose: “The report does not adequately address impact”
Mat: An easy one to deal with. But look at the route it took to get there.
The ToR had a lot of evaluation questions, too many for the budget, and not much to say about method. The bid team looked at the questions, at the projects’ own monitoring data and at the money, and did what bid teams do:
“We’ll use a contribution analysis approach because the projects will have their own quant data; we can quality assess it and collect quicker and cheaper qualitative data to combine with theirs. We will sidestep the impact indicators and questions entirely because they’re set at way too high a level, would likely require a decade’s worth of evidence from a three-year project, and anyway there’s a massive conceptual leap between the outcomes and the impact; too big to address with this budget. Thus we can produce a Mixed Methods Outcomes-focused Contribution Analysis Reduced-Questions Approach that fits the budget! Hooray!”
And in our defence, inception agreed it. By then the commissioner’s quantitative specialist had moved on, so nobody queried the absence of our own quant data collection, the focus on contribution rather than attribution, or the questions that had gone missing.
At comment stage somebody went back to the original ToR, skipping past our technical proposal and the inception report, and spotted the discrepancy. No impact. No quant. Questions unanswered.
We sent them the inception report. That was the end of it.
Jonathan: I’d push back on the “Hooray”.
My side wrote a ToR asking for impact on a budget that couldn’t buy it. Mat’s right about that and I owned it in the first post. But his side saw the problem, designed around it and said so only in a technical proposal and an inception report that nobody on my side read closely. That’s the cheese-grater out of the drawer again.
And it only cost nothing at the end because the inception report had written the compromise down. Mat’s team could find the paragraph that said impact was out of scope, and send it back. Without that paragraph there is nothing to send. The same comment then sits in the Difficult Middle and costs him days – or more.
A Difficult Middle: “We now have updated information, please change your findings”
Mat: A case study found a project short of its fundraising target. Raising extra money was one of the things the project was set up to do. The evaluation then ran late, and by the time the report went in the figures had moved. Back came the comment: the shortfall is now much smaller, please update your findings.
Ouch. There wasn’t much of a choice. The client had newer data and was asking us to use it, and you don’t tell a client their own figures don’t count. It looks like a small change, but it meant reopening the budget data – what had been committed against what had actually been secured – then the spending forecasts, and then every judgement resting on them: efficiency, effectiveness, the theory of change pathways, relevance, the recommendations. The question became how much we could afford to spend making the case study right again without breaking the bank.
We got lucky. The project had over-achieved in a couple of critical areas, so the shortfall mattered less than it might have. For a while, though, I had my head in my hands.
Jonathan: I wouldn’t have made that comment. An evaluation reports what was true when the fieldwork was done. If the figure has moved since, the commissioner can say so in the management response, which is ours to write and costs the supplier nothing.
Asking the evaluator to update it is a choice. And why was the update needed at all? Evaluations slip for all sorts of reasons and often they are ours: approvals sitting in an inbox, access to partners that takes months to arrange, a steering group that can’t find a date. The delay often starts with us. A fixed price then turns our delay into the supplier’s cost.
A GottaDo: “The report does not address future programme design”
Mat: This one was our fault. The contract clearly asked for lessons on the design of future donor programmes. The team concentrated on lessons for the implementing agencies and government departments and missed it. Nothing upstream caused that — no impossible ToR, no budget that wouldn’t stretch. We just didn’t do it.
For a while it looked like a disaster: new data collection, new analysis, the team pushing back hard. In the end they had the data and most of the analysis already. I spent a day and a half going through very long annexes to confirm it. After that it was a matter of revisiting old material, some further analysis and setting the recommendations out properly.
Because it was a clear gap against a clear contract, we, the prime, asked the consultant team to make good the shortfall at their own expense. That’s a normal commercial call. The contract said what was required, it hadn’t been delivered, and the firm wasn’t going to pay twice for the same work.
It’s also how risk reaches the individual consultant. Nobody behaves badly. Each step down the chain is reasonable on its own.
Jonathan: And from where I sat it was invisible. The comment was fair, it cost the commissioner nothing and the matrix came back resolved. Nothing in it told me that the resolution had been paid for by people at the bottom of a subcontracting chain, working unpaid.
Put the three comments side by side and the pattern is plain. In the first there was no cost, because a document had already settled it. In the second the firm paid. In the third, individuals. From the first to the third, the cost settles a step further down the chain each time, and each step is harder for the person who wrote the comment to see.
What didn’t happen
Jonathan: None of these was a finding bent to suit the commissioner.
That isn’t only our experience. A research group at the LSE surveyed academics who had recently completed research and evaluations commissioned by government, asking about pressure at three stages: setting the specification, managing the work, and commenting on the draft.
Only one of the three showed a significant relationship with how supportive of policy the finished report turned out, and it wasn’t the comments stage. Where government had set the research questions, half the reports came out supportive. Where the researchers had set them, under a quarter did. Requests for changes to the draft made no significant difference at all, and of those asked to change something, three-quarters reached a compromise rather than simply doing as they were told.
So it’s the specification that matters, which takes us straight back to the first post and the ToR.
One doubt about the mechanism, though. Do researchers go looking for failing public policy when left to their own devices? “Public policy does what was intended” is not exactly a career-enhancing headline. The study can’t tell us whether government narrowed the questions or academics chose the interesting ones.
Morris and Clark surveyed American evaluators and found that where they had felt pressured to misrepresent findings, they judged it had actually happened in about one case in six. Pressure happens often and rarely changes the report. And Mat has never had a report fall over at comment stage.
Mat: Two caveats from my side. The LSE study covered academics, who have tenure and don’t live or die by the next contract. The authors approached researchers at commercial organisations too and got no replies, so the people with most to lose by holding the line are the ones nobody has managed to ask.
And none of it can catch what I described earlier – commercial worry dressed up as pressure-testing – because the person answering the survey doesn’t know either.
Jonathan: But your own three comments say the same thing. In none of them did a finding change. What changed was who did the work.
Mat and Jonathan: Findings get defended. Extra work gets paid for, by somebody. The comments matrix is where the bill gets passed down.
The other way it goes wrong
Mat: Not every matrix is a fight. On one programme the client and the implementing partner went through the report correcting typos and grammar and never went near the substance. No argument about the findings, because nobody had engaged with them.
Jonathan: I’d guess that’s the more common failure, and in some ways the worse one. A finding nobody bothers to contest is one nobody is going to act on either.
The risk nobody pays for
Mat and Jonathan: Some risk is never settled at all.
Suppose everyone decides a point isn’t worth another round. A thinly evidenced conclusion survives. A recommendation reads more confidently than the analysis behind it. The report is approved, the invoices go in and everyone moves on.
The report stays. It goes into the programme’s files as evidence, and three years later another team treats it as established and builds on it. The risk has left this evaluation and become somebody else’s evaluability problem.
We can’t point to a case that proves it. But anyone who has inherited a baseline and wondered what it was ever based on will recognise it.
Post two followed risk down a chain of contracts, settling on whoever was least able to carry it. This one travels forward in time instead, and settles on people who were never party to the contract.
Acceptance
Mat and Jonathan: Eventually the matrix closes. The commissioner accepts the report, the evaluator has accepted the comments or explained why not, and the marriage of convenience gets its final signature.
What gets accepted is a bill, settled a long way from where the risk started and usually by whoever was least able to say no.
Which leaves the question for our final post. If both sides can see the risk moving down the chain, and the comments matrix is only where it finally shows, what would it take to stop it moving, without breaking a system that exists for good reasons?
A note on AI. Both authors used AI tools to help draft and edit this post. We did so in line with the UK Evaluation Society’s good practice guidance on the use of AI in evaluation: with human direction throughout, and with the two of us taking full responsibility for the final text and the views in it.
A note on AI from the authors. We used AI tools for some of the drafting of this post, in line with the UK Evaluation Society’s good practice guidance on the use of AI in evaluation. The creative process – the arguments, the disagreements, the sarcastic humour, and the editing – was ours throughout. Ownership of the material and responsibility for the views in it rest with us.
Our thanks to Jonathan Kuhn-Patrick and Mat Pritchard for this guest blog and for contributing their perspective to the UK Evaluation Society’s work.
References
The LSE GV314 Group (2014) ‘Evaluation under contract: government pressure and the production of policy research’, Public Administration, 92(1), pp. 224–239.
McNulty, J. (2012) ‘Symbolic uses of evaluation in the international aid sector: arguments for critical reflection’, Evidence & Policy, 8(4), pp. 495–509.
Morris, M. and Clark, B. (2013) ‘“You want me to do WHAT?” Evaluators and the pressure to misrepresent findings’, American Journal of Evaluation, 34(1), pp. 57–70.
How to cite this article
Kuhn-Patrick, J. and Pritchard, M. (2026) Part 4: Terms of acceptance Available at: https://evaluation.org.uk/part-4-terms-of-acceptance/ (Accessed: [insert date]).
Jonathan Kuhn-Patrick
Jonathan Kuhn-Patrick is an Independent Evaluation Consultant and international development and evaluation specialist with more than 30 years’ experience spanning humanitarian response, fragile states and complex programme evaluation. His career has included frontline NGO leadership, senior evaluation, policy and operational roles within DFID and FCDO, and extensive work across Africa, the Middle East and South Asia.
Now an independent consultant, Jonathan specialises in evaluability assessments, evaluation quality assurance and monitoring, evaluation and learning (MEL) framework design. His recent work has focused on fragile states, humanitarian response, climate resilience and infrastructure transparency across Asia, Africa, the Middle East and Latin America.
Jonathan led the development of the UK Evaluation Society’s good practice guidance on the use of AI in evaluation and continues to support professional learning in this area. He is also a former member of the UK Evaluation Society’s Board of Trustees.
Mat Pritchard
