Through the perspectives of an experienced commissioner, and evaluator, this series explores the realities of evaluation Terms of Reference. Two voices – Jonathan Kuhn-Patrick (commissioner) and Mat Pritchard (evaluator) share their views in an open conversation.
Each article in the series examines the assumptions, tensions and trade-offs that shape evaluation in practice, drawing on decades of experience from both sides of the table.
New to this series? We recommend starting with Terms of Endearment, followed by Terms of Engagement, where Jonathan and Mat explore why evaluation Terms of Reference often become a "polite fiction" and set the scene for Part 3 in this series. In this third article, Mat and Jonathan explore how delivery-stage negotiations expose hidden risks, why those risks often fall on the least powerful people in the system, and what commissioners and evaluators can do differently.
- Introduction
- The inception stage: where deferred honesty comes due
- Risk 1: When the evaluation no longer matches the brief
- The team on paper versus the team that turns up
- Risk 2: When financial risk becomes delivery risk
- Risk 3: Reputation and the hidden cost of failure
- The tools already exist – so why aren't they used?
- Designing flexibility into contracts
- Where the risk really ends up
Mat and Jonathan: In the first two articles, we pulled apart some of the ‘polite fictions’ that exist in evaluation Terms of Reference: the sky-high ambition, paired with the distinctly ground-level budget; the assumption that the project is evaluable, when data is actually sparse and/or unusable; the shared ‘performance of confidence’ that defers the hard questions. We then followed the risk into procurement, where commissioners of evaluations have to choose an evaluator somehow. We concluded that the cans kicked down the road at both ToR and procurement stages run out of road in implementation. And that is what we’re going to talk about now.
The Inception Stage — Where deferred honesty comes due
Jonathan: So far we’ve talked about the TOR and the bid. Both of these stages gloss over some risks, assumptions and expectations. But we’ve reached the implementation stage, and this is where the deferred honesty of the ToR and the bid is finally called in – under contract now, money committed, the leverage redistributed. The risk transfer stops being abstract; the risks become real hazards and both the commissioner and the contractor need to mitigate them in the harsh light of implementation reality.
What strikes me is how the guidance handles this stage – where it handles it at all. Better Evaluation, the reference source half our profession keeps open in a browser tab, points out something odd about the standard inception report: the template has no section for recording changes between the original terms of reference and what the inception report now proposes. So the changes happen, but there’s nowhere official to write them down. UNDP is more honest about it, describing the inception report as the evaluation team’s interpretation of the terms of reference. Interpretation. Of the document everyone just signed a contract against. And our own Magenta Book solves the problem a third way: the word “inception” never appears in it. One source has no space for the renegotiation, one admits it openly, and one pretends the stage doesn’t exist. Every practitioner knows it’s where the real negotiation happens.
Mat: So if this is where the real negotiation happens, let’s look under the bonnet at what’s actually on the table: the now-contracted risks that have to be dealt with.
Risk 1: When the evaluation no longer matches the brief
There’s Risk 1: the budget was too low to deliver the full range of results optimistically included in the TOR. The tenderer’s method was slashed to fit the budget, while the language was carefully couched to give the impression that the results will be delivered as per TOR – or close to.
What comes home to roost? The gap between the carefully couched words, the evaluation that can realistically be produced with the money, and the commissioner’s expectations.
In the real world, tenders still often over-promise even when the couching is careful — and commissioners don’t necessarily update their expectations when they read it, still expecting something close to the TOR. That gap becomes a significant problem.
So what happens? Occasionally the tenderer grimly maintains the fiction – delivering the letter of the tender at lowest cost and pretending the result matches the terms of reference. Usually, they negotiate.
And “negotiate” undersells what happens at that table. I’ve watched whole OECD-DAC criteria go – impact and sustainability are usually first against the wall, which should worry anyone who remembers why we evaluate at all. Questions get consolidated until three become one. The survey disappears and a literature review arrives in its place. Outcome Harvesting – useful, and priced accordingly – silently leaves the methodology. The senior leads are deleted from the team, the juniors are promoted into their slots, and the seniors who remain are refocused on “QA and advisory” – a day a month, if that. The deliverables shrink. Sometimes the penalties shrink too, which tells you everyone knows what’s coming. Item by item, the evaluation the ToR described is dismantled and something smaller assembled from the parts – and the inception report signs it all off as the plan.
The team on paper versus the team that turns up
Jonathan: Yes, and it’s where an important gap can open up. I’ve been burned (and seen others burned) by bids that have promised Rolls Royce senior teams, and when the negotiation starts, suddenly a Reliant Robin team gets put up as a replacement. For commissioners, this can feel very close to a “bait and switch” strategy: win the bid with the promise of highly experienced staff, then swap them out for younger, cheaper (and more profitable) folk once the contract is safely signed.
And this is not just an individual experience of mine. The Canadian Office of the Procurement Ombud reviewed exactly this last year (September 2025), in a report it titled – with admirable directness – “Bait and Switch”. In nine of the seventeen contracts examined, some or all of the individuals named in the winning bid never performed any work on the contract. In seven of the seventeen, the replacements were not evaluated as meeting or exceeding the qualifications of the people they replaced – even though the contract required it and those qualifications had won the work. The Ombud’s careful conclusion: those seven contracts may not have been awarded to the best-value supplier.
Two things in the report however made me pause. The first is the timing. Some named individuals were replaced 237 or 311 days after award. That’s life moving at its normal pace: no one can hold a diary open for ten months while government makes up its mind. But others were replaced within 12, 23 and 28 days of signature, which sounds like scheduled departures: nobody’s circumstances change that fast. The second is who the Ombud heard it from: industry itself told the review that the procurement process takes so long that some suppliers propose their best people knowing they will be unavailable by the time work starts. The regulator wrote that down. The suppliers’ own explanation is our post two, in evidence.
This is Canadian federal contracting, and a judgemental sample of seventeen files, so I’m not claiming a global prevalence figure. I don’t need to. I recognised the pattern from my own commissioning years the way you recognise a photograph of your own street.
Mat: While I agree to an extent, I also think it’s worthwhile pointing out that the study’s findings show a fair bit of nuance with such replacements, and the authors don’t conclude that bait-and-switch is always the case. I also wonder why the various departments did not check replacements against the terms of reference. They clearly could have, and indeed should have.
From my experience, no company I’ve worked for used “bait and switch” as a deliberate tactic. The occasional senior CV goes in with availability doubts attached – but rarely for a critical role, and every company I’ve known has sometimes declined to tender rather than bid a Team Leader it couldn’t source. And replacement done honestly looks completely different. One large programme I knew put up an interim team leader for inception and said so in the bid — a systems person, there to build the frameworks and processes, with a planned handover to a national team leader for delivery. The handover came later than the plan said – transitions usually do – but it came, and nobody cried foul, because nobody had been fooled. That’s the whole distinction. Bait-and-switch isn’t the swap; it’s the concealment. Say who’s coming and who’s going before signature, and a replacement becomes what it should always have been: a design choice.
Jonathan: You’re right, and here’s the uncomfortable commissioner-side answer to your question: the controls existed and mostly sat unused. The standard replacement clause in the Canadian system requires substitutes to have merely “similar” qualifications – a commissioner has to actively modify it to “meet or exceed”, and in several of the reviewed files nobody did. The boilerplate ships with the fiction pre-installed, and someone has to notice and switch it off. In other files, no reason for the original person’s unavailability had to be given at all – which makes swapping people rather easy. A clause that names the risk but never enforces it is our old friend from post one: QA as ritual. Naming a risk is not managing it.
But some commissioners did the job properly, and the Ombud names them as models. One simply confirmed availability at award. Another, handed three unfamiliar faces for the first three tasks, quoted the bid certification back at the contractor, tested the claim that a replacement “scored the same”, found it didn’t, and made them produce someone who genuinely did. That’s all it took: reading your own contract and having the nerve to enforce it. The levers were there. Since 2024 the Canadian system has gone further, requiring signed proof of consent from every named individual before they can be proposed: post two’s “firm commitment”, finally in writing – a quiet concession that the provisional yes was never a commitment at all.
Mat and Jonathan: Put our two accounts side by side and you have the whole argument of this series in one place. ‘Similar’ was allowed to mean ‘worse’. The reasons were never asked for. The levers sat unpulled. And still, suppliers sometimes named people they doubted they could deliver. Pick either of our accounts as the sole explanation and you’ve missed it – they aren’t rivals, they’re the same system seen from opposite ends of the table. A process slow enough to make availability a fiction. Contracts that never require the fiction to be explained. Nobody whose job it is to notice. And the Ombud’s report is honest in both directions – in more than half the files reviewed, the replacement of people made no difference to who deserved to win. The report holds both at once. That’s what a systemic problem looks like: not villains on either side, but a game whose rules reward neither side for honesty about who will actually turn up.
Mat: There’s a final twist the Ombud’s framework can’t see. All that machinery – meet or exceed, equivalent scores, replacement grids – polices one thing: fidelity to the criteria that won the work. It assumes the criteria were right. Years ago, we proposed a team leader for an environmental project who met the ToR to a T: the doctorate, the decades of deep research experience, everything the specification demanded. He failed – because the job was never research. It was management, budgets, reporting, and the political dance around a client and donor who both held strong, fixed and wrong ideas about the work. The client insisted his replacement be “of the same calibre”, so we (reluctantly) provided one, and he failed identically. Only at the third attempt did everyone accept that the criteria themselves were the fiction. The team leader who finally made it work would have scored lower than either predecessor against the original ToR. Meeting the specification and being able to do the job are different things — and occasionally they’re opposites.
Risk 2: When financial risk becomes delivery risk
But back to the risk register. Risk 2 is the financial risks that go beyond Risk 1’s “too much work for not enough money”. Look at how evaluation contracts increasingly pay: payment by results; milestone payments released only when the commissioner signs off a deliverable; retention, a slice of the fee held back until final acceptance; and, more and more, KPI penalty clauses, where missing a quality gate or a deadline triggers a deduction. Every one of them is a risk transfer dressed as an incentive.
In many of these cases, the lucky winner of the bid has to dance holding a number of delicately balanced plates: one containing their overhead and margin; one holds the quality of their technical results; another contains the all-important relationship with the client that can so easily see both of the other plates dropped and shattered.
Jonathan: I’ll defend these financial approaches, because I’ve sat on the commissioning side. They exist because commissioners get burned in a number of ways – for example, reports that aren’t the quality they are expected to be. When you can’t watch delivery day to day, retention and payment-by-results are how you protect public money. The answer isn’t to abolish them.
And yet.
I’ve since sat on the other side of the same table, costing a bid where the payment mechanism was punitive enough that a couple of poor quarters would have wiped the margin, and three in a row could have sunk the contract – and the “poor quarter” wasn’t always the supplier’s fault. Access collapses. A partner underperforms. The security picture turns. I helped design mechanisms like that as a commissioner; now I read them as a supplier and wince. Both things are true: they’re necessary, and they can be brutal.
Risk 3: Reputation and the hidden cost of failure
Mat: This segues nicely into Risk 3 – the reputational risks that come with the technical and financial risks above. The saying goes, “You’re only as good as your last contract,” and if you’re unlucky enough to be on a contract where one or more of the delicately balanced plates gets dropped, well, that can have very large effects on reputation.
Jonathan: And reputational risk doesn’t arrive on a level playing field. It lands on top of the “sunk cost” risks already carried in from the earlier stages – and compounds them.
One of those is the strength of the overall market. We’ve already seen that some ways of procurement – such as framework contracts – can have the paradoxical impact of reducing competition, not increasing it.
Pile financial and reputational risk on top and the tilt steepens: deep pockets can absorb a penalty or a held payment; smaller players and individuals, without the cashflow reserves to ride out slow milestones, can’t – and some go under.
Mat: I agree completely. The overall procurement system needs to be cognisant of where the real impact of such risks lands. Primes often have a balance sheet deep enough to ride out a deduction. It’ll hurt them, sure, but it won’t wipe them out. They’ve got revenue flowing from other projects to help ride out the problem. It’s the individual associate on a day rate, final invoice held while the gate is argued over, who is most at risk. Most individual consultants don’t have as wide a range of ongoing projects to help cover the cashflow (or even revenue hole!) of a deduction. So the hardest consequences fall on those with the least shielding.
The tools already exist – so why aren't they used?
Jonathan: It would be easy to end there – three risks, a shrug, and a mutual agreement that the system is not fit for purpose. But here’s the rub: fixing the delivery stage doesn’t need new powers. It mostly needs commissioners to use the ones they already have.
Cox and Barbrook-Johnson interviewed nine commissioners and ten contractors in one UK government department, and their interviewees produced a list of measures available under existing rules: a post-award scoping stage to finalise the methodology once the contract is safe; interim stage gates to review what’s working; KPIs that let deliverables and milestones shift where learning justifies it, rather than registering every shift as failure; a live, shared risk register instead of a risk annex nobody reopens; responsive resourcing within an overall budget envelope; ongoing dialogue in place of intermittent reporting against milestones.
One department only, so read it with that in mind – but I’ve yet to meet the UK evaluation contract these couldn’t apply to. The findings were gathered about complexity-appropriate evaluation, but there’s nothing complexity-specific about a stage gate or a sensible KPI. The mechanisms are procedural so they travel.
So why does the toolbox stay shut? Not for want of knowing how. The development sector spent the 2010s building exactly this muscle under the banner of adaptive management – problem-driven iterative adaptation, Doing Development Differently, donor rulebooks rewritten to license programmes that learn as they go. I commissioned evaluations inside that world. We were funding programmes explicitly designed to change course — and buying their evaluations on contracts explicitly designed to prevent it. The thinking wasn’t missing. It was in the building, one corridor away, and it stopped dead at procurement’s door.
Cox and Barbrook-Johnson’s interviewees explain why, and it’s the part I’d ask every frustrated evaluator to sit with. Adaptive delivery got adopted where someone senior carried the risk of flexibility – a champion, a rulebook, an owner with the authority to say this deviation is learning, not failure. Contract management has no equivalent. Active management costs more commissioner time than the traditional kind, needs line managers who tolerate it, and demands competencies the commissioning profession has never systematically built.
And behind all of it sits procurement’s genuine nightmare: vary a contract too far and you may invalidate the competition that awarded it, forcing a re-tender. When your evaluator proposes the smaller, more honest study at inception and the commissioner says the ToR is the ToR, obstinacy can be the wrong diagnosis. Sometimes it’s a research manager who knows that saying yes could unwind the entire procurement – with no Smart Rules to stand behind, nobody senior whose job it is to call the flex learning rather than failure. The tools sit in the box because opening it is expensive and career-risky for one specific, identifiable person. It’s the same incentive story this series has told from the first post – adaptive management arrived everywhere except the desk where the contract lives.
Designing flexibility into contracts
Mat: I’ve seen this from the other side – the so-called “adaptive programme” that nevertheless ran into problems because the contract couldn’t flex with the technical changes. Technical workstreams could be redesigned, but the Logframe and Theory of Change were locked into the contract and could only be updated through a long and expensive Contract Amendment.
Engineers in earthquake zones know that they have to design flexibility into the structure from the foundations upwards. If you have any joint between a flexible part of the building and a rigid part, THAT’s where the building cracks in an earthquake. So they build flexibility into the whole building from the outset, choosing tolerances, signing off on the designs – and being accountable if things go wrong.
Commissioners of both projects AND evaluations need to do the same thing. There’s no point putting adaptive programmes on top of non-adaptive contracts. They need to build-in that flexibility: establish what distinguishes “learning flexibility” from “programme failure”; who decides; what the tolerances are; how the process gets implemented; and who owns it. Engineers can’t be arguing about design tolerances in the middle of an earthquake, just like programme implementation or evaluation shouldn’t be arguing about how to interpret KPIs as deadlines loom. And whoever owns that call, it can’t be the associate standing in the doorway with an invoice riding on the answer
Where the risk really ends up
Mat and Jonathan: Watch where the risk travels. The commissioner transfers it to the prime through the payment mechanism, the KPIs, the retention. The prime transfers it to the sub-contractor through back-to-back terms and margins trimmed at each tier. The sub-contractor transfers it to the individual consultant — the associate in the doorway, whose invoice is the one held while everyone above argues about whether a gate was met. At every step, risk flows from whoever has the power to move it towards whoever lacks the power to refuse it, and it settles on the person least able to carry it. The commissioner is protecting public money, the prime its balance sheet, the sub-contractor its margin. Each transfer is locally rational. The cascade is the system working exactly as designed — which is the problem.
And the tools to stop it are not missing. That’s what this post keeps finding, from every direction. The clauses existed; ‘similar’ was allowed to mean ‘worse’.
The levers existed; two commissioners pulled them and it worked. The flexible contract mechanisms exist; they sit in a box few are paid to open. The adaptive thinking exists; it stopped at procurement’s door. What’s missing is what the earthquake engineers insisted on: the flex designed in before the ground moves, and a name signed against it. Every mechanism in this post — retention, penalties, replacement clauses, the inception renegotiation itself — is the system correcting at delivery for something that wasn’t said at specification. The correction always costs more than the honesty would have. More money, more time, more goodwill — usually all three, and the bill lands hardest at the bottom of the cascade. The grown-up move isn’t perfection at delivery. It’s honesty before signing, so there is less to correct after it — and tolerances agreed while everyone can still afford to be honest about them.
Delivery-stage mechanisms transfer money and time. Those, at least, can be argued over, invoiced, and eventually paid. But there is one more thing that gets transferred under pressure, and it’s the one nobody writes into the contract: the finding itself. What happens to an inconvenient truth when the person reporting it is the person whose invoice is being held? That’s the next post.
A note on AI from the authors. We used AI tools for some of the drafting of this post, in line with the UK Evaluation Society’s good practice guidance on the use of AI in evaluation. The creative process – the arguments, the disagreements, the sarcastic humour, and the editing – was ours throughout. Ownership of the material and responsibility for the views in it rest with us.
Our thanks to Jonathan Kuhn-Patrick and Mat Pritchard for this guest blog and for contributing their perspective to the UK Evaluation Society’s work.
Join the discussion
Comments are reviewed by a moderator before they’re published.
References
Better Evaluation, ‘Inception report’, betterevaluation.org (accessed July 2026).
Cox, J. and Barbrook-Johnson, P. (2021) ‘How does the commissioning process hinder the uptake of complexity-appropriate evaluation?’, Evaluation, 27(1), pp. 32–56.
HM Treasury (2026) The Magenta Book: Central Government Guidance on Evaluation. London: HM Treasury.
Office of the Procurement Ombud (2025) “Bait and Switch”: Procurement Practice Review of Replacement Resources in Professional Services Contracts. Ottawa: OPO, September.
UNDP (2011) Outcome-Level Evaluation: A Companion Guide to the Handbook on Planning, Monitoring and Evaluating for Development Results. New York: UNDP.
How to cite this article
Kuhn-Patrick, J. and Pritchard, M. (2026) Part 3: Terms of execution Available at: https://evaluation.org.uk/part-3-terms-of-execution/(Accessed: [insert date]).
Jonathan Kuhn-Patrick
Jonathan Kuhn-Patrick is an Independent Evaluation Consultant and international development and evaluation specialist with more than 30 years’ experience spanning humanitarian response, fragile states and complex programme evaluation. His career has included frontline NGO leadership, senior evaluation, policy and operational roles within DFID and FCDO, and extensive work across Africa, the Middle East and South Asia.
Now an independent consultant, Jonathan specialises in evaluability assessments, evaluation quality assurance and monitoring, evaluation and learning (MEL) framework design. His recent work has focused on fragile states, humanitarian response, climate resilience and infrastructure transparency across Asia, Africa, the Middle East and Latin America.
Jonathan led the development of the UK Evaluation Society’s good practice guidance on the use of AI in evaluation and continues to support professional learning in this area. He is also a former member of the UK Evaluation Society’s Board of Trustees.
Mat Pritchard
No comment yet, add your voice below!