In this timely conversation, explore what commissioners and consultants actually need to be talking about when it comes to AI in evaluation.
Cormac Quinn is an evaluation advisor with the UK government and Jonathan Kuhn-Patrick is an evaluation consultant and is the Society’s AI lead. Both are UK Evaluation Society Trustees.
In Goethe’s poem The Sorcerer’s Apprentice, a young apprentice enchants a broom to fetch water, marvelling at the efficiency, until the broom won’t stop, flooding the room because the apprentice lacks the knowledge to control it.
Today, commissioners often feel like the apprentice, aware of AI’s speed and efficiency but uncertain how to maintain control. Consultants find themselves as the enchanted broom, more powerful than before but still needing to convince nervous commissioners they won’t flood the room.
The result is a relationship that is trying to navigate new and uncharted waters with poor communication between key sections of the crew. While many organisations (including UKES) have published principles and the UK government have developed an AI in evaluation checklist, these provide frameworks rather than answers to the specific, awkward questions that arise in evaluation procurement and implementation.
Here’s what that awkwardness actually looks like from both sides.
On tender quality: Who wrote this proposal?
Cormac (commissioner): I am increasingly worried that as AI develops, proposals will tend toward becoming technically flawless with perfect structures and polished language. This begs the question, will I be evaluating the consultant’s capability or that of their corporate AI?
The key risk for me is that bids will progressively appear to be methodology robust on paper, yet evaluation implementation could reveal consultants who struggle to articulate and take forward agreed evaluation approaches in practice. In other words, we could end up appointing consultants who lack necessary technical expertise but do have excellent AI prompt engineering skills.
This leads me to ask: should we place greater weight on team presentations and real-time methodological defence in the tender process, rather than relying primarily on written proposals?
Jonathan (consultant): Your concern about the paper-practice gap is legitimate. I’ve seen consultants with impressive proposals struggle in implementation myself. But here’s my question: what exactly are you hiring me to do?
If you’re commissioning professional work, the evaluation question should be whether my outputs are reliable and methodologically sound—not whether they emerged from some romantically pure creative process. This isn’t a university examination assessing my unaided abilities.
Think about how we handled previous technological transitions. When statistical software replaced hand calculations, we didn’t insist consultants prove they could compute regression coefficients manually. When word processors replaced typewriters, we didn’t test typing speed. We focused on whether the analysis was sound and the outputs reliable. AI represents a similar transition. You probably don’t care whether I use SPSS or STATA, Word or LaTeX—you care whether my evaluation is methodologically robust and well-executed.
The paper-practice gap you’re worried about isn’t new, and it’s not caused by AI. I’ve reviewed plenty of pre-AI proposals with flawless methodology descriptions from consultants who couldn’t deliver. The solution you’re suggesting—weighting team presentations and real-time methodological defence more heavily—is exactly right. These assess professional capability regardless of tool use. Ask candidates to defend choices without preparation time. Present scenarios requiring methodological judgment. Request samples of previous work with client verification. These reveal whether someone understands evaluation or just understands AI prompts.
So yes, adjust procurement away from written elegance toward demonstrated competence. But frame it as assessing implementation capability, not policing tool use.
On disclosure and pricing: The transparency penalty
Cormac (commissioner): In order to fully understand risks, it is important to have full transparency in terms of which AI applications will be used in evaluations for which evaluation elements (including their known limitations and mitigation strategies). It is also important to be notified of any AI changes during the evaluation, due to the speed of its development.
I am also conscious that AI delivers significant cost and time savings and need to understand how this impacts on cost. For example, a literature review that did take two weeks may now be completed in three days with AI support. How do I fairly compare bids when some consultants are fully transparent about AI use and efficiency gains while others offer partially information on use?
Jonathan (consultant): Here’s my problem: transparency creates competitive disadvantage when Terms of Reference (ToR) are silent about AI.
If I disclose that I’ll use AI for literature reviews and thematic coding, I’m also revealing that my pricing reflects efficiency gains. When AI reduces a two-week literature review to three days of human-verified work, what should I charge? If I price for three days, I look suspiciously cheap. If I price for two weeks to match market rates, I’m being dishonest about my actual costs.
Meanwhile, competitors who don’t mention AI can price as if everything requires traditional manual effort—even if they’re using exactly the same tools. They simply stay silent because you haven’t asked. So my transparent proposal gets questioned for “lacking human input” while silent competitors presumably use similar tools without comment. The honest consultant gets penalised; the opaque one gets rewarded.
Most ToR make no mention of AI.
That silence forces me to choose: volunteer information that might disadvantage my bid? Or stay quiet like my competitors, risking conflict if you discover AI use later?
Specifically, I should disclose when AI is used for:
- Data collection and analysis – interviews, surveys, coding, thematic analysis, statistical work. This is core evaluation methodology where you need to understand my approach and verify quality.
- Literature reviews and evidence synthesis – when AI is systematically searching, filtering, or synthesizing sources. You need to know my verification protocols here.
- Client data processing – anything involving your sensitive data, especially where safeguarding applies. You need to know what systems handle your information.
I shouldn’t have to disclose every instance of AI for:
- General drafting support – grammar checking, restructuring prose, formatting tables. This is equivalent to spell-check and adds no methodological risk.
- Administrative tasks – scheduling, file organization, basic correspondence. Not relevant to evaluation quality.
- Standard professional tools – if I’m using widely-adopted AI features in Excel, Word, or other standard software, this isn’t meaningfully different from using those tools’ existing features.
The practical test
Would this AI use require different verification protocols, create additional risk to your data, or materially change my methodological approach? If yes, disclose it and explain my mitigation. If no, it’s implementation detail.
For pricing, I propose transparency about efficiency gains paired with honest effort-based pricing: “Literature review: AI-assisted systematic search and initial coding (40 hours), senior evaluator verification and relevance assessment (24 hours), total £X.” This shows you where AI created efficiency, where human expertise remained essential, and lets you compare bids fairly.
But this only works if ToR specify disclosure requirements. Otherwise we’re back to the prisoner’s dilemma where silence wins.
On quality assurance: Different kinds of mistakes
Cormac (commissioner): With the advent of AI, new quality assurance systems are now needed for evaluation processes and outputs. This could include validation processes of AI generated data, human review protocols, audit trails. In addition, I am very aware of the need for AI to also comply with existing safeguarding (especially for vulnerable populations) and data management (especially for sensitive data) requirements.
Based on this, I am mindful that consultants using and overseeing AI will need different skills from what was required previously.
Jonathan (consultant): Agreed on all counts—but let’s be specific about what’s genuinely new versus what’s newly urgent.
Quality assurance for AI-generated outputs does require explicit protocols, but consultants are not infallible with or without AI. We make mistakes in coding interviews, miss relevant literature, introduce bias. These are human errors predating AI by decades. AI helps me make fewer of these mistakes through systematic cross-checking and pattern identification. But I remain accountable for every output—the tool’s limitations don’t absolve me, they increase my obligation to verify rigorously.
Your concerns about safeguarding and data management are critical, and here’s where AI genuinely changes the risk profile. When I process sensitive data or work with vulnerable populations, I need to answer:
- Which AI systems will handle this data, and where does that data go? (Cloud processing, model training, data retention)
- How do I ensure AI-generated interview summaries preserve anonymity and don’t introduce identifying details through synthesis?
- What happens if an AI transcription service processes interviews with abuse survivors—where’s that audio stored, who has access, what’s the retention policy?
These aren’t theoretical questions. We’d need to reject client-provided AI platforms if their data handling didn’t meet safeguarding standards. This requires different skills: understanding data processing agreements, assessing vendor security, documenting data flows, implementing access controls. Traditional evaluation training didn’t cover these, and not every consultant has caught up.
Frame verification around output quality rather than input purity. Ask how I verify AI-generated coding schemes against underlying data. Ask about audit trails showing methodological decisions. Ask which systems process sensitive data and how I’ve assessed their compliance. These are legitimate quality questions.
"...don't ask me to prove my work is "sufficiently human." The question is whether it's sufficiently accurate, methodologically sound, and safely managed—regardless of tool use."
But don’t ask me to prove my work is “sufficiently human.” The question is whether it’s sufficiently accurate, methodologically sound, and safely managed—regardless of tool use.
Cormac (commissioner): My main issue here is that AI can produce mistakes at scale. A human making coding mistakes produces inconsistent results that is far easier to spot, while an AI making systematic errors can produce consistent but fundamentally flawed analysis. That’s qualitatively different.
Jonathan (consultant): Exactly right—which is precisely why verification protocols must be systematic, not spot-check. If I use AI for thematic coding, I don’t just sample-check a few codes. I verify:
- Do the themes actually appear in the underlying data, or has AI invented patterns?
- Are quotes accurately attributed and in context?
- Has AI missed important themes because they don’t fit expected patterns?
- Are there systematic biases in what AI highlights versus what it misses?
This is more rigorous than many traditional coding approaches, where a single analyst’s interpretation might go unquestioned. The risk you’ve identified—systematic, consistent, confidently wrong analysis—requires systematic, thorough verification. That’s a failure of human oversight if I submit unverified work, not a failure of AI.
The risk is inadequate quality assurance, whether that’s failing to catch human error or AI error. My professional obligation is identical either way: don’t submit work I haven’t verified to meet professional standards.
Summary & suggestions
In the above conversation, we have identified three key tensions:
- How to assess genuine capability versus AI tool proficiency?
- How to balance transparency with competitive fairness?
- How to ensure adequate quality assurance of AI supported work?
These are not theoretical concerns, they’re playing out in procurements right now.
Where we go from here
Despite these tensions, we share the same fundamental concerns that evaluations are methodologically sound, ethically conducted, and deliver reliable findings.
The awkwardness emerges because procurement frameworks and oversight of the evaluation cycle is trying to catch up with practice shifting at breakneck speed. Commissioners write ToRs based on current understanding while consultants have to adapt to AI development as they take forward evaluation.
Perhaps the path forward isn’t better rules but better conversations. Move beyond “should we allow AI?” toward “which AI applications genuinely improve which evaluation tasks, and what quality contract approaches ensure quality?” Commissioners need comfort with uncertainty about tools while remaining rigorous about outputs. Consultants need to explain methods rather than defend tool choices.
For Commissioners:
Specify output quality requirements rather than input restrictions. Instead of “consultants must not use AI without permission,” specify “all AI-generated analysis must be verified against source data with quality assurance approaches documented.” Instead of trying to assess whether proposals are AI-written, conduct technical interviews where candidates defend methodological choices. Define adequate standards rather than policing AI tool use. Create space for consultants to flag ongoing AI concerns and allow flexibility for them to adapt as AI evolves.
For Consultants:
Default to transparency even when ToR are silent. Rather than simply stating “we will use AI for efficiency,” explain specifically: “We’ll use AI for initial literature scanning, with all sources verified against originals and relevance decisions made by senior evaluators.” Price actual effort while being honest about efficiency gains. Develop quality assurance processes robust enough to catch both human and AI errors.
The apprentice didn’t intend to flood the room, and the broom was only following instructions. The solution isn’t to ban magic, it’s to develop better understanding of how it works and clearer communication about risks.
Tell us what you need by registering your interest
We are developing a range of resources and training to support practitioners in applying the Good Practice Guidelines, and we want to ensure what we produce meets the needs of the evaluation community.
Registering your interest helps us gauge demand and plan delivery — it does not commit you to anything, and we will not share your details with third parties.
What challenges are you facing? What solutions have you found? Share your experiences through the UK Evaluation LinkedIn group or contact us directly. Let’s continue this conversation!