Most AI vendors can give a compelling demonstration. Far fewer can show a system running in production, measured against a written standard, operated through change, and handed over cleanly. These twenty questions are designed to surface that difference early, using evidence rather than presentations.
Why vendor selection deserves a structured question set
Choosing the right partner has a measurable effect on outcomes. Research from MIT's NANDA initiative reported that AI deployments purchased from specialized vendors and built through partnerships succeeded about 67% of the time, while internal builds succeeded about one-third as often.1 The same research found that most generative AI pilots were not yet producing measurable revenue acceleration, which makes the choice of partner a material decision.
The market also makes that choice harder. Gartner estimated that of the thousands of vendors claiming agentic AI capabilities, only about 130 offered genuine agentic products, describing the relabeling of existing chatbots and automation tools as "agent washing."2 Gartner separately predicted that over 40% of agentic AI projects will be canceled by the end of 2027 due to escalating costs, unclear business value, or inadequate risk controls.2
Public-sector procurement guidance offers a sound foundation. The UK government's Guidelines for AI procurement recommend focusing on the challenge rather than a specific solution, conducting a data assessment before procurement, avoiding black-box algorithms and vendor lock-in, and planning for lifecycle management, including testing, knowledge transfer, and end of life.3 The World Economic Forum's AI Procurement in a Box toolkit, developed with governments, provides complementary guidance for responsible AI acquisition.4 The questions below translate that guidance into a practical evaluation of private-sector delivery partners.
Ask for artifacts, not assurances. A vendor that delivers can show an evaluation report, a runbook, a monitoring dashboard, and a handover plan. A vendor that only presents will describe them.
How to use the question set
- Send the questions in writing before the first substantive meeting, and request supporting artifacts where indicated.
- Score each answer from 0 to 3 using the scoring sheet at the end of this page.
- Involve a multidisciplinary panel, as the UK guidelines recommend: the business owner, a technical lead, security, legal or commercial, and the team that will operate the system.3
- Verify the strongest answers through reference calls and a short paid discovery or evaluation sprint.
Group A: Delivery and evidence
1. Show us a system you built that is in production today. What does it do, and what did it change?
Why it matters: Production is where AI value is realized and where most difficulty sits. Pilot experience alone does not demonstrate delivery capability.
Strong answer: A specific workflow, volume handled, time in production, before-and-after metrics with their measurement method, and a reference contact who operates the system.
Red flags: Only demonstrations or proofs of concept; metrics without baselines; references limited to executive sponsors who do not use the system.
2. Who will do the work, and where will they sit?
Why it matters: Outcomes depend on the engineers who learn the workflow. MIT's research attributed most pilot shortfalls to systems that do not adapt to real workflows.1
Strong answer: Named engineers and a delivery lead, their relevant experience, and a plan to work alongside your operators in the workflow.
Red flags: Senior people in the sales process who are not on the delivery team; an unnamed offshore team; no time planned with end users.
3. How do you scope a mission, and what does the scope document contain?
Why it matters: Unclear business value is one of the leading reasons Gartner cites for projects abandoned after proof of concept.5
Strong answer: A one-page scope with workflow boundaries, measured baselines, a definition of done, an accuracy bar, evidence gates, an exception path, and named owners on both sides.
Red flags: Scope defined by features or hours; success defined as "deployment" rather than a business outcome.
4. What evidence will we see at each milestone, and what happens if it is not there?
Why it matters: Evidence gates let you stop or re-scope early at low cost instead of discovering problems at launch.
Strong answer: Defined gates (baseline, offline evaluation, shadow mode, limited production) with explicit proceed, re-scope, or stop decisions, and commercial terms that respect those decisions.
Red flags: A single acceptance event at the end; reluctance to define stop conditions.
Group B: Evaluation practice
5. How will you define "good enough" before you build?
Why it matters: A threshold set after results arrive tends to move to match them. Model providers' evaluation guidance explicitly warns against "vibe-based" evaluation.6
Strong answer: A written accuracy bar with a primary metric, severity tiers, sample size, confidence interval method, and a comparison to the human baseline, signed before build.
Red flags: "We will iterate until you are happy"; a single headline accuracy figure with no sample size.
6. May we see a real evaluation report from a past engagement, appropriately redacted?
Why it matters: The quality of the evaluation artifact is the best predictor of the quality of the evaluation practice.
Strong answer: A report showing dataset construction, labeling method, results by segment and severity, error analysis, and known limitations.
Red flags: Public benchmark scores offered in place of task-specific results; no error analysis.
7. How do you test for adversarial and security failures?
Why it matters: Generative systems introduce new attack surfaces catalogued in the OWASP Top 10 for LLM Applications, including prompt injection, sensitive information disclosure, and excessive agency.7
Strong answer: Structured red teaming against a documented harms list, followed by systematic measurement, consistent with Microsoft's guidance that red teaming identifies risks but does not measure their prevalence.8 Ideally, testing is independent of the build team.
Red flags: Reliance on the model provider's safety filters alone; no testing of tool-calling or data-access paths.
8. If you use a model to grade outputs, how do you know the grader is right?
Why it matters: Model-based grading scales well but can be miscalibrated. OpenAI lists failing to calibrate automated metrics against human evaluation as an anti-pattern.6
Strong answer: A stated agreement rate between the model grader and expert human graders on a held-out set, with periodic recalibration.
Red flags: No human calibration; the same model grading its own outputs without checks.
Group C: Security and data
9. Where will our data go, and who can access it?
Why it matters: Data handling determines regulatory exposure and trust. The UK guidelines call for a data assessment and clear governance of consent, storage, and access.3
Strong answer: A data flow diagram covering every processor and model provider, residency, retention, access controls, and whether data is used for training.
Red flags: Uncertainty about sub-processors; default training on customer data; no data flow documentation.
10. Which AI governance and security frameworks does your practice follow?
Why it matters: Recognized frameworks make controls comparable and auditable.
Strong answer: Specific mapping to frameworks such as the NIST AI RMF and its Generative AI Profile, ISO/IEC 42001, and Google's Secure AI Framework, with examples of the controls in practice.91011
Red flags: Framework names without implemented controls; certifications that cover only unrelated parts of the business.
11. How do you limit what an AI system or agent can do?
Why it matters: Excessive agency, granting systems more permissions or autonomy than needed, is a recognized LLM risk category.7
Strong answer: Least-privilege credentials, allow-listed tools, human approval for high-impact actions, rate and cost limits, and full action logging.
Red flags: Agents using broad service accounts; no audit trail of actions taken.
12. How do people oversee and override the system?
Why it matters: Human oversight is a requirement for high-risk systems under the EU AI Act, including the ability to override outputs and stop the system.12
Strong answer: A designed exception path with routing rules, a review queue, and a tested stop mechanism.
Red flags: Oversight described only as "users can report issues."
Group D: Operations and drift
13. Who operates the system after launch, and what does that include?
Why it matters: AI systems change as data, users, and underlying models change. Gartner cites escalating costs and unclear value as drivers of cancellations.2
Strong answer: A defined operating model with monitoring, incident response, scheduled re-evaluation, and named responsibilities, offered either as a managed service or as a structured handover.
Red flags: Support limited to break-fix; no post-launch plan.
14. How will you detect performance degradation?
Why it matters: The EU AI Act's post-market monitoring provision calls for active, systematic collection and analysis of performance data across a system's lifetime.13
Strong answer: Production sampling against the accuracy bar, drift signals on inputs and outputs, exception-rate tracking, and defined thresholds that trigger review.
Red flags: Monitoring limited to uptime and latency.
15. What happens when the underlying model provider changes or retires a model?
Why it matters: Model updates can change behavior without any change to your code.
Strong answer: Version pinning, a regression evaluation suite run before any model change, and a documented migration process.
Red flags: Automatic adoption of new model versions without re-evaluation.
16. How do you manage cost per transaction in production?
Why it matters: Gartner's analysis of abandoned projects highlights costs that are less predictable than with other technologies.5
Strong answer: A unit-cost model per transaction, cost monitoring, consumption limits, and options for routing simpler cases to lower-cost models.
Red flags: No unit economics; cost estimates based only on pilot volume.
Group E: Commercial terms and exit
17. How is your fee linked to outcomes?
Why it matters: Commercial structure shapes behavior across the engagement.
Strong answer: Milestone payments tied to evidence gates, with clear terms for re-scoping or stopping.
Red flags: Large upfront commitments before any evidence; multi-year licenses required to start.
18. What do we own at the end?
Why it matters: The UK guidelines recommend avoiding vendor lock-in and considering open standards and appropriate licensing.3
Strong answer: Clear ownership of code, prompts, configurations, evaluation datasets, labeled data, and documentation.
Red flags: Evaluation sets or prompts treated as vendor property; proprietary formats with no export path.
19. If we part ways, how does handover work?
Why it matters: Lifecycle management, including knowledge transfer and end of life, is one of the UK guidelines' top considerations.3
Strong answer: A documented exit plan with a runbook, architecture documentation, training for your team, and a defined transition period.
Red flags: No exit plan; transition support only at undefined additional cost.
20. Which engagements have you declined or stopped, and why?
Why it matters: A partner who will say no protects your budget and credibility.
Strong answer: Specific examples of re-scoping or stopping when the evidence did not support proceeding, with what was learned.
Red flags: No examples; every use case presented as a fit.
The scoring sheet
| Group | Questions | Weight | Score (0 to 3 each) | Artifacts received | Notes |
|---|---|---|---|---|---|
| A. Delivery and evidence | 1 to 4 | 25% | ___ / 12 | Production reference, sample scope document | |
| B. Evaluation practice | 5 to 8 | 25% | ___ / 12 | Redacted evaluation report, red-team summary | |
| C. Security and data | 9 to 12 | 20% | ___ / 12 | Data flow diagram, control mapping | |
| D. Operations and drift | 13 to 16 | 15% | ___ / 12 | Sample runbook, monitoring dashboard | |
| E. Commercial and exit | 17 to 20 | 15% | ___ / 12 | Draft terms, exit plan |
- Scoring guide
- 0 = no answer or assurance only; 1 = described but not evidenced; 2 = evidenced with an artifact; 3 = evidenced and verified by a reference or test.
- Weighted total
- Sum of (group score / 12 × weight) = ____%
- Gating rule
- Any score of 0 in Groups B or C requires resolution before contract.
- Panel
- Business owner: ____ Technical lead: ____ Security: ____ Commercial or legal: ____ Operations: ____
Weight evidence over polish. A vendor that scores 2 or 3 across evaluation and security, with artifacts to show for it, is likely to deliver in production.
Sources
- MIT report: 95% of generative AI pilots at companies are failing — Fortune (reporting on MIT NANDA, "The GenAI Divide"), 2025
- Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027 — Gartner, 2025
- Guidelines for AI procurement — UK Government Office for Artificial Intelligence, 2020
- AI Procurement in a Box — World Economic Forum, 2020
- Gartner Predicts 30% of Generative AI Projects Will Be Abandoned After Proof of Concept By End of 2025 — Gartner, 2024
- Evaluation best practices — OpenAI, 2025
- OWASP Top 10 for LLM Applications 2025 — OWASP Gen AI Security Project, 2024
- Planning red teaming for large language models (LLMs) and their applications — Microsoft, 2025
- Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1) — National Institute of Standards and Technology, 2024
- ISO/IEC 42001:2023 Artificial intelligence — Management system — International Organization for Standardization, 2023
- Secure AI Framework (SAIF) — Google, 2023
- EU AI Act, Article 14: Human Oversight — Regulation (EU) 2024/1689, 2024
- EU AI Act, Article 72: Post-Market Monitoring — Regulation (EU) 2024/1689, 2024
Want a second pair of eyes on your draft?
Bring your completed template to a thirty-minute working session and we will review it with you, free of charge.
Book a working session →