Resources and guides.
Research summaries, step-by-step guides and templates you can use with any vendor. Free to read, share and adapt.
Research summaries.
Evidence-based analysis of how AI systems perform and fail in production, and how expertise is verified.
The Multilingual Agent Benchmark: What LLM Systems Lose Outside English
Published benchmarks show LLM accuracy, safety, and cost shift sharply outside English. Here is the evidence and how to measure it before you deploy.
Read the research →A Failure Taxonomy for Production Agents
A structured taxonomy of how LLM agents fail in production, drawn from OWASP, NIST, MITRE, Microsoft and academic research, with detection and severity guidance.
Read the research →What Live Vetting Catches
AI cheating tools and fake candidates have weakened remote hiring. The evidence on what still works, and a six-gate vetting process for experts.
Read the research →Step-by-step guides.
Practical guides for deploying AI, testing it and moving it past the pilot stage.
The FDE Mission Playbook: Scoping, Running and Landing Forward-Deployed AI Work
A step-by-step guide to forward-deployed engineering missions: team shape, week-by-week plan, evidence gates, RACI, risks, handover and metrics.
Read the playbook →The Red-Team Engagement Playbook: Independent Adversarial Testing for LLM Apps and Agents
How to scope, run and report an independent red-team test of an LLM application or agent: threat models, attack categories, severity and retesting.
Read the playbook →From Pilot Purgatory to Production: A Step-by-Step Route for AI Pilots
A practical route for the AI pilot that has not shipped: diagnose the stall, pick the workflow, set the accuracy bar, stage the rollout and operate it.
Read the playbook →Ready to use, with any vendor.
Templates we use in our own work, with worked examples.
The Mission Brief: A One-Page Scope Template for Production AI
The one-page scope behind every forward-deployed AI mission: fields, rationale, a fill-in template, and an illustrative worked example.
Use the template →The Written Accuracy Bar: Defining "Good Enough" Before You Build AI
A framework and fill-in template for setting AI accuracy thresholds: metrics by task, severity tiers, sample sizes, confidence intervals, and sign-off.
Use the template →The AI Vendor Question Set: 20 Questions That Reveal Delivery Capability
Twenty questions for AI vendor selection across delivery, evaluation, security, operations, and exit, with strong answers and red flags for each.
Use the template →Clear trade-offs, including ours.
Side-by-side comparisons of the ways to buy AI delivery, with sources for every figure.
Two practical checklists.
Copy them, adapt them and share them with your team.
The production readiness bar
- A written accuracy bar exists: a number, agreed before testing, that defines "good enough".
- The evaluation set was built adversarially by someone aiming to find failures, beyond happy-path demos.
- Domain experts evaluated the outputs, beyond the team that built the system.
- Failure modes are documented with reproduction steps and ranked by severity.
- Every exception routes to an accountable person, and that route has been tested end to end.
- Rollback exists and has been rehearsed.
- An accountable reviewer approved the verdict, on the record.
The vetting questions that matter
- Was expertise assessed in depth? Take-home and recorded challenges can now be solved by AI.
- Who ran it? A practitioner in the same discipline can probe depth, beyond recall.
- Was identity verified? Knowing who you are working with is part of the evidence.
- Is there a written assessment you can keep? A documented verdict can be checked and shared.
- What happens if the match does not work? Look for a written replacement guarantee.
- Does performance feed back into future matching? Quality should improve with every engagement.
New research and playbooks, as we publish them.
Follow Quantabase on LinkedIn and X, or book a working session to apply any of these resources to your own system.