Leadership · lesson 2 of 4 · 15 min
Select and evaluate models
Compare models and vendors using task-specific evidence rather than general benchmarks or reputation.
- Read a public benchmark skeptically
- Run a task-specific bake-off
- Ask vendors the questions that bind
Public benchmarks do not predict your task
Leaderboard scores measure performance on standardized academic tasks that resemble nobody's actual work. They are also subject to contamination, selective reporting, and tuning to the test. Use them to build a shortlist of three, never to make the decision. The only evidence that transfers is performance on your inputs, judged by your rubric.
Change in a deployed system's behavior over time, caused by vendor model updates, changing inputs, or a shifting operating environment, detectable only through repeated evaluation against a fixed set.
A finance team pinned a model version for a reconciliation workflow and ran the fixed set monthly. Four months in, the pinned version was deprecated with 30 days notice. Because the set existed, the migration was a two-day scoring exercise instead of a quarter of firefighting.
Run the bake-off on your own set
Take the fixed evaluation set from the intermediate path, run all three candidates blind, and have two reviewers score independently against the rubric. Record accuracy, the unacceptable-failure rate, latency, cost per task at your real volume, and the variance across runs. Variance is the metric teams forget and the one that generates support tickets.
The contract is a control
Ask, and get it in writing: is our data used for training; what is the retention period and can it be set to zero; where is data processed and stored; what is the subprocessor list and how are we notified of changes; what is the notice period for model version changes; can we pin a version; what is the incident notification SLA; what happens to our data at termination. A vendor who will not answer these in a contract has answered them.
Plan the exit before you enter
Model dependency is a concentration risk. Before deployment, know what breaks if the vendor doubles the price, deprecates the version, changes behavior overnight, or suffers an outage on your busiest day. Keep prompts and evaluation sets in your own repository, abstract the vendor behind an interface, and confirm that a second model can run your set at an acceptable score. Portability is cheap to design in and expensive to retrofit.
Write the eight vendor questions
Take the eight contract questions above and send them to a vendor you currently use. Record which ones they will put in writing.
The answers you cannot get in writing are your actual risk register entries.
0 of 5 practice steps checked. Saved on this device.
Vendor answered six of eight contractually. Refused: a notice period for model version changes, and a subprocessor change notification. Both became register entries. Mitigation: pin the version where the API allows it, run the evaluation set weekly rather than monthly on the workflows that cannot tolerate silent change.
Check your understanding
Answer both to mark the lesson complete. You can try again. This is practice, not a graded exam.
1. A model tops the public leaderboards. What does that justify?
2. Which measure is most often omitted from a model comparison and most often causes production complaints?
Three things to remember
- Benchmarks shortlist; your evaluation set decides.
- Score accuracy, unacceptable-failure rate, cost at real volume, and variance.
- Put data use, retention, version notice, and exit in the contract.

The full oversight framework behind this path, across identity, data, AI, vendors, and risk.
Examples are composites written for teaching unless a source is cited. Figures come from the sources listed where given.