Deciding With Evidence
Use vetoes and weights in a decision matrix, run a bake-off on your own test set, and record the decision so it can be revisited.
Objective
By the end of this lesson, you can build a decision matrix that separates hard requirements from preferences, plan a small bake-off between options, and write an architecture decision record.
Vetoes and weights
A decision matrix scores each option on factors you care about. It goes wrong in one common way: a dealbreaker gets treated as one factor among many, and a high score elsewhere hides it.
Split your factors in two:
- Vetoes are requirements. If an option fails one, it is out. Example: "client records may not leave systems approved for them."
- Weights are preferences. Speed, team fit, and quality trade against each other.
The open model and the blend option are within one point. When scores are that close, the matrix has done its job: it narrowed the field. Now test.
Run a bake-off
A bake-off runs each finalist on the same test set and compares results. Public leaderboards measure general skills. Only your own questions show how a model does for your users.
- Collect 50 to 200 real tasks, de-identified, including hard and sensitive ones.
- Include each language and reading level your users bring.
- Write the expected answer or a grading rule for each task.
- Run every finalist with the same prompt and settings.
- Score correctness, refusals when it should refuse, speed, and failures by group.
- Have a person review a sample of outputs, not only the scores.
Keep the test set. It becomes your regression suite: every future model change runs through it (Module 7).
Write it down: the ADR
An architecture decision record (ADR) is a short document that records one decision. It lets the next person know what you knew and when to look again.
ADR-004: Model access for the intake assistant
Status: Accepted
Context: Client notes include personal information. Staff of two, no
dedicated ML engineer. Spanish and English required.
Options: (1) hosted API, (2) open-weight model self-hosted,
(3) managed service + our own retrieval.
Decision: Option 3. Vetoed option 1 on data handling terms.
Option 2 scored close but needs skills we do not have.
Consequences: We own prompts, retrieval, tests, logs. Vendor
dependency on the managed service.
Exit plan: Test set + prompts are portable; re-run bake-off on a
replacement within one month if terms or quality change.
Revisit: Every 6 months, or when the provider retires the model.
Before you accept a build / buy / blend decision
0 of 7 doneGo deeper: Own the code: building internal tools over rented software
Discussion
Ask. Answer. Build together.
Loading…