lesson
60 min

Deciding With Evidence

Use vetoes and weights in a decision matrix, run a bake-off on your own test set, and record the decision so it can be revisited.

Objective

By the end of this lesson, you can build a decision matrix that separates hard requirements from preferences, plan a small bake-off between options, and write an architecture decision record.

Vetoes and weights

A decision matrix scores each option on factors you care about. It goes wrong in one common way: a dealbreaker gets treated as one factor among many, and a high score elsewhere hides it.

Split your factors in two:

  • Vetoes are requirements. If an option fails one, it is out. Example: "client records may not leave systems approved for them."
  • Weights are preferences. Speed, team fit, and quality trade against each other.
Run it. The hosted API is vetoed even though its total would tie the self-hosted option. Then try two changes: set the data_control weight to 1 and see how the ranking moves; then remove the veto by setting must_have = dict() and notice how a dealbreaker can hide inside a good score.
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

The open model and the blend option are within one point. When scores are that close, the matrix has done its job: it narrowed the field. Now test.

Run a bake-off

A bake-off runs each finalist on the same test set and compares results. Public leaderboards measure general skills. Only your own questions show how a model does for your users.

  1. Collect 50 to 200 real tasks, de-identified, including hard and sensitive ones.
  2. Include each language and reading level your users bring.
  3. Write the expected answer or a grading rule for each task.
  4. Run every finalist with the same prompt and settings.
  5. Score correctness, refusals when it should refuse, speed, and failures by group.
  6. Have a person review a sample of outputs, not only the scores.

Keep the test set. It becomes your regression suite: every future model change runs through it (Module 7).

Write it down: the ADR

An architecture decision record (ADR) is a short document that records one decision. It lets the next person know what you knew and when to look again.

ADR-004: Model access for the intake assistant
Status: Accepted
Context: Client notes include personal information. Staff of two, no
  dedicated ML engineer. Spanish and English required.
Options: (1) hosted API, (2) open-weight model self-hosted,
  (3) managed service + our own retrieval.
Decision: Option 3. Vetoed option 1 on data handling terms.
  Option 2 scored close but needs skills we do not have.
Consequences: We own prompts, retrieval, tests, logs. Vendor
  dependency on the managed service.
Exit plan: Test set + prompts are portable; re-run bake-off on a
  replacement within one month if terms or quality change.
Revisit: Every 6 months, or when the provider retires the model.

Before you accept a build / buy / blend decision

0 of 7 done

Go deeper: Own the code: building internal tools over rented software

Ask. Answer. Build together.