lesson
90 min

Building the Package

Write the architecture document, risk register, and evaluation plan, using templates that tie every module of the course together.

Objective

By the end of this lesson, you can produce the three capstone deliverables using a shared structure, and check them against the rubric before you present.

1. The architecture document

Use this outline. Each section points back to the module that taught it.

1. Problem statement and go / no-go result           (Module 1)
2. Non-AI baseline and why AI is or is not added      (Module 1)
3. Build / buy / blend ADR with exit plan             (Module 2)
4. Component diagram: channels, app, model(s),
   retrieval, tools, approval queue, logs              (Modules 3-5)
5. Prompt, output schema, and tool contracts          (Module 3)
6. RAG design: sources, chunking, index, floor,
   citations, retrieval metrics                        (Module 4)
7. Workflow vs agent choice; approval gates; budgets  (Module 5)
8. Data flow map: consent, provenance, redaction,
   retention, minors                                   (Module 6)
9. Security: injection, exfiltration, secrets, scopes (Module 8)
10. Reliability: caching, routing, retries, fallbacks (Module 9)
11. Deployment, monitoring, kill switch, runbook       (Module 10)
12. Open questions and assumptions                     (all)

Write why for every component. "We use a vector store" is a fact. "We use the existing PostgreSQL database with a vector extension because the corpus is a few hundred documents and we already secure and back it up" is a decision.

2. The risk register

A risk register lists what could go wrong, how likely and how bad it is, who owns it, and what you will do about it. Score likelihood and impact from 1 (low) to 3 (high), and multiply.

FieldExample
RiskAssistant gives wrong eligibility information
Likelihood (1-3)3
Impact (1-3)3
Score9
OwnerProgram lead
MitigationAssistant never answers eligibility; routes to staff; test cases on never-regress list
SignalEligibility-topic questions answered without a handoff in logs
Run it. Two risks have no owner. Assign each one to a role. Then add three risks of your own from Modules 6 to 10, such as a stale cache, an unapproved agent action, or a failed kill switch. Re-run and read the sorted list. Does the order match your gut? If not, which is wrong: your gut or your scores?
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

The first run downloads Python into your browser (several MB).

Cover at least these categories: accuracy, privacy, security, fairness across languages and groups, safety in crisis situations, reliability, vendor dependency, and staff workload. A risk with no owner is not managed.

3. The evaluation plan

Test set:     size, sources (de-identified), case types (common, edge,
              harmful, out of scope), languages, who labeled it
Metrics:      answer correctness, refusal accuracy, retrieval recall@k,
              citation validity, latency p95, all split by language
Launch bar:   e.g. common cases >= target; harmful cases = 100% refused
Never-regress: list of critical cases; one failure blocks release
Judge:        criteria graded by LLM judge, calibration plan vs people,
              or "not used" with reason
Red team:     scope, harms, participants (incl. community members), dates
Stop criteria: measured conditions that trigger the kill switch or
              rollback, written before launch
Review cadence: weekly sample review, monthly test run, owner for each

Set the actual target numbers yourself, based on the harm of a wrong answer. Do not borrow numbers from other projects. Label any number you assume.

Final check before you present

Rubric self-check

0 of 11 done

Your hardest call

Private. Saved only in this browser. No one else can see it.

Go deeper: Software Development, Simply: How to Build the Right Thing, Faster, With Fewer Headaches

Ask. Answer. Build together.