Building the Package
Write the architecture document, risk register, and evaluation plan, using templates that tie every module of the course together.
Objective
By the end of this lesson, you can produce the three capstone deliverables using a shared structure, and check them against the rubric before you present.
1. The architecture document
Use this outline. Each section points back to the module that taught it.
1. Problem statement and go / no-go result (Module 1)
2. Non-AI baseline and why AI is or is not added (Module 1)
3. Build / buy / blend ADR with exit plan (Module 2)
4. Component diagram: channels, app, model(s),
retrieval, tools, approval queue, logs (Modules 3-5)
5. Prompt, output schema, and tool contracts (Module 3)
6. RAG design: sources, chunking, index, floor,
citations, retrieval metrics (Module 4)
7. Workflow vs agent choice; approval gates; budgets (Module 5)
8. Data flow map: consent, provenance, redaction,
retention, minors (Module 6)
9. Security: injection, exfiltration, secrets, scopes (Module 8)
10. Reliability: caching, routing, retries, fallbacks (Module 9)
11. Deployment, monitoring, kill switch, runbook (Module 10)
12. Open questions and assumptions (all)
Write why for every component. "We use a vector store" is a fact. "We use the existing PostgreSQL database with a vector extension because the corpus is a few hundred documents and we already secure and back it up" is a decision.
2. The risk register
A risk register lists what could go wrong, how likely and how bad it is, who owns it, and what you will do about it. Score likelihood and impact from 1 (low) to 3 (high), and multiply.
| Field | Example |
|---|---|
| Risk | Assistant gives wrong eligibility information |
| Likelihood (1-3) | 3 |
| Impact (1-3) | 3 |
| Score | 9 |
| Owner | Program lead |
| Mitigation | Assistant never answers eligibility; routes to staff; test cases on never-regress list |
| Signal | Eligibility-topic questions answered without a handoff in logs |
The first run downloads Python into your browser (several MB).
Cover at least these categories: accuracy, privacy, security, fairness across languages and groups, safety in crisis situations, reliability, vendor dependency, and staff workload. A risk with no owner is not managed.
3. The evaluation plan
Test set: size, sources (de-identified), case types (common, edge,
harmful, out of scope), languages, who labeled it
Metrics: answer correctness, refusal accuracy, retrieval recall@k,
citation validity, latency p95, all split by language
Launch bar: e.g. common cases >= target; harmful cases = 100% refused
Never-regress: list of critical cases; one failure blocks release
Judge: criteria graded by LLM judge, calibration plan vs people,
or "not used" with reason
Red team: scope, harms, participants (incl. community members), dates
Stop criteria: measured conditions that trigger the kill switch or
rollback, written before launch
Review cadence: weekly sample review, monthly test run, owner for each
Set the actual target numbers yourself, based on the harm of a wrong answer. Do not borrow numbers from other projects. Label any number you assume.
Final check before you present
Rubric self-check
0 of 11 doneYour hardest call
Go deeper: Software Development, Simply: How to Build the Right Thing, Faster, With Fewer Headaches
Discussion
Ask. Answer. Build together.
Loading…