Logging and Monitoring
Log enough to debug without storing what you must not, and watch quality and fairness as closely as uptime.
Objective
By the end of this lesson, you can design structured logs for an AI system, choose service and quality metrics, and set alert thresholds that page a person when something goes wrong.
Ship in stages
Before monitoring, a word on releasing. AI changes, including a new prompt, model, or document set, can shift behavior in ways tests miss. Release them in stages:
- Shadow mode: run the new version alongside the old one and compare outputs, without showing users.
- Canary: send a small share of traffic to the new version and watch the metrics.
- Full rollout only when metrics hold. Keep the old version ready to switch back.
- Feature flag: a setting that turns a feature on or off without a new deployment. This becomes your kill switch (next lesson).
Structured logs
Observability means being able to understand what a system is doing from what it records. It starts with structured logs: one record per request with named fields, not loose text.
Log these:
- Request ID, timestamp, and which feature was used.
- Model name and version, prompt version, retrieval index version.
- Latency, tokens used, success or error type.
- Which tools were called, and whether a person approved.
- Retrieved chunk IDs and validation results.
- Guardrail triggers and fallbacks used.
- User language (if known) for fairness checks.
Do not log raw personal data by default. Redact before logging (Module 6). If you must keep some conversation text for debugging, keep it in a restricted store with a short deletion date. Never log raw conversations with minors.
What to monitor
| Layer | Metrics | Example alert |
|---|---|---|
| Service | Error rate, p50 and p95 latency, timeouts | Error rate above 5% for 10 minutes |
| Quality | Validation failures, citation failures, "I don't know" rate, thumbs-down rate | Validation failures double in an hour |
| Safety | Guardrail triggers, crisis-language events, blocked actions | Any crisis event without a handoff logged |
| Fairness | All of the above, split by language and channel | Spanish error rate well above English |
| Drift | Weekly test set score on production config | Score drops below launch bar |
p95 latency is the time within which 95% of requests finish. It shows what the slow tail feels like, which an average hides.
The first run downloads Python into your browser (several MB).
Watch quality, not only uptime
A system can be "up" and still wrong. Uptime monitors will stay green while a model change makes answers worse. Run your test set against the live configuration on a schedule. Have a person review a small random sample of real (redacted) conversations every week.
Quiz: Observability check
Go deeper: Systems Administration, Simplified: A Practical Blueprint for Uptime, Security, and Scale
Discussion
Ask. Answer. Build together.
Loading…