lesson
60 min

Logging and Monitoring

Log enough to debug without storing what you must not, and watch quality and fairness as closely as uptime.

Objective

By the end of this lesson, you can design structured logs for an AI system, choose service and quality metrics, and set alert thresholds that page a person when something goes wrong.

Ship in stages

Before monitoring, a word on releasing. AI changes, including a new prompt, model, or document set, can shift behavior in ways tests miss. Release them in stages:

  • Shadow mode: run the new version alongside the old one and compare outputs, without showing users.
  • Canary: send a small share of traffic to the new version and watch the metrics.
  • Full rollout only when metrics hold. Keep the old version ready to switch back.
  • Feature flag: a setting that turns a feature on or off without a new deployment. This becomes your kill switch (next lesson).

Structured logs

Observability means being able to understand what a system is doing from what it records. It starts with structured logs: one record per request with named fields, not loose text.

Log these:

  • Request ID, timestamp, and which feature was used.
  • Model name and version, prompt version, retrieval index version.
  • Latency, tokens used, success or error type.
  • Which tools were called, and whether a person approved.
  • Retrieved chunk IDs and validation results.
  • Guardrail triggers and fallbacks used.
  • User language (if known) for fairness checks.

Do not log raw personal data by default. Redact before logging (Module 6). If you must keep some conversation text for debugging, keep it in a restricted store with a short deletion date. Never log raw conversations with minors.

What to monitor

LayerMetricsExample alert
ServiceError rate, p50 and p95 latency, timeoutsError rate above 5% for 10 minutes
QualityValidation failures, citation failures, "I don't know" rate, thumbs-down rateValidation failures double in an hour
SafetyGuardrail triggers, crisis-language events, blocked actionsAny crisis event without a handoff logged
FairnessAll of the above, split by language and channelSpanish error rate well above English
DriftWeekly test set score on production configScore drops below launch bar

p95 latency is the time within which 95% of requests finish. It shows what the slow tail feels like, which an average hides.

Run it. Two slow failures push p95 and the error rate past the thresholds, so the alert fires. Change the two failing entries to (1100, True) and (1300, True). Does the alert still fire? Then add a field for language to each log line.
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

The first run downloads Python into your browser (several MB).

Watch quality, not only uptime

A system can be "up" and still wrong. Uptime monitors will stay green while a model change makes answers worse. Run your test set against the live configuration on a schedule. Have a person review a small random sample of real (redacted) conversations every week.

Quiz: Observability check

  1. 01Which field helps most when answer quality suddenly drops?

    Choose one.

  2. 02Your dashboard shows 99.9% uptime. Does that mean answers are good?

    Choose one.

Go deeper: Systems Administration, Simplified: A Practical Blueprint for Uptime, Security, and Scale

Ask. Answer. Build together.