lesson
60 min

Model Routing and Fallbacks

Send each task to the right model by type, stakes, and data sensitivity, and keep serving people when a model or provider fails.

Objective

By the end of this lesson, you can design a model router, build retries with exponential backoff, and write a fallback chain that ends with a path to a person.

Model routing

Not every task needs the largest model. Model routing sends each request to the model that fits it:

  • By task type. Classifying a message or pulling out a date is often handled well by a small, fast model. Drafting a sensitive letter may need a larger one.
  • By stakes. High-stakes outputs get a stronger model and human review.
  • By data sensitivity. Requests with personal data go only to models and providers approved for that data, sometimes a local model.

Routing rules should be explicit code you can test, not a model deciding for itself. Every route must pass your test set (Module 7) before it goes live.

Run it. Then add a rule: tasks in Spanish with high stakes go to the large model plus human review. Add a task to test it. Notice that the PII rule is checked first. Why does order matter?
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

The first run downloads Python into your browser (several MB).

Retries and backoff

Calls fail: timeouts, rate limits (a provider capping how many requests you can send), brief outages. Retry, but carefully:

  • Exponential backoff: wait longer after each failure (0.5 s, 1 s, 2 s...). Instant retries pile load onto a struggling service.
  • Jitter: add a small random amount to each wait so many clients do not retry at the same moment.
  • Retry limit: stop after a few tries.
  • Only retry safe calls. Retrying a "send message" call can send it twice. Use an idempotency key, a unique ID that lets the receiver ignore duplicates.
  • Timeouts: set a maximum wait on every call. Waiting forever is a failure too.

Fallback chains

A fallback chain is the ordered list of what to try when something fails. It must end somewhere a person can still get help.

Run it. The primary model times out three times, then the backup answers. Make backup raise TimeoutError too (copy the raise line from primary). What does the person see now? That last step is the most important one in the chain.
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

A good chain for an intake assistant:

  1. Primary model, with retries.
  2. Backup model or provider, already tested on your test set.
  3. A non-AI answer: FAQ search or fixed replies for common questions.
  4. A clear message with a phone number, text line, or front-desk hours.

Test the chain on purpose. Turn off the primary in a test environment and watch what users would see.

A national example

Reliability design

0 of 8 done

Go deeper: Systems Administration, Simplified: A Practical Blueprint for Uptime, Security, and Scale

Ask. Answer. Build together.