Model Routing and Fallbacks
Send each task to the right model by type, stakes, and data sensitivity, and keep serving people when a model or provider fails.
Objective
By the end of this lesson, you can design a model router, build retries with exponential backoff, and write a fallback chain that ends with a path to a person.
Model routing
Not every task needs the largest model. Model routing sends each request to the model that fits it:
- By task type. Classifying a message or pulling out a date is often handled well by a small, fast model. Drafting a sensitive letter may need a larger one.
- By stakes. High-stakes outputs get a stronger model and human review.
- By data sensitivity. Requests with personal data go only to models and providers approved for that data, sometimes a local model.
Routing rules should be explicit code you can test, not a model deciding for itself. Every route must pass your test set (Module 7) before it goes live.
The first run downloads Python into your browser (several MB).
Retries and backoff
Calls fail: timeouts, rate limits (a provider capping how many requests you can send), brief outages. Retry, but carefully:
- Exponential backoff: wait longer after each failure (0.5 s, 1 s, 2 s...). Instant retries pile load onto a struggling service.
- Jitter: add a small random amount to each wait so many clients do not retry at the same moment.
- Retry limit: stop after a few tries.
- Only retry safe calls. Retrying a "send message" call can send it twice. Use an idempotency key, a unique ID that lets the receiver ignore duplicates.
- Timeouts: set a maximum wait on every call. Waiting forever is a failure too.
Fallback chains
A fallback chain is the ordered list of what to try when something fails. It must end somewhere a person can still get help.
A good chain for an intake assistant:
- Primary model, with retries.
- Backup model or provider, already tested on your test set.
- A non-AI answer: FAQ search or fixed replies for common questions.
- A clear message with a phone number, text line, or front-desk hours.
Test the chain on purpose. Turn off the primary in a test environment and watch what users would see.
A national example
Reliability design
0 of 8 doneGo deeper: Systems Administration, Simplified: A Practical Blueprint for Uptime, Security, and Scale
Discussion
Ask. Answer. Build together.
Loading…