Every LLM provider has outages, and every provider has rate limits. If your product calls exactly one, you have inherited their risk posture as your own. When Groq rate-limits at 2am or a free endpoint silently starts queuing, your users do not see "provider degraded"; they see your product failing. The multi-provider fallback playbook is how you build a product that degrades to a weaker answer instead of an error page, and it is the architecture behind two systems I ship: ArchMind, which runs a six-provider fallback chain ending in an 18-rule heuristic engine that needs zero API keys, with a keep-warm GitHub Actions workflow to keep the free endpoints alive, and ResumeMasterAI, which sits behind the routing gateway I wrote about earlier. This essay is the pattern, including the unglamorous parts.
Projects: ArchMind AI and ResumeMasterAI.
The premise
Single-vendor AI products are single points of failure wearing a nice UI. Provider risk is not hypothetical: rate limits, regional incidents, deprecated models, pricing changes, silent latency degradation. Any one of these can take a feature from "works" to "errors out" without your deploy logs showing a thing, because you changed nothing; the world did.
The design goal is not "99.999% uptime" (I will not quote a number I cannot defend). The goal is that when a provider fails, the user gets a worse answer, not an error. Every failure path ends in a weaker answer, never an error page.
The economics help too. OnRamp opens with free tiers and a full nine-step autopilot run ends up cheaper than one paid call, which means the fallback chain and the cost model are the same mechanism. Building the chain is not resilience theatre bolted onto an expensive product; it is the cheapest way to keep a multi-provider product alive while the invoice stays near zero. The lesson from OnRamp is blunt: model routing is a cost-control problem before a capability problem, and the same logic applies to ordering failover tiers.
The chain design
ArchMind runs: Groq, then NVIDIA, then OpenRouter, then Gemini, then Ollama, then HuggingFace, then an 18-rule heuristic engine. Estate360 runs a tighter pool: Groq, Gemini, Mistral, NVIDIA. OnRamp routes free-tier-first: OpenRouter, Gemini, Groq and NVIDIA up front, with DeepSeek, Qwen, Zhipu, Moonshot, Mistral, OpenAI, Anthropic, HuggingFace and Ollama behind.
The order is not arbitrary. Design the chain by three axes, in this order:
- Cost. Free and cheap tiers first, paid tiers behind. This is why ArchMind and OnRamp lead with free tiers, and why OnRamp's whole economics hold: a full autopilot run ends up cheaper than one paid call.
- Latency. Fast providers early for interactive paths; slow but capable providers later for batch.
- Capability. The deepest reasoning model is not first in the chain; it is reserved for the queries that need it, which is the routing problem, not the fallback problem.
Mechanics that matter:
- Per-query-type chains. OnRamp routes per query type rather than with one global chain; a classification call and a synthesis call do not deserve the same fallback order.
- Health checks and circuit breakers. If a provider starts failing or latency-spiking, trip the breaker and skip it for a cooldown window instead of paying the timeout on every request.
- Timeout budgets, not infinite patience. Each hop gets a budget; when it expires, fail over, do not retry forever.
- Retry vs failover, decided in advance. Retry the same provider only for transient, idempotent failures (rate limit with a short reset, blip). Anything else, or repeated failure, fails over down the chain.
- Log the decision. Every failover is logged with the reason. Without that log you cannot tune the chain or detect a provider slowly dying.
- Health means signals, not pings. Watch error rate and latency on real traffic, not just whether the endpoint answers a synthetic hello. A provider can respond 200 and still be useless because every response is a queue timeout or a degraded model.
| Stage | Role in chain | When it serves |
|---|---|---|
| Groq / NVIDIA / OpenRouter / Gemini | Free-tier first line | Default traffic; fast, cheap, occasionally rate-limited |
| Ollama / HuggingFace / Mistral | Secondary free capacity | When the front line trips or queues |
| DeepSeek / Qwen / Zhipu / Moonshot / OpenAI / Anthropic | Paid and frontier tiers | Hard reasoning or when free tiers are exhausted |
| 18-rule heuristic engine (ArchMind) | Zero-key floor | Every provider unavailable; deterministic scored baseline |
The floor: degraded but useful beats a 500
The single most important decision in ArchMind was the last link: not another provider, but a deterministic, 18-rule heuristic engine that requires zero API keys. When every provider in the chain is down, the product does not error out; it returns a scored, rule-based baseline. The lesson generalizes: every chain should terminate in something that cannot fail, even if that something is deliberately shallow. "Service degraded, here is the deterministic answer" beats a 500, beats a spinner that never resolves, and beats a blank page every single time.
Design your floor around what your product fundamentally does. For ArchMind it is rule-based architecture scoring. For a support bot it might be canned answers to the twenty most common intents. For a resume tool it might be deterministic checks (quantified impact present or absent, tense consistency). The floor will not impress anyone, and it is the reason nobody gets blocked.
Free-tier-first routing
Free-tier-first is routing applied to your fallback chain, and it inherits the same lesson I learned building ResumeMasterAI: model routing is a cost-control problem before a capability problem. Route by expected task difficulty; the easy 80% of queries never touch a paid model. Log which tier served each query; never silently swap models mid-session without logging. Silent swaps destroy your ability to answer the question "why did quality drop on Tuesday".
One warning: free tiers trade latency and rate limits for cost. That is the correct trade for batch and forgiving-UX paths, and the wrong trade for latency-critical interactive flows. Put the SLO, not the price tag, in charge of the interactive path.
The unsexy parts (where fallback chains actually live or die)
- Keep-warm pings. ArchMind runs a scheduled GitHub Actions workflow to keep the free endpoints alive. Free tiers that sleep or throttle on idle will fail you exactly when traffic arrives.
- Alerting on fallback rate. A rising fallback rate is your early warning that a provider is degrading. If nobody gets paged on it, the chain is decorative.
- Budget caps per provider. A retry storm against a paid provider is a billing incident. Caps and alerts, always.
- Kill switches. You need a way to disable a provider without a deploy, because the failure mode you are routing around is often an account or key problem, not code.
- Test the failover like backups. Pull the plug on purpose, in staging, and watch what happens. ArchMind exercises the fallback chain in CI precisely because it is the part most likely to break. An untested fallback chain is a hypothesis, not a control.
- Postmortem the failovers. When the chain fires in production, write down which link failed, how long detection took, and whether the floor served. Those notes are how the chain earns its keep; without them you are just hoping the next outage lands on a well-tested path.
- Drill the cold case, not just the warm one. Most drills test the first failover hop. Also test the total collapse: every provider down, only the heuristic engine left, and confirm the product still answers with its deterministic baseline instead of an error page.
When not to build this
Honest scoping, because resilience machinery has a real cost:
- Internal tools used by five people who can tolerate a retry: skip the chain, keep the provider interface abstraction.
- A single provider with an SLA you actually trust (and budget for): the SLA is your fallback plan; know its terms.
- MVPs where speed beats resilience: ship the happy path, abstract the provider interface anyway, and add the chain when the first provider wobble costs you a customer.
The abstraction is non-negotiable even when the chain is not: a thin provider interface costs days, and it converts "we are down until the vendor recovers" into a config change later.
The takeaway
Order the chain by cost, latency and capability. Give it a zero-key floor. Log every decision. Alert on fallback rate. Pull the plug monthly. And accept the trade honestly: you are buying resilience with complexity, and the only unforgivable version of this work is the one nobody has ever tested.
If you want a multi-provider LLM architecture that fails soft instead of hard, that is my work. Fixed scope from ₹25,000, fixed quote in writing, free 15-minute intro call: https://kunjshah.in/contact.