Every AI product I have shipped eventually hits the same invoice problem: the bill grows because every single request pays flagship prices, including the thousands of requests that never needed a frontier model. ResumeMasterAI, my career platform for scoring and tailoring resumes against job descriptions, hit that wall hard. The fix was not a cheaper model and it was not a bigger discount from a provider. The fix was a routing gateway in front of the models, and it cut my API spend by 65% while actually improving output quality. Here is how it works, what I measured, and where it breaks.
The project is public: ResumeMasterAI is built on Python, LangGraph, an AI Gateway layer and ChromaDB, with the source at github.com/KunjShah01/job-snipper. What follows is the architecture, not a sales pitch.
The invoice problem
The naive architecture for an LLM feature is one model behind one endpoint. Every request, from "rewrite this bullet point" to "weigh four constraints across a career arc and propose a positioning strategy", goes to the same large model. It works, demos well, and then the first real invoice arrives.
The uncomfortable part is not the total. It is the distribution. A large fraction of the traffic in a production agent is shallow: short rewrites, extractions, formatting, classification, confirmation steps. These are tasks where a small, cheap model is not a compromise; it is the correct instrument. Paying flagship prices for them is pure waste, and it scales linearly with adoption.
There is a second-order problem too. A single hard-coded model couples your product to one provider's pricing, availability and rate limits. The invoice problem and the resilience problem have the same root cause: you have not told the system which requests are actually hard.
The insight: model size should match task complexity
The models are not the bottleneck. The allocation is. Task complexity varies enormously across requests, and model capability varies enormously across tiers, so the obvious move is to match them. Most of the time, the honest answer to "which model should handle this" is: the smallest one that clears the quality bar for this specific task, not the biggest one on the roadmap slide.
What changed in my head, building this, is that routing is a systems problem, not a model problem. You need a classifier, a routing policy, a checkpoint trail, an eval suite and a human review loop. Miss any one of those and the savings are imaginary or the quality quietly rots.
| Request profile | Route | Why |
|---|---|---|
| Short rewrite, extract, classify, format | Small, cheap model | Shallow task; big model adds cost, not correctness |
| Retrieval-heavy answer from known docs | Small model plus retrieval | The work is in the context, not the reasoning |
| Multi-constraint reasoning, ambiguous brief | Large model | Genuine hard reasoning; this is what you pay the flagship for |
| Strict output structure, high stakes | Large model plus validation | Structure violations are cheap to detect, expensive to miss |
The implementation: classifier, route, checkpoint, spot-check
The gateway sits between the product logic and the providers. Every request passes through four stages.
1. The classifier reads the request, not the vibes
The complexity classifier looks at concrete, checkable signals: input length, task type, whether the request actually needs retrieval, and what output structure is required. No embeddings mysticism, no free-form judgment calls. A five-token "make this punchier" request and a paste of a full career history with three constraints never get scored the same way, because the inputs differ in ways that matter.
2. The router applies the policy
Easy, well-structured tasks go to small, cheap models. Hard reasoning, ambiguous briefs and strict structure requirements go to large models. The policy is explicit in code, versioned, and reviewable. When the policy is wrong, you fix the policy, not the prompt salad.
3. Checkpointing plus human-in-the-loop review
LangGraph gives me checkpoints at graph boundaries, so any run can be inspected, replayed or paused for human review. High-stakes outputs (a full tailored resume, say) can be gated on a review step instead of shipped blind. This is not bureaucracy; it is how you keep a cheap routing decision from becoming an expensive customer-facing mistake.
4. Spot-checks close the loop
Every routing decision is logged: the classifier's read of the request, the model chosen, the latency, the cost, and the downstream quality signal where one exists. A sample of those logs gets human review. That is the raw material for tuning the classifier and for catching quiet quality drift.
Measuring it honestly
Anyone can claim savings. The measurement discipline is the product. Three numbers came out of the routing gateway work: API cost dropped 65%, match precision improved 3x, and the routing layer adds under 200ms per request.
A few things about how to read those numbers without bullshit. Cost is measured per successful task, not per request. If the cheap model produces garbage and the user retries, you paid twice and saved nothing; the metric that matters is cost per task that actually completes and is accepted. Quality was held by an eval suite plus human review, not by faith in the classifier. The quality floor is the constraint; cost is the variable you optimize under it.
And the 3x precision figure deserves the honest footnote: it came from the routing gateway and its bounded agent steps, not from a magic model. Small models handle the shallow work they are actually good at, large models get reserved for genuine reasoning, and the step count is bounded so the agent cannot wander. Better allocation, better outputs. Anyone selling you a "smarter model" for that gain is hiding the real mechanism.
The catch
Routing is a cost-control problem before it is a capability problem, and it has exactly one dangerous failure mode: the misroutes are quiet. Send a hard task to a small model and you do not get an exception; you get a fluent, plausible, wrong answer. Nobody files a bug titled "the classifier was optimistic today". The user just gets worse output and leaves.
So the non-negotiables, in order of importance:
- Keep the eval suite as the arbiter. Routing decisions are only as good as the evals that grade them. If your evals are stale, your router is confidently stale too.
- Human review stays in the loop for high-stakes outputs, at whatever sampling rate you can sustain. A spot-check habit beats a quarterly audit.
- Log every routing decision, including the easy ones. When quality dips, you need to know whether the router got dumber, the model changed, or the traffic mix did.
- Budget for the fallback. When the cheap model fails validation, escalate to the large model instead of shipping. The 200ms overhead is worth paying to avoid one bad customer-facing answer.
If you cannot say what your wrong-route rate is this week, you do not have a routing gateway; you have a cost lottery.
When routing is not worth it
Routing adds moving parts: a classifier to tune, policies to maintain, logs to read, evals to keep honest. It is not free, and it is not always worth it. Skip it when:
- Traffic is low. If you send a handful of requests a day, the engineering time costs more than the flagship calls ever will.
- It is a single-use internal tool. If one team uses it for one workflow, the variance in task complexity is probably tiny anyway.
- It is a prototype. Prove the product works first. Add routing when the invoice proves the need, not before.
- You cannot eval it. If you have no way to detect wrong answers, routing is just a way to fail at lower cost.
The rule I use: route when the request stream is heterogeneous, the volume justifies the machinery, and the eval suite can actually see quality differences. Everything else is optimization theater.
What I would tell you before you build this
Start with logging, not routing. Instrument every request with its true token cost for a few weeks and look at the distribution. You will find the cheap tier of traffic is bigger than you think, and that is your business case, grounded in your own data rather than my 65%.
Then build the classifier from three signals, not twelve: input length, task type, retrieval need. Add output structure once the first version is live. Resist adding signals because they seem clever; every signal is a maintenance burden with a failure mode.
Finally, wire the human review loop before the router goes to production. The review loop is what teaches the classifier what "hard" actually means in your domain. Without it, you are routing on generic assumptions about difficulty, and those are wrong in the ways that matter most.
I build this kind of system for teams: fixed scope from ₹25,000, fixed quote in writing, free 15-minute intro call. If your LLM bill is growing faster than your user base, that is a routing problem, not a model problem. That is my work.