Stop Betting Your Inference Budget on One Model

The $120-in-75-minutes problem

You can burn $120 of Claude credits in 75 minutes. Someone on Hacker News did it last week. $200/month Max plan. Moderate coding task with sub-agents. Claude ignored the directive to send sub-agent work to cheaper models, spun up several flagship instances in parallel, and the meter redlined.

That thread tells you everything. Not because $120/hour will bankrupt you — plenty of teams spend more — but because the person paying had no lever. The subscription was flat-rate until it wasn’t. The sub-agents were a black box. Model choice was binary: flagship or not. When the bill spiked, all they could do was kill the job and complain.

Echo (tracerml.ai) showed up on HN aiming straight at that trap. The move: route each request across a pool of open-weight models — GLM-5.2, Kimi K2.7, and friends — and only spend heavy when the request earns it. Old trick, new domain. It’s the same idea that made random forests beat single trees.

The oracle experiment is the whole thesis

Before building a router, the creator ran a sanity check. Take a pool of open-weight models. Run them all on the same eval set. With hindsight, pick the best model (or blend) for each prompt. What score do you get?

The “oracle” beat every single model in the pool by a margin that matters. Of course you can’t ship an oracle — it needs the right answer up front — but it sets the ceiling. It shows how much you leave on the table by betting on one provider.

The claim: a learned router can claw back a big chunk of that gap. On the first eval mix, Echo matched Claude on aggregate while spending about a third of the inference cost. It’s early. Routing still whiffs on coding and agentic traces where per-decision scoring is hard. Wait for third‑party validation before you staple numbers into your architecture doc. But the direction is clear: smarter routing buys more quality per dollar.

Weak models win specific fights

Here’s the unlock: a model that’s weaker overall still wins specific sub-tasks. Not “good enough.” Best. Kaggle veterans have seen this movie. Ensembles topped leaderboards for years because a diverse forest of mediocre trees beat one exquisitely tuned tree on messy, real data.

Benchmarks trained us to crave one crown — GPT-5 vs Claude 4.5 vs Gemini 2.5 — but that single score smears together many task shapes: math, prose, tool-use, structured extraction, long‑context recall. Average them, name a champ, call it a day. The average hides reality. Kimi can dominate Chinese‑language math. GLM can crush JSON extraction. Claude can win multi‑step reasoning.

If your workload is truly one note, fine — pick the winner for that note and ship. Most production apps aren’t. One support session can parse a ticket, extract entities, query a database, summarize rows, then draft a reply. Those are four different jobs with four different optimal models. Sending them all to Claude is a choice — and it’s probably not the cheapest way to hit your bar.

Dynamic compute allocation is the real product

The piece worth stealing, even if you never use Echo, is dynamic compute allocation. Not just which model — how much inference this request deserves. A one‑sentence classification doesn’t need the same budget as a 40‑turn agent trace. Yet most stacks treat both the same: same model, same context, same reasoning depth.

Every other layer is smarter. Databases have query planners. CDNs have cache tiers. Video has adaptive bitrate. Only your LLM path shoves everything through the same premium pipe.

The hard part is triage before you spend. You need a cheap difficulty classifier that can say “send this to the fast lane” or “spin up collaboration” — and it has to save more than it costs. Echo is one attempt. OpenRouter is the closed‑model routing incumbent; watch what they ship next. More are coming. If you build LLM infra, build around this abstraction.

The subsidized-plan asterisk

Here’s the fair pushback. If you’re on a heavily subsidized flat‑rate — Claude Max at $200/month, Cursor Ultra, ChatGPT Enterprise — your effective price is already far under API list. SemiAnalysis pegs Anthropic’s per‑token cost to Max subscribers around $0.99 per million vs $20 per million on the raw API. That’s a 20x subsidy.

At that price, “one‑third the cost” vs API list won’t move you. One‑third of list is still 6–7x more than a subsidized plan. If you’re a solo dev or a small team chewing through Max, a router won’t save you today.

Zoom out. Those subsidies exist to lock in usage before the game changes. They won’t last. Anthropic already pulled Claude 5 off subscription usage on the Pro plan in July. When the subsidies end — and they will — the spread between “one flagship on API” and “smart open‑weight ensemble” decides who can ship at scale without torching margin.

Enterprises already live in that world. Compliance and retention rules knock out consumer subs. They pay full API rates. They’ll be first to push routers into production.

What to do this week

Don’t migrate to Echo yet. It’s new. The routing policy is opaque (by design — that’s the product). The “Fable‑level” claim needs outside evals. Keep your production calm.

Do this instead:

• Add per‑request cost and latency logging to every LLM call, tagged by task type. By Friday, you should know which 20% of request types eat 60% of your budget.
• Take your top three task types and run them on two open‑weight models — GLM‑5.2 and Kimi K2.7 are easy to access. Test on your prompts, not someone else’s benchmark. You’ll likely find at least one task where an open‑weight model is “good enough” at a fraction of the cost. That’s your first routing rule, hand‑rolled.
• Get your taxonomy and dashboards ready so when a real router proves itself — Echo, OpenRouter’s smart routing, or a new entrant — you can evaluate it in a day, not a quarter.

The next winners won’t be the teams married to one model. They’ll be the ones who built routing muscle early while everyone else argued about which flagship deserved the crown.