One API in Front of Every LLM: Building a Cost-Aware Model Gateway

One of our autonomous code-review agents went dark for the better part of two days. It wasn’t a bug, a bad deploy, or an outage at a provider. It had simply spent all its money. The pipeline had grown, review by review, until it quietly exhausted its monthly model budget mid-cycle, and every subsequent review failed on a billing error. The worst part wasn’t the downtime. It was that when we went looking for where the money had gone, which step, which model call, which retry loop, nobody could answer. We had a monthly invoice and a vague sense of unease, and nothing in between.

That incident crystallized a belief that now shapes how we run AI at LegalZoom: cost-per-token is a sticker price that tells you almost nothing about what completing a real task will actually cost. This is the story of the gateway we built to fix that, and the surprisingly counterintuitive lessons we learned along the way.

Why the invoice lied to us

We adopted AI coding agents broadly and fast, and it worked. But success created two coupled risks. First, a single premium vendor came to dominate our AI spend, and pricing, availability, and capabilities all sit outside your control when that happens. Second, that spend was being driven by per-token pricing, which hides the real economics of getting work done.

Here is why the sticker price misleads. Two models with similar per-token rates can differ several-fold (and sometimes more) on the same task, because token efficiency, the number of agent steps a model needs to converge, and caching behavior all vary wildly. A “cheap” model that takes twice as many turns and writes its cache aggressively can cost far more than an “expensive” one. The only number that matters is realized end-to-end cost per completed task, and almost nobody measures it.

So we set two goals. Buy ourselves portability across providers as a hedge against lock-in, and build the instrumentation to optimize for cost-per-task instead of cost-per-token. The hard part, it turned out, was that portability is nearly worthless on its own.

One API, then a proxy that thinks

The foundation is a model-agnostic gateway that puts one API in front of 400+ models across 60+ providers, with automatic failover and price/performance routing. We adopted an open-source multi-provider gateway layer for this deliberately, pairing it with an open-source coding-agent harness, precisely so that no single closed vendor or tool sits on the critical path of our developer workflow.

But a raw multi-provider gateway doesn’t understand our routing decisions. So in front of it we built a thin FastAPI proxy that speaks the exact API our coding agents already expect, and makes feature-aware routing decisions based on both the requested model and the content of each request:

                    ┌─────────────────────────┐
   coding agent ───►│  routing proxy (FastAPI) │
                    └───────────┬─────────────┘
                                │  inspect model + request content
              ┌─────────────────┼──────────────────────┐
              ▼                 ▼                        ▼
      premium frontier    direct provider        multi-provider gateway
      (passthrough,       (server-side web        (everyday coding,
       untouched)          fetch/search,           requested model
                           tool-search turns)      preserved, ~3-7x cheaper)

The routing rules are explicit, not magic:

• Premium frontier requests pass straight through, untouched. We rewrite nothing, so existing user-level cost controls still apply. If you ask for the top-tier model, you get it, billed normally.
• Requests that need vendor-only server-side capabilities, web fetch, web search, active tool-search turns, route to the vendor that actually supports them. We don’t downgrade a request that depends on a feature only one provider offers.
• Everything else, everyday coding work, routes to the cheaper multi-provider gateway, with the requested model preserved.

That last path is where the savings live. Routing ordinary coding work through open-source models via the gateway came in several-fold cheaper (roughly 3–7×) than the premium frontier model for equivalent mid-tier work, with quality we were willing to ship.

The HTTP 400 that nearly killed the whole idea

Feature-aware routing sounds clean until you hit the real incompatibility: cheaper providers reject vendor-specific content blocks. When a conversation has used vendor server-side tools, the message history accumulates blocks (tool-search, tool-reference, server-tool-use) that other providers have never heard of. Forward that history verbatim and you get an HTTP 400.

The naive answer is “pin that whole conversation to the vendor forever.” That’s expensive. One tool-discovery turn early in a session would condemn every follow-up to the premium path. Instead, the proxy strips the stale tool-search history blocks before forwarding plain follow-up turns to a cheaper provider. Once tool discovery is done and the conversation goes back to ordinary coding, cheap routing resumes mid-session. This reconciliation logic is unglamorous, and it is exactly what makes the abstraction pay off in practice.

Keeping the credentials out of the hot path

Credentials are governed; inference traffic is fast. An auth broker provisions managed gateway API keys via browser-based PKCE auth, with secrets held in a managed secrets store for rotation and governance. But the broker is not in the request path: actual inference traffic goes directly from the client to the gateway and providers. We get credential governance without paying proxy overhead on every token.

Making “where did the money go” answerable

Remember the incident where nobody could say where the money went? We made that impossible by design. The proxy emits per-response provider attribution and sets tracing span tags (provider, route, requested model versus upstream model, and generation id) into our observability platform. The result is that a single dashboard dimension covers every route, no matter which of the 60+ providers served a given request. You can slice spend and behavior by route without instrumenting each provider separately.

What the broken pipeline taught us that the architecture didn’t

The migration of our broken code-review pipeline became the proving ground, and it taught us more than the architecture did.

The single most counterintuitive finding: cache writes, not cache reads, can dominate your bill. When we finally got per-step visibility on the original vendor-direct path, the cache hit rate looked healthy, comfortably high. And yet cache writes were the majority of total cost, far outweighing the reads they enabled. A high hit rate felt like proof of efficiency; it was nothing of the kind. If you only track hit rate, you are flying blind on cost. Track cache write versus read cost separately.

We also learned to name our algorithm correctly, then give it a stop condition. The pipeline ran a generate-verify-refine loop we’d informally called “gradient descent.” It wasn’t gradient descent. It was closer to CEGAR (counterexample-guided abstraction refinement), and the mislabel mattered, because it had no explicit convergence check. Every pass ran unconditionally whether or not the output had stopped improving. Adding a CEGAR-style stop condition with cached dispatch eliminated entire passes of pure waste. Combined with tiering agent effort by PR size, lowering verify turns, and deleting a step that added latency with no value, the same workload migrated to a gateway-routed model with caching enabled cost several-fold less per review on cached runs.

The cheaper model also turned out to be complementary to human review, not a replacement. We validated quality on real PRs, not benchmarks. The cheaper gateway-routed model caught everything the human reviewer found, plus additional issues, and it also missed some bugs the human caught. The honest conclusion is that it augments human review; it does not replace it.

The throughline of every lesson above: benchmark providers, not models, on realized task cost. Don’t compare vendor list prices. Run the same representative production workload across providers, measure end-to-end realized cost broken down by step and by cache behavior, validate output quality, and only then route.

And portability only pays off if you actually exercise it. Once the gateway abstraction existed, switching providers became roughly a config-level change. But we’d never tested that switch under load until the budget crisis forced us to, and a capability you’ve never exercised is a capability you don’t really have. Abstraction is a promise; benchmarking is the proof.

The next agent won’t spend in the dark

The bigger shift here is cultural, not architectural. We stopped treating model spend as a fixed cost of doing AI and started treating it as an engineering surface, something you instrument, benchmark, and optimize like latency or memory. One API in front of every LLM buys you the option to route intelligently. Feature-aware routing, request reconciliation, per-step and cache-aware observability, and disciplined cost-per-task benchmarking are what turn that option into real savings without giving up advanced capabilities.

We’re continuing to push per-step cost visibility deeper and to fold more workloads onto the gateway as we benchmark them. The next agent that grows quietly won’t be able to spend its budget in the dark. We’ll see exactly where every dollar goes, and we’ll know whether it was worth it.