From Model Sprawl to a Shared ML Platform

Not long ago, shipping a model at LegalZoom looked different on every team. One team wrapped a scikit-learn pipeline in a hand-rolled Flask app. Another stood up a bespoke service to call a hosted LLM. A third copied the second team’s repo, changed the secrets, and called it a day. Each came with its own deployment scripts, its own auth story, its own way of loading secrets, and its own (usually absent) telemetry. A data scientist with a working notebook still faced weeks of plumbing before anything served real traffic.

That is model sprawl, and it is expensive in a quiet way. Nobody owns the duplication. Every new model re-pays the same tax on deployment, versioning, data contracts, auth, secrets, and observability. The fixes one team discovers never reach the others. We decided to consolidate onto a single paved path — but with a constraint we cared about as much as the consolidation itself: it could not become a mandate that funneled every workload through one rigid pattern.

The challenge

The hard part was the heterogeneity. We needed one runtime to serve genuinely different shapes of work:

• Classic ML prediction. A conversion-funnel dropoff predictor and a learning-to-rank upsell engine — synchronous, low-latency, feature-engineering-heavy inference.
• LLM agents. A customer-facing conversational assistant and a voice agent, with server-side conversation state, tool calling, and streaming.
• Tool servers. Shared capabilities (knowledge search, legal search, URL fetch) that multiple agents need to reuse.

A platform tuned only for REST predict endpoints would have rejected the agents. One built only for LLM orchestration would have been a poor home for a tree-based pipeline that just needs to load a model and return a score in a few milliseconds. We wanted both, behind consistent conventions, without pretending they are the same thing.

There was also an infrastructure trap waiting for us. Our early runtime used load-balancer-only ingress. That made external and SaaS integrations awkward, and worse, it forced in-cluster service-to-service calls to hairpin all the way out through external ingress and back in. Internal traffic paid public-internet latency to talk to a neighbor.

The approach

We built one shared platform as a monorepo, where each model, agent, or tool server is its own service. Every service is a FastAPI app, gets the same scaffolding and shared libraries, and follows the same staged path from experimentation to production. Deployment is standardized on Kubernetes, with a separate deployment repo and ArgoCD manifests reconciling what actually runs. Secrets come from a managed secrets store, not from a .env someone pasted into a pipeline. Python packaging uses uv, and datasets and model artifacts are versioned so every deployment ties back to a known data and model snapshot.

The serving layer deliberately mixes two styles. Classic models expose a REST predict route. LLM-backed work is served as agents and as hosted MCP servers over HTTP. For LLM access, a standard predict contract looks roughly like this:

class PredictRequest(BaseModel):
    prompt: str
    model_id: str
    system: str | None = None
    max_tokens: int = 1024
    temperature: float = 0.0

class PredictResponse(BaseModel):
    completion: str
    input_tokens: int
    output_tokens: int
    latency_ms: float

Behind that contract, the platform can proxy to AWS Bedrock and other providers, so a caller asks for a model_id and gets back completion text, token counts, and latency without owning a provider integration.

The two consumer-model services show how much the same runtime can stretch. The funnel dropoff predictor is a FastAPI inference service that loads separate desktop and mobile joblib pipelines (an AdaBoost ensemble). The pipelines do their own time-feature engineering, log transforms, one-hot encoding, and scaling internally on raw session features via pandas — so the service stays thin and the modeling logic stays versioned with the artifact. The upsell ranker — a newer workload we’ve been moving from exploration toward production — is a CatBoost YetiRank learning-to-rank model, selected from candidate variants by an automated model-selection pipeline. In its serving path it calls a separate max-cart-value predictor over HTTPX to set a budget, scores candidate products with a CatBoost Pool, ranks them by expected value, and greedily selects offers under that budget and a cap. Two completely different ML stacks, one serving convention.

On the agent side, the customer assistant is FastAPI plus PydanticAI, with server-side conversation management. Chat history and feedback live in Postgres, with a lightweight web UI for quick internal testing. The voice agent runs as a separate realtime conversational service.

One of the highest-leverage decisions was centralizing agent tools behind a shared MCP server. Rather than each agent reimplementing knowledge search, legal search, product lookup, and URL fetch, we register those tools once (using FastMCP, with read-only annotations) and let agents consume them two ways: by direct code import, or over HTTP via streamable MCP. The HTTP path is the interesting one — when we improve or add a tool, every agent that consumes it over HTTP picks up the change without redeploying. Tool evolution gets decoupled from consumer release cycles.

To kill the ingress hairpin, we evolved from load-balancer-only ingress toward Kong plus Istio. Kong handles external routing, authentication, and exposure of HTTP endpoints; Istio handles in-cluster service-to-service routing. Internal calls now stay inside the mesh instead of round-tripping through external ingress, and external/SaaS integrations stopped being a fight.

Observability is built in rather than bolted on. Every service gets Datadog tracing via DDTrace, latency metrics, and token tracking by default. The customer assistant goes further with OpenTelemetry tracing and a desktop OTEL viewer for deep LLM debugging — invaluable when you need to see exactly which tool call or prompt turn blew up a latency budget. Persistence follows one path too: Postgres (with Alembic multi-schema migrations) synced to Snowflake via Fivetran, so request/response logs, conversation history, and feedback all land in analytics tables without per-team glue.

The trade-offs

The most important decision was refusing to make the platform a mandate. A shared runtime that everyone is required to route through is just centralized sprawl with a single point of failure. So our guidance points the other way: if a team can evaluate an agent SDK like PydanticAI or LangChain with a direct API key and does not need our Python runtime, that is the right call. The shared platform earns its place when a central platform team owns the agent logic, or when a team needs a Python stack but has no Python infrastructure to run it on. We actively discourage using the platform as a generic LLM proxy. It wins by being the easiest paved path for the workloads it fits, not by being the only road.

That choice has a cost: we give up the tidiness of “everything goes through one gateway.” Some LLM traffic we could have metered centrally now runs on direct SDK calls we do not see. We think that is the right trade. A bottleneck that every AI feature in the company depends on is a worse outcome than a little decentralization.

The monorepo is the other trade. Shared scaffolding and libraries are a real accelerant — a new service inherits deployment, auth, secrets, and telemetry for free. But a monorepo of independently deployed services demands discipline about shared dependencies, so one team’s upgrade does not surprise another’s release.

What we learned

A paved path beats a mandate. The platform spread because using it was genuinely faster than rolling your own, not because anyone was forced onto it. Standardizing the boring layers — deployment, secrets, telemetry, the persistence-to-analytics path — is where most of the value lives; it is the part every team was rebuilding badly. The MCP-over-HTTP pattern, which lets shared tools improve without coordinated redeploys, is a decoupling we will keep leaning on. And fix your ingress topology early: hairpinning internal traffic through external ingress is the kind of quiet tax that is easy to miss and well worth removing.

We still have heterogeneity to absorb and guidance to keep current as agent SDKs and the MCP ecosystem move. But the through-line holds: one runtime, every shape of model — and a deliberate choice not to make it the only door.