An LLM gateway is a proxy that sits between your applications and large-language-model providers. Instead of each app calling OpenAI, Anthropic, or Gemini directly, they all call the gateway, and the gateway gives you one API, one place to manage keys, and one point to enforce routing, guardrails, caching, cost control, and observability across every model.
If you’ve ever wired if provider == "openai" … elif "anthropic" … into application code, copied an API key into three services, or tried to answer "how much did the support team spend on tokens last month?", those are the problems an LLM gateway exists to solve.
What a gateway gives you
- One API. An OpenAI-compatible surface in front of many providers, so switching models is a config change, not a rewrite. Adoption is usually just a base-URL change.
- Routing & fallback. Pick the fastest or cheapest model that meets your bar, and fail over to a backup provider when one is down.
- Guardrails. Redact PII and leaked secrets, and block prompt injection, before a request reaches a provider.
- Caching. Serve repeat requests without paying for another upstream call.
- Keys & budgets. Scoped virtual keys, rate limits, and spend caps per team or project, provider keys never leave the gateway.
- Observability. One place to see latency, cost, errors, and what was routed where.
Why not just call providers directly?
Direct calls work until they don’t. The first outage, the first model deprecation, the first finance question, or the first compliance review turns "call the API directly" into glue code spread across every service. A gateway centralises that glue into one governed entry point, which is also the only place you can enforce a policy consistently.
Where sovereignty fits in
Standard gateways stop at routing and guardrails. The newer requirement, driven by GDPR, India’s DPDP Act, and sectoral rules like HIPAA, is data residency enforced per request. A sovereign AI gateway adds two stages most don’t ship: it classifies regulated data on every call and hard-locks it to an in-region provider, then records the decision into a tamper-evident trail so the proof generates itself.
Self-host or managed
Because a gateway is just a stateless process in the request path, you can run it yourself (your keys, your data path) or consume it managed. Routeplane is a single container, self-host it or point your client at the managed endpoint with a base-URL change.
The short version: an LLM gateway is the control point for everything between your app and the model. Once you have one, routing, guardrails, cost, and, if you build for it, sovereignty all become policy you set in one place rather than code you scatter everywhere.
Compare gateways side by side, or start with the quickstart.