Concepts

Architecture

One control plane between your applications and every LLM provider. Here is what sits inside the gateway, and what flows through it on every request.

Routeplane sits between your application and the model providers. Everything it does happens in one pass over the request, before a provider is chosen and again on the way back — there is no sidecar to run, no agent to install, and no SDK to adopt.

The request path

Every call takes the same route. The order matters more than it looks, and one step in particular is deliberately placed where it is.

The Routeplane request lifecycle in six stages. One: authentication, a constant-time lookup of the virtual key against an in-memory registry, resolving the tenant, tier and capability set. Two: sovereignty, classifying the request body for regulated personal data and determining a required region, before any masking. Three: inbound guardrails, masking personal data in every message. Four: provider selection, restricting to region-resident providers where required, dropping providers whose circuit is open, and ordering the rest by the chosen strategy. Five: the attempt loop, calling a provider and recording the outcome against its circuit breaker and latency average, falling back to the next candidate on failure. Six: the response, masking personal data in the reply and recording usage plus a signed audit entry. A request with no resident provider available exits early with 422; a request where every candidate fails returns 500.
The synchronous request lifecycle. Classification runs before masking — a deliberate ordering, explained below.

Why classification comes before masking

Masking personal data and detecting personal data are the same problem read in opposite directions. If masking ran first, the classifier would inspect a message whose Aadhaar numbers and phone numbers had already been replaced with placeholders, find nothing regulated, and conclude the request was free to route anywhere.

So classification runs against the original text and masking runs after it. The residency decision is made on what you actually sent; the provider only ever sees the masked version.

Sovereign routing

This is the part with no equivalent in other gateways, and the reason the ordering above is load-bearing. Most gateways route by the region you declare. Routeplane reads the request body and enforces the region the content requires.

The sovereign routing decision. A request is classified for regulated personal data. If none is found, it routes normally using the requested provider or fallback chain. If regulated data is found, the gateway determines whether a jurisdiction is required, from the content itself or the x-routeplane-residency header. If one is required, routing is restricted to providers resident in that jurisdiction — a hard constraint that overrides the requested provider. If a resident provider is available the request is routed within the jurisdiction; if none is available the request is refused with 422 rather than being routed somewhere non-compliant.
Residency is a hard constraint: it overrides the provider you asked for, and fails closed.

Two properties are worth stating plainly, because they are the ones an auditor will ask about:

  • It overrides you. If the content requires a jurisdiction, the requested provider loses. A header cannot widen the constraint, only narrow it.
  • It fails closed. When no resident provider is eligible the request is refused with 422. It is never quietly downgraded to a non-compliant route, because a compliance control that degrades under pressure is not a control.

Detection today covers the India/DPDP profile — Aadhaar (checksum-validated, not merely pattern-matched), PAN, email, and phone. The engine itself is jurisdiction-generic: adding a profile means adding recognisers, not rewriting routing.

Failure and fallback

A gateway's value is mostly visible when a provider is having a bad day. Routeplane tracks the health of each provider independently and routes around the unhealthy ones without you doing anything.

Provider fallback behaviour. A request with a fallback chain first has providers with an open circuit skipped, since they have failed recently and are not retried until recovery. The survivors are ordered by the chosen strategy — priority, weighted, cost, or latency — and the first candidate is tried. On success the response is served, with a header naming which provider actually served it. On failure, if the response is streaming and the first chunk has already been sent, the stream simply ends rather than switching providers mid-answer. Otherwise the next candidate is tried, and if no candidates remain the request returns 500. Every attempt, successful or not, updates that provider's circuit breaker and latency average.
Fallback across a provider chain, with one deliberate limit on streaming responses.

The streaming rule is the non-obvious one. Once the first chunk of a streamed answer has been sent, the gateway is committed to that provider. If it then fails, the stream ends rather than restarting on another provider — because the alternative is splicing the first half of one model's answer onto the second half of another's and returning it as a single coherent response. Ending honestly is better than that.

Failed attempts still feed the provider's health signals, so a provider that is failing gets its circuit opened and stops being tried until it recovers.

Providers

Requests fan out to OpenAI, Anthropic, Azure OpenAI, Google Gemini, AWS Bedrock, Mistral, Cohere, Groq, DeepSeek, Together, Fireworks, xAI and OpenRouter — fourteen adapters in all, including a generic adapter for any OpenAI-compatible endpoint, which covers self-hosted Ollama, vLLM and LocalAI.

Each adapter translates the canonical request into that provider's native API and back, so a fallback chain can span providers whose wire formats have nothing in common. Selection is made per request by your routing policy, and overridden by sovereign routing whenever regulated data applies.

What runs where

Provider credentials are held by the gateway and resolved server-side against your virtual key. Your application holds one Routeplane key and never sees a provider key, which is what makes rotating a provider credential an operation you perform in one place rather than a redeploy of every service.

Tenants are separated at the deployment topology — shared pools and dedicated environments — rather than by a filter on a shared table.

Design properties

Some characteristics follow from the architecture rather than from tuning, and those are the ones worth relying on:

  • No garbage collector. The data plane is Rust, so there is no runtime GC and therefore no GC pause to appear in your tail latency.
  • A lock-free hot path. Circuit breakers and latency tracking use atomics rather than mutexes, so provider health accounting does not serialise concurrent requests.
  • Scale to zero. The gateway is serverless and scales down to nothing when idle, which is why a dedicated environment is not priced like one.
  • Bounded, non-blocking telemetry. Observability is off the request path. Under pressure it drops records and counts the drops rather than slowing the request that produced them.