Provider guide

Self-hosted models through one OpenAI-compatible gateway

Front your own Ollama, vLLM, or LocalAI server with Routeplane and get the same fallback, guardrails, residency, and cost attribution as the hosted providers — while your model stays on your infrastructure.

The generic self-hosted adapter points at any OpenAI-compatible server you run — Ollama, vLLM, LocalAI, Text Generation Inference, or LM Studio. Routeplane wraps it with fallback, in-data-plane guardrails, and residency, so your local model plugs into the same gateway as the hosted providers.

Because the model runs on your own infrastructure, self-hosting is the strongest residency story of all — the data never leaves your region.

Drop in your OpenAI client

Point your existing OpenAI SDK at the gateway and add two headers — your virtual key and the provider. Nothing else about your code changes.

bashcurl
curl https://api.routeplane.ai/v1/chat/completions \
  -H "content-type: application/json" \
  -H "x-routeplane-api-key: rp_your_gateway_key" \
  -H "x-routeplane-provider: self_hosted" \
  -d '{"model":"llama3.1","messages":[{"role":"user","content":"Hello!"}]}'
pythonopenai SDK
import openai

client = openai.OpenAI(
    api_key="rp_your_gateway_key",
    base_url="https://api.routeplane.ai/v1",
    default_headers={
        "x-routeplane-api-key": "rp_your_gateway_key",
        "x-routeplane-provider": "self_hosted",   # route to Self-hosted (Ollama / vLLM / LocalAI)
    },
)

resp = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)
typescriptopenai SDK
import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: 'rp_your_gateway_key',
  baseURL: 'https://api.routeplane.ai/v1',
  defaultHeaders: {
    'x-routeplane-api-key': 'rp_your_gateway_key',
    'x-routeplane-provider': 'self_hosted', // route to Self-hosted (Ollama / vLLM / LocalAI)
  },
});

const completion = await client.chat.completions.create({
  model: 'llama3.1',
  messages: [{ role: 'user', content: 'Hello!' }],
});
console.log(completion.choices[0]?.message.content);

Or use the Routeplane SDK

The Routeplane SDKs subclass the official OpenAI clients, wire up the x-routeplane-* headers for you, and add typed access to routing metadata and the non-OpenAI surfaces.

bashinstall
pip install routeplane
pythonrouteplane SDK
from routeplane import Routeplane

client = Routeplane(
    api_key="rp_your_gateway_key",
    provider="self_hosted",   # route to Self-hosted (Ollama / vLLM / LocalAI)
)

resp = client.chat.completions.create(
    model="llama3.1",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)
bashinstall
npm i @routeplane/sdk
typescript@routeplane/sdk
import { Routeplane } from '@routeplane/sdk';

const client = new Routeplane({
  apiKey: process.env.ROUTEPLANE_API_KEY!, // rp_...
  provider: 'self_hosted',                 // route to Self-hosted (Ollama / vLLM / LocalAI)
});

const completion = await client.chat.completions.create({
  model: 'llama3.1',
  messages: [{ role: 'user', content: 'Hello!' }],
});
console.log(completion.choices[0]?.message.content);

Add a fallback chain

Try your local model first; fall back to OpenAI when it can’t serve. Make the provider header a comma-separated list and the gateway walks it in order, skipping any provider whose circuit is open.

bashcurl
curl https://api.routeplane.ai/v1/chat/completions \
  -H "content-type: application/json" \
  -H "x-routeplane-api-key: rp_your_gateway_key" \
  -H "x-routeplane-provider: self_hosted,openai" \
  -d '{"model":"llama3.1","messages":[{"role":"user","content":"Hello!"}]}'

Self-hosted (Ollama / vLLM / LocalAI) on the gateway

The adapter supports chat completions, native streaming, embeddings, rerank, and text-to-speech (all as faithful passthroughs to your server’s OpenAI-compatible endpoints).

Base URL is the server root — no /v1

Set SELF_HOSTED_BASE_URL to the root of your server, without a /v1 suffix — for example http://vllm.internal:8000 or Ollama’s http://localhost:11434. The adapter appends /v1/chat/completions, /v1/embeddings, and the rest for you.

Strict-runtime toggle

Some runtimes reject stream_options.include_usage. Set SELF_HOSTED_STREAM_INCLUDE_USAGE=off to omit it for those servers. Egress from the adapter is SSRF-guarded.

Sovereign routing

The model runs where you run it. Set SELF_HOSTED_REGION to the region your infrastructure sits in, and the gateway can hard-lock regulated requests to it — the strongest residency guarantee, since the data never leaves your environment.

Keep reading

Frequently asked questions

Is Routeplane a drop-in replacement for calling Self-hosted (Ollama / vLLM / LocalAI) directly?

Yes. The gateway is OpenAI-compatible, so switching is a base-URL change — point your existing OpenAI SDK at https://api.routeplane.ai/v1 and set the x-routeplane-provider header to self_hosted. You keep your code and gain automatic fallback, in-data-plane PII guardrails, sovereign routing, and per-team cost attribution.

Which Self-hosted (Ollama / vLLM / LocalAI) models can I use through Routeplane?

Any model your Self-hosted (Ollama / vLLM / LocalAI) endpoint exposes — model ids pass through the gateway verbatim.

Can I enforce data residency on Self-hosted (Ollama / vLLM / LocalAI) requests?

Yes. Add the x-routeplane-residency header (for example IN) and, when a request carries regulated personal data, the gateway restricts routing to providers eligible in that region — overriding the provider header if it has to. The Self-hosted (Ollama / vLLM / LocalAI) adapter reads SELF_HOSTED_REGION to declare where it is resident.

Route your first request this week.

Point your existing OpenAI-compatible client at Routeplane, set one header, and get fallback, guardrails, and sovereign routing across every provider.

Request access → Read the quickstart