Inference Providers

OrcaRouter

OrcaRouter is a model routing gateway with an OpenAI-compatible endpoint at https://api.orcarouter.ai/v1. It fronts models from multiple providers (OpenAI, Anthropic, Google, DeepSeek, Qwen, and others) behind a single API key, and can route each request to the best upstream automatically. It also runs gateway-level, zero-trust security for AI agents on the same endpoint — screening every prompt/response and governing every tool call on a default-deny basis.

Capabilities with PrivateGPT

CapabilityStatus
Model discovery (/v1/models)✅
Tokenizer endpoint (/tokenize)❌
Embeddings✅
Tool / function calling✅ model-dependent
Structured output✅ model-dependent
Streaming✅
Vision / image input✅ model-dependent

Setup

1

Get an OrcaRouter API key

  1. Sign up at orcarouter.ai.
  2. Go to the console and create an API key. Keys start with sk-orca-.
  3. Note the model IDs you want to use — OrcaRouter model IDs are namespaced, e.g. openai/gpt-5.5 or anthropic/claude-sonnet-5.
2

Run PrivateGPT

$OPENAI_API_BASE=https://api.orcarouter.ai/v1 \
>  OPENAI_API_KEY=your-orcarouter-api-key \
>  private-gpt serve

Store the API key in an .env file or use OPENAI_API_KEY as an environment variable to avoid exposing it in shell history.

Advanced profile example

1# settings-model.yaml
2llm:
3  default_model: openai/gpt-5.5
4
5models:
6  - name: openai/gpt-5.5
7    type: llm
8    mode: openai
9    context_window: 128000
10    support_tools: true
11    support_reasoning: true
12    sampling_params:
13      temperature: 0.7
14
15  - name: anthropic/claude-sonnet-5
16    type: llm
17    mode: openai
18    context_window: 200000
19    support_tools: true
20    support_reasoning: true
21    sampling_params:
22      temperature: 0.7
23
24  - name: google/gemini-2.5-pro
25    type: llm
26    mode: openai
27    context_window: 1048576
28    support_tools: true
29    support_reasoning: true
30    sampling_params:
31      temperature: 0.7

Run with a profile:

$OPENAI_API_BASE=https://api.orcarouter.ai/v1 \
>  OPENAI_API_KEY=your-orcarouter-api-key \
>  PGPT_PROFILES=model \
>  uv run python -m private_gpt

settings.yaml override

You can also set the endpoint directly in settings.yaml instead of environment variables:

1openai:
2  api_base: https://api.orcarouter.ai/v1
3  api_key: ${ORCAROUTER_API_KEY:}

Notes

  • Model IDs are namespaced by upstream provider (openai/, anthropic/, google/, deepseek/, qwen/, …). See the full catalog at orcarouter.ai/models.
  • Embeddings are available through /v1/embeddings, e.g. openai/text-embedding-3-small.
  • Because OrcaRouter does not expose /tokenize, set context_window explicitly in your model profiles for accurate token management.
  • The routing model orcarouter/auto picks an upstream per request based on task type and difficulty. For deterministic structured output, prefer a fixed model such as openai/gpt-5.5 instead.