> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.privategpt.dev/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.privategpt.dev/_mcp/server.

# Overview

> PrivateGPT works with any OpenAI-compatible server. This page covers common local setups.

PrivateGPT connects to **any OpenAI-compatible LLM server** via `OPENAI_API_BASE`. If your server responds to `GET /v1/models` and `POST /v1/chat/completions`, it works — whether that is a local binary, a cloud endpoint, or a self-hosted service.

```bash
OPENAI_API_BASE=https://your-openai-compatible-server/v1 private-gpt serve
```

The server handles model inference; PrivateGPT handles the API, retrieval, document processing, and orchestration on top.

---

## Common local setups

The guides below cover popular self-hosted options. These are examples — not an exhaustive list.

#### [Ollama](/providers/ollama)

Easiest local setup. One command to pull and run any model.

#### [LM Studio](/providers/lmstudio)

GUI-based desktop app. Great for exploring and switching models without a terminal.

#### [LlamaCPP Server](/providers/llamacpp)

Lightweight binary, full tokenizer support. Best for CPU inference and GGUF models.

#### [vLLM](/providers/vllm)

Highest throughput. Structured output support. Best for production and multi-user deployments.

## Cloud gateways

#### [DaoXE](/providers/daoxe)

Multi-model gateway. Access GPT, Claude, Grok, GLM families through one OpenAI-compatible endpoint.

#### [OrcaRouter](/providers/orcarouter)

Model routing gateway. Access 150+ models from OpenAI, Anthropic, Google, DeepSeek and more through one endpoint.

---

## Feature matrix

| Capability                       |   Ollama   | LM Studio | LlamaCPP Server | vLLM |
| -------------------------------- | :--------: | :-------: | :-------------: | :--: |
| Model discovery (`/v1/models`)   |      ✅     |     ✅     |        ✅        |   ✅  |
| Tokenizer endpoint (`/tokenize`) |      ❌     |     ✅     |        ✅        |   ✅  |
| Embeddings endpoint              |      ✅     |     ✅     |        ✅        |   ✅  |
| Tool / function calling          |     ✅ †    |    ✅ †    |       ✅ †       |  ✅ † |
| Structured output (JSON schema)  |      ❌     |     ❌     |        ❌        |   ✅  |
| Streaming                        |      ✅     |     ✅     |        ✅        |   ✅  |
| Vision / image input             |     ✅ †    |    ✅ †    |       ✅ †       |  ✅ † |
| Audio input                      | ⚙️ Limited |     ❌     |        ❌        |   ❌  |

† Model-dependent — the server supports the protocol, but the loaded model must also support the capability.

### Impact of a missing tokenizer endpoint

> **Warning**
>
> When the server does not expose `/tokenize` (Ollama), PrivateGPT falls back to a **character-based estimate (4 chars = 1 token)** for counting tokens. This can cause:
>
> * Inaccurate context-window management on very long inputs
> * Potential context overflow for models with smaller windows (e.g. 4k, 8k)
>
> **Mitigation:** Set `context_window` explicitly in a [detailed model profile](/configuration/advanced) to a conservative value. This tells PrivateGPT exactly how many tokens it can safely use.

### Structured output

Only **vLLM** exposes the structured output (JSON schema enforcement) endpoint used by PrivateGPT for reliable tool calls and schema-constrained responses. With other providers, PrivateGPT falls back to prompt-based JSON extraction, which is less reliable for complex schemas.

---

## Example models

The provider pages use the following models as examples. Any OpenAI-compatible model works.

| Role       | Model                                                                                                    | Size                                 | Notes                                             |
| ---------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------ | ------------------------------------------------- |
| LLM        | `qwen3.5:35b` (Ollama) / `unsloth/Qwen3.5-35B-A3B-GGUF` (GGUF) / `Qwen/Qwen3.5-35B-A3B-GPTQ-Int4` (vLLM) | \~24 GB (Ollama) / \~18 GB (Q4 GGUF) | Mixture-of-experts; strong reasoning and tool use |
| Embeddings | `mxbai-embed-large` (Ollama) / `mixedbread-ai/mxbai-embed-large-v1`                                      | \~670 MB                             | 1024-dim, strong multilingual retrieval           |

---

## Embedding auto-discovery

> **Note**
>
> Embedding models are auto-discovered from `/v1/models` when `embedding.auto_discover_models` is enabled, which is the default behavior. You only need to define embedding models explicitly in a [detailed model profile](/configuration/advanced) if you want to override discovery or your provider does not expose them as expected.

Example manual embedding model config in `settings-model.yaml`:

```yaml
embedding:
  default_model: mxbai-embed-large

models:
  - name: mxbai-embed-large
    type: embedding
    mode: openai
    context_window: 512
```