Inference Providers

LM Studio

LM Studio is a desktop application for discovering, downloading, and running GGUF models locally. Its built-in local server exposes an OpenAI-compatible API with full tokenizer support.

Capabilities with PrivateGPT

CapabilityStatus
Model discovery (/v1/models)✅
Tokenizer endpoint (/tokenize)✅
Embeddings✅
Tool / function calling✅ model-dependent
Structured output❌
Streaming✅
Vision / image input✅ model-dependent

Setup

1

Install LM Studio

Download and install from lmstudio.ai. Available for macOS, Windows, and Linux.

2

Download models

  1. Open LM Studio.
  2. Go to the Discover tab (magnifying glass icon).
  3. Search for a model. Example:
    • LLM: search unsloth Qwen3.5-35B-A3B and pick a Q4 quantization (~18 GB)
    • Embeddings: search mxbai-embed-large
  4. Click the model and select a quantization (Q4_K_M is a good default).
  5. Click Download.
3

Start the local server

  1. Click the Developer tab (left sidebar, </> icon).
  2. Select your downloaded model from the dropdown.
  3. Click Start Server.

The default server address is http://localhost:1234.

To serve an embeddings model simultaneously, scroll down in the Developer panel and load a second model under “Embedding model”.

4

Run PrivateGPT

OPENAI_API_BASE=http://localhost:1234/v1 private-gpt serve

Advanced profile example

# settings-model.yaml
llm:
  default_model: qwen3-35b-a3b-q4_k_m
embedding:
  default_model: mxbai-embed-large-v1
models:
  - name: qwen3-35b-a3b-q4_k_m
    type: llm
    mode: openai
    context_window: 32768
    tokenizer: Qwen/Qwen3.5-35B-A3B
    support_tools: true
    support_reasoning: true
    sampling_params:
      temperature: 0.6
      top_p: 0.95
      top_k: 20
      min_p: 0.0
  - name: mxbai-embed-large-v1
    type: embedding
    mode: openai
    context_window: 512

Generate this automatically (with LM Studio server running):

OPENAI_API_BASE=http://localhost:1234/v1 \
  uv run python scripts/auto_discover_models.py --out settings-model.yaml

Troubleshooting

CORS errors from the browser

Enable CORS in LM Studio: Developer → Server Settings → Enable CORS.

Model name doesn’t match

LM Studio uses the file name as the model ID. Check the exact name with:

curl http://localhost:1234/v1/models

Use the id field from the response as the model name in your profile.