Reference

Troubleshooting

Models not discovered at startup

Symptom: PrivateGPT starts but shows no models, or API calls return a “no models available” error.

Causes and fixes:

  1. LLM server not running. Make sure your LLM server is up before starting PrivateGPT.

    $curl http://localhost:11434/v1/models   # Ollama
    $curl http://localhost:8000/v1/models    # vLLM / LlamaCPP

  2. Wrong OPENAI_API_BASE. Verify the URL matches your server’s address and includes /v1.

    $OPENAI_API_BASE=http://localhost:11434/v1 private-gpt serve

  3. Auto-discovery disabled. Check that PGPT_LLM_AUTO_DISCOVER_MODELS is not set to false.

Connection refused inside Docker

When running PrivateGPT in Docker and pointing at a server on the host machine, localhost refers to the container, not the host.

macOS / Windows: use host.docker.internal:

$docker run -p 8080:8080 \
>  -e OPENAI_API_BASE=http://host.docker.internal:11434/v1 \
>  zylonai/private-gpt:latest

Linux: use --network host instead:

$docker run --network host \
>  -e OPENAI_API_BASE=http://localhost:11434/v1 \
>  zylonai/private-gpt:latest

Context window overflow

Symptom: Long conversations or large documents produce truncated responses or errors from the LLM server.

Cause: Without a tokenizer endpoint (Ollama), PrivateGPT estimates token count at 4 chars = 1 token, which can be inaccurate.

Fix: Set context_window explicitly in a detailed model profile to a value below the model’s actual limit:

1models:
2  - name: qwen3.5:35b
3    type: llm
4    mode: openai
5    context_window: 28000   # conservative — leaves headroom

Embedding dimensions mismatch

Symptom: Error message containing Embedding dimensions mismatch during ingestion or retrieval.

Cause: The vector store was initialized with a different embedding model (different output dimensions) than the one currently configured.

Fix: Either:

  • Switch back to the original embedding model, or
  • Wipe the vector store and re-ingest your documents: 
    $make wipe

Ensure embed_dim in your profile matches the model’s output dimension (e.g. 1024 for mxbai-embed-large):

1vectorstore:
2  embed_dim: 1024

Tokenizer download fails (HuggingFace gated models)

Symptom: Startup fails with an authentication error when downloading a tokenizer from HuggingFace.

Cause: The model’s tokenizer repository is gated and requires a HuggingFace token.

Fix:

  1. Request access to the model on HuggingFace.
  2. Generate an access token.
  3. Pass the token via environment variable: 
    $PGPT_HUGGINGFACE_TOKEN=hf_... private-gpt serve
    Or set it in your profile: 
    1huggingface:
    2  access_token: hf_...

Port already in use

Symptom: [Errno 48] Address already in use on startup.

Fix: Change the port with the --port flag:

$private-gpt serve --port 8081

Or for Docker:

$docker run -p 8081:8080 -e OPENAI_API_BASE=... zylonai/private-gpt:latest

Profile file not found

Symptom: FileNotFoundError: Settings file not found for profile 'foo'.

Cause: PGPT_PROFILES=foo was set but settings-foo.yaml does not exist in the settings folder.

Fix: Make sure the file exists and is in the correct location. For Docker, it must be mounted:

$-v ./settings-foo.yaml:/home/worker/app/settings-foo.yaml