Skip to main content

Using OpenAI as the LLM backend

Starting from v1.9.0, you can route chat generation to an OpenAI-compatible hosted API instead of the bundled local inference container.

When to Use This Mode​

Common reasons:

  • you do not have a local GPU available
  • you want to use GPT-4-class hosted models
  • you want fast setup without local model management

Configuration​

Set these variables in your .env file:

LLM_BACKEND=openai
OPENAI_API_KEY=<your-api-key>
OPENAI_MODEL=gpt-4o-mini
# Optional for OpenAI-compatible providers:
# OPENAI_BASE_URL=https://...

Required in this mode:

  • LLM_BACKEND=openai
  • OPENAI_API_KEY
  • OPENAI_MODEL

Optional:

  • OPENAI_BASE_URL for compatible APIs (for example Azure OpenAI or local OpenAI-compatible gateways)

Inference Container Behavior​

When LLM_BACKEND=openai, the bundled inference container is not required for chat generation and can be turned off in your compose deployment.

Embeddings still run locally in the backend pipeline.

Privacy Note​

In OpenAI backend mode, chat request/response content leaves your infrastructure and is sent to your configured OpenAI-compatible endpoint.

If strict data residency is required, keep LLM_BACKEND=local.

/api/model-info in OpenAI Mode​

GET /api/model-info now returns metadata immediately in OpenAI mode. The UI no longer waits on a local inference runtime in this configuration.

See also: Environment Variables.