AI & Inference

Companion Hub includes built-in support for running and routing AI models. The Onboarding Wizard configures your inference backend, recommended models, and optional cloud API keys on first login.

Inference backends

Hub detects your hardware (CPU, RAM, GPU vendor, VRAM) and recommends models that fit. You can override preferences in Settings → AI.

Ollama on your host

By default, Hub connects to Ollama on your host machine at port 11434. Containers reach it via host.docker.internal:11434.

Install Ollama

Download from ollama.com  or:

curl -fsSL https://ollama.com/install.sh | sh

Pull a model:

ollama pull llama3.2
# or via Hub CLI:
cihub models install llama3.2
cihub models list

Bind Ollama to all interfaces

Ollama listens on 127.0.0.1 by default. Docker containers need it reachable on the host network interface. Set:

# Linux (systemd override)
sudo systemctl edit ollama

Add:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Then restart:

sudo systemctl restart ollama

On macOS/Windows with Docker Desktop, host.docker.internal usually works without changing the bind address.

Confirm containers can reach Ollama:

docker run --rm curlimages/curl -s http://host.docker.internal:11434/api/tags

A JSON object listing your installed models means the connection works. Connection refused means Ollama is not bound to all interfaces — repeat the step above.

GPU acceleration

NVIDIA (CUDA): Install the NVIDIA Container Toolkit  so Docker can pass GPU devices. Ollama auto-detects CUDA when drivers are installed.

AMD (ROCm/HIP): Install ROCm drivers for your GPU. Ollama supports AMD GPUs on Linux with ROCm.

Verify the GPU is actually being used — load a model, then check the PROCESSOR column while it is resident:

ollama ps

100% GPU means acceleration is working. 100% CPU means Ollama fell back — the model may be too large for your VRAM, or the drivers are not visible to Ollama.

Hub’s hardware inspector reads GPU info and tags recommended models accordingly in the wizard. Cross-check what Hub detected under Settings → System, or see the System Inspector.

vLLM on your host

Use vLLM when you want high-throughput chat/decoding on a GPU. Hub does not run vLLM inside its stack — you start it on the host (or point at a remote server), then wire it in Settings → AI.

vLLM serves an OpenAI-compatible API at /v1. Hub probes it from inside Docker, so the endpoint must be reachable from containers:

Quick start (NVIDIA)

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --api-key vllm-local

Verify from the host:

curl -s http://127.0.0.1:8000/v1/models -H "Authorization: Bearer vllm-local"

In Hub Settings → AI:

1. Select vLLM as the chat backend.
2. Endpoint URL: http://host.docker.internal:8000 (or leave blank to use the default).
3. API key: must exactly match --api-key (e.g. vllm-local).
4. Click Re-check — the card should turn green when Hub sees your model.
5. Keep Ollama running for embeddings (Companion Memory needs nomic-embed-text or similar).

Container-side check (same path Hub uses):

docker run --rm curlimages/curl -s \
  http://host.docker.internal:8000/v1/models \
  -H "Authorization: Bearer vllm-local"

Standardized inference variables for apps

Apps that use AI can opt in to standardized Hub inference variables via hub_integration.inference in their config.json. At install time, Hub resolves the correct values based on your settings and hardware, then writes them into the app’s app.env.

Hub variable keys

Example in config.json

Map Hub-resolved values to the env variable names your app expects:

{
  "hub_integration": {
    "inference": {
      "llm_base_url": "LLM_API_BASE",
      "llm_api_key": "LLM_API_KEY",
      "chat_model": "LLM_DEFAULT_CHAT_MODEL",
      "embedding_model": "LLM_DEFAULT_EMBEDDING_MODEL",
      "ollama_host": "OLLAMA_HOST"
    }
  }
}

Apps without hub_integration.inference receive no AI variables — zero overhead for non-AI apps.

Resolution order

For each variable, Hub resolves in this order:

1. Your Hub-wide preference (Settings → AI)
2. Hardware-aware recommendation from the model registry
3. Omitted — the app must handle the variable being absent

If Ollama is unavailable and no cloud provider is configured, Hub omits all inference variables and logs a warning.

MCP integration for agent apps

Agent apps (like OpenClaw) can declare hub_integration.mcp_client: true to receive Hub MCP endpoints:

Enable MCP on your Hub:

cihub mcp setup
cihub mcp config

See Companion Agent for the agent ecosystem.

CLI model management

cihub models list
cihub models install mistral
cihub models rm mistral
cihub status    # shows installed models in the Models section