How to Run LocalAI on Linux: OpenAI-Compatible Local Server

Tested on: Ubuntu 26.04 LTS · Ubuntu 24.04 LTS · Debian 12 (NVIDIA RTX 3070 and CPU-only configurations) — Last updated: June 2026

LocalAI is a free, open-source server that exposes an OpenAI-compatible REST API backed entirely by your own hardware. Any application or library that talks to the OpenAI API — including the official Python openai package, LangChain, LlamaIndex, and AutoGen — works with LocalAI by changing a single base URL. No API keys, no rate limits, no data leaving your machine.

Prerequisites

• Docker installed and running (docker --version should return 24.x or newer)
• For GPU acceleration: NVIDIA driver 525+ installed (nvidia-smi works from the host)
• At minimum 8 GB RAM for small 3B–7B models; 16 GB+ for comfortable 13B inference
• Disk space: plan for 4–8 GB per quantized model file
• Basic familiarity with curl and reading JSON output

Why LocalAI Instead of Just Ollama

If you only want to run a chat model locally, Ollama is faster to set up. LocalAI's specific advantage is full OpenAI API drop-in compatibility — including endpoints Ollama doesn't expose, such as /v1/images/generations for Stable Diffusion and /v1/audio/transcriptions for Whisper. If your codebase already uses the OpenAI SDK and you want to point it at your own server without touching application code, LocalAI is the right tool.

Hardware Expectations

Step 1 — Install with Docker

CPU-Only Setup

# Create a persistent directory for model files
mkdir -p ~/localai/models

# Pull and run the CPU image
docker run -d 
  --name localai 
  -p 8080:8080 
  -v ~/localai/models:/models 
  -e MODELS_PATH=/models 
  localai/localai:latest-cpu

# Tail startup logs — wait for "LocalAI API is listening"
docker logs -f localai

Startup on first run takes 10–30 seconds. You'll see the ready message:

localai  | time="..." level=info msg="LocalAI API is listening on 0.0.0.0:8080"

NVIDIA GPU Setup

If you haven't installed the NVIDIA Container Toolkit yet, do that first:

distribution=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey 
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list 
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' 
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo systemctl restart docker

# Run LocalAI with CUDA 12 support
docker run -d 
  --name localai 
  --gpus all 
  -p 8080:8080 
  -v ~/localai/models:/models 
  -e MODELS_PATH=/models 
  localai/localai:latest-cublas-cuda12-core

# Confirm GPU is detected in logs
docker logs localai 2>&1 | grep -i "cuda|gpu|nvidia" | head -10

Expected output includes something like:

localai  | time="..." level=info msg="CUDA device found: NVIDIA GeForce RTX 3070"

Docker Compose (Recommended for Persistent Deployments)

Create ~/localai/docker-compose.yml:

version: '3.8'
services:
  localai:
    image: localai/localai:latest-cublas-cuda12-core
    container_name: localai
    ports:
      - "8080:8080"
    volumes:
      - ./models:/models
    environment:
      - MODELS_PATH=/models
      - THREADS=8
      - CONTEXT_SIZE=4096
      # Optional: preload a model on startup
      - PRELOAD_MODELS=llama-3-8b-instruct
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      interval: 30s
      timeout: 10s
      retries: 5

cd ~/localai
docker compose up -d
docker compose logs -f

Install Without Docker (Binary)

Useful for bare-metal installs or when Docker overhead matters:

LOCALAI_VERSION=v2.22.0

curl -Lo localai 
  "https://github.com/mudler/LocalAI/releases/download/${LOCALAI_VERSION}/local-ai-Linux-x86_64"

chmod +x localai
sudo mv localai /usr/local/bin/local-ai

# Verify
local-ai --version

# Create model directory and run
mkdir -p ~/localai/models
local-ai 
  --models-path ~/localai/models 
  --address 0.0.0.0:8080 
  --threads $(nproc)

For GPU builds, download the CUDA variant from the releases page: local-ai-Linux-x86_64-cuda12. The binary is self-contained; no separate CUDA install is required beyond the NVIDIA driver.

Step 2 — Load Models

Method 1: Gallery API (Easiest)

LocalAI maintains a curated model gallery with pre-written configs. Browse available models:

curl -s http://localhost:8080/models/available 
  | python3 -m json.tool 
  | grep '"id"' 
  | head -20

Install a model from the gallery — LocalAI handles the download and config automatically:

# Install Llama 3 8B Instruct from the gallery
curl http://localhost:8080/models/apply 
  -H "Content-Type: application/json" 
  -d '{"id": "huggingface@thebloke/llama-2-7b-chat-gguf/llama-2-7b-chat.Q4_K_M.gguf"}'

# Poll until the job is complete (status: "succeeded")
curl -s http://localhost:8080/models/jobs | python3 -m json.tool

Method 2: Manual GGUF Model with YAML Config

For models not in the gallery, or when you need custom parameters:

cd ~/localai/models

# Download Llama 3 8B Instruct Q4_K_M (~4.7 GB, good quality/speed balance)
wget https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF/resolve/main/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf

Create the corresponding YAML config. The filename without extension becomes the model's API name:

# ~/localai/models/llama3-8b.yaml
name: llama3-8b
parameters:
  model: Meta-Llama-3-8B-Instruct.Q4_K_M.gguf
  top_k: 80
  top_p: 0.95
  temperature: 0.7
  max_tokens: 2048
context_size: 4096
threads: 8
f16: true
template:
  chat_message: |
    <|start_header_id|>{{.RoleName}}<|end_header_id|>

    {{.Content}}<|eot_id|>
  chat: |
    <|begin_of_text|>{{.Input}}<|start_header_id|>assistant<|end_header_id|>

  completion: |
    {{.Input}}

Restart LocalAI (or just wait — it hot-reloads configs in Docker). Verify the model appears:

curl -s http://localhost:8080/v1/models | python3 -m json.tool

Step 3 — Use the API

Chat Completion via curl

curl http://localhost:8080/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{
    "model": "llama3-8b",
    "messages": [
      {"role": "system", "content": "You are a concise Linux expert."},
      {"role": "user", "content": "What does the -z flag do in bash conditional expressions?"}
    ],
    "temperature": 0.7,
    "max_tokens": 512
  }' | python3 -m json.tool

Python — Drop-In OpenAI Replacement

pip install openai

from openai import OpenAI

# Point the client at LocalAI — api_key can be any non-empty string
client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="localai"
)

response = client.chat.completions.create(
    model="llama3-8b",
    messages=[
        {"role": "system", "content": "You are a Linux expert."},
        {"role": "user", "content": "Explain inode exhaustion and how to diagnose it."}
    ],
    temperature=0.7
)

print(response.choices[0].message.content)

That same code, unmodified, works against the real OpenAI API if you swap the base_url and api_key. That's the entire value proposition.

Streaming Responses

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="localai")

stream = client.chat.completions.create(
    model="llama3-8b",
    messages=[{"role": "user", "content": "List 5 useful awk one-liners."}],
    stream=True
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

print()

Embeddings

# First install a dedicated embedding model (smaller, faster than LLMs)
curl http://localhost:8080/models/apply 
  -H "Content-Type: application/json" 
  -d '{"id": "bert-embeddings"}'

# Generate embeddings
curl http://localhost:8080/v1/embeddings 
  -H "Content-Type: application/json" 
  -d '{
    "model": "bert-embeddings",
    "input": "LocalAI is an OpenAI-compatible local inference server."
  }' | python3 -m json.tool | head -20

Image Generation

# Download Stable Diffusion 1.5
cd ~/localai/models
wget https://huggingface.co/runwayml/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.safetensors

# ~/localai/models/stablediffusion.yaml

name: stablediffusion

backend: diffusers

parameters:



Further Reading

How to Install Ollama on Linux
How to Install Open WebUI with Docker
Running AI Agents on Linux Servers
How to Install Docker on Linux