RunPod: GPU Cloud for AI Workloads on Linux

Tested on: RunPod Community Cloud and Secure Cloud · Ubuntu 24.04 container images · CUDA 12.1 — Last updated: June 2026

RunPod is a GPU cloud platform built for AI workloads: rent NVIDIA A40s, A100s, H100s, or RTX 4090s by the hour with full Linux access via SSH. It undercuts AWS and GCP on GPU pricing by aggregating idle capacity from independent data centers. This guide covers everything from provisioning your first pod to automating deployments with the API, running Ollama and Stable Diffusion, fine-tuning LLMs, and keeping your bill under control.

Prerequisites

• A RunPod account at runpod.io with prepaid credits ($10–20 to start)
• An SSH key pair on your local machine (~/.ssh/id_ed25519 or RSA)
• Basic familiarity with SSH and Linux terminal usage
• Python 3.10+ locally if you plan to use the RunPod Python SDK

Why RunPod for AI Workloads

Consumer GPUs top out at 24 GB VRAM (RTX 4090). A 70B parameter model in fp16 needs roughly 140 GB just to load — impossible on a single consumer card. RunPod gives you access to A100 SXM (80 GB) or multi-GPU configurations without the capital expenditure. The spot pricing model makes it viable even for experimentation:

Prices fluctuate based on availability. Community Cloud is cheaper but uses third-party hardware with no data-at-rest encryption. Secure Cloud runs in RunPod-operated facilities with stronger isolation — use it for proprietary datasets or compliance-sensitive workloads.

Account Setup and SSH Keys

After creating your account and adding credits, register your SSH public key before creating any pods — this is how you'll authenticate to all pods automatically:

# Display your public key:
cat ~/.ssh/id_ed25519.pub
# ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA... user@machine

# If you don't have a key yet, generate one:
ssh-keygen -t ed25519 -C "runpod-access"

Paste the output into runpod.io → Account → Settings → SSH Public Keys. Every pod you create will automatically accept this key — no per-pod configuration needed.

Creating and Configuring a Pod

Navigate to Pods → Deploy. The key decisions:

GPU type: RTX 4090 is the best starting point for inference and image generation. For fine-tuning, step up to A100 PCIe (40 GB) or A100 SXM (80 GB).

Template: RunPod's official templates save significant setup time:

• RunPod PyTorch — PyTorch + CUDA pre-installed, Python 3.11. Use this for custom ML workloads.
• RunPod Stable Diffusion — AUTOMATIC1111 WebUI pre-configured with port mapping.
• Ubuntu 24.04 — Clean slate for custom Docker setups.

Disk sizing: Container disk is ephemeral local storage on the host. Set it to 20–30 GB for the OS and working files. Do not store models here — they disappear when the pod is terminated. Use network (persistent) storage for models and datasets.

Network storage: Add a persistent volume at pod creation time. It mounts at /workspace by default and survives pod restarts and terminations. At ~$0.07/GB/month, a 100 GB volume costs $7/month — far cheaper than re-downloading 40 GB model weights every session.

Connecting via SSH

RunPod does not expose pods with direct IP addresses. SSH connections route through a RunPod gateway with a unique port per pod. After your pod reaches Running status (typically 1–3 minutes):

# Get the exact command from: Pod card → Connect → SSH over exposed TCP
# It looks like this:
ssh root@ssh.runpod.io -p 14823   # port number is unique to your pod

# With explicit key:
ssh -i ~/.ssh/id_ed25519 root@ssh.runpod.io -p 14823

# Verify GPU on connect:
nvidia-smi

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.08   Driver Version: 535.161.08   CUDA Version: 12.2    |
|-------------------------------+----------------------+----------------------+
| GPU  Name             Temp  Perf  Pwr:Usage/Cap  Memory-Usage  GPU-Util    |
|   0  NVIDIA A40         38C    P0    72W / 300W   0MiB / 46068MiB   0%     |
+-----------------------------------------------------------------------------+

# Verify PyTorch + CUDA (on PyTorch template):
python3 -c "import torch; print(torch.cuda.is_available(), torch.version.cuda, torch.cuda.get_device_name(0))"
# True 12.1 NVIDIA A40

Add a ~/.ssh/config entry locally to avoid retyping the port each session:

Host runpod-current
    HostName ssh.runpod.io
    Port 14823
    User root
    IdentityFile ~/.ssh/id_ed25519

Then connect with ssh runpod-current. Update the port when you create a new pod.

Running Ollama on RunPod

Ollama on a cloud GPU gives you access to 70B+ models that don't fit in consumer VRAM. An RTX 4090 (24 GB) handles 13B–30B models comfortably. For llama3.1:70b, use an A40 (48 GB) or A100 SXM (80 GB):

# Install Ollama:
curl -fsSL https://ollama.com/install.sh | sh

# Start the Ollama service:
ollama serve &

# Pull a model — this saves to /workspace/.ollama/ if you set OLLAMA_MODELS:
export OLLAMA_MODELS=/workspace/.ollama
ollama pull llama3.1:70b    # ~40 GB download; only once if using persistent storage
ollama pull llama3.1:8b     # ~5 GB for quick testing

# Interactive session:
ollama run llama3.1:8b "Explain LoRA fine-tuning in two paragraphs"

# API endpoint (runs on localhost:11434 by default):
curl http://localhost:11434/api/generate 
  -H "Content-Type: application/json" 
  -d '{"model": "llama3.1:8b", "prompt": "What is RLHF?", "stream": false}' | jq .response

To expose the Ollama API publicly, add a TCP port mapping in pod settings (Port: 11434), then access it at the public URL shown in Connect → TCP Port Mappings.

Stable Diffusion on RunPod

The fastest path is the official RunPod SD template, which ships AUTOMATIC1111 with port 3001 pre-mapped. Deploy it and click Connect → HTTP Port 3001 to open the WebUI in your browser. No further configuration needed.

For ComfyUI (more flexible, lower memory overhead):

# Store everything in persistent storage:
cd /workspace

git clone https://github.com/comfyanonymous/ComfyUI.git
cd ComfyUI
pip install -r requirements.txt

# Download a base model to the correct path:
mkdir -p models/checkpoints
wget -q -O models/checkpoints/sd_xl_base_1.0.safetensors 
  https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/resolve/main/sd_xl_base_1.0.safetensors

# Launch — bind to all interfaces so RunPod's port proxy can reach it:
python main.py --listen 0.0.0.0 --port 8188

Add port 8188 to your pod's TCP port mappings, then access via the exposed URL. On an RTX 4090, SDXL generates a 1024×1024 image in under 4 seconds.

Fine-Tuning LLMs with Unsloth

Unsloth reduces VRAM usage by 60–70% versus naive QLoRA, making fine-tuning feasible on smaller GPUs. This example fine-tunes Llama 3.1 8B on a custom JSONL dataset:

pip install unsloth trl datasets

from unsloth import FastLanguageModel
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

# Load model with 4-bit quantization — fits in 16 GB VRAM:
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Meta-Llama-3.1-8B-Instruct",
    max_seq_length=2048,
    load_in_4bit=True,
    dtype=None,   # auto-detect
)

# Attach LoRA adapters:
model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=32,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing="unsloth",
)

# Your training data: JSONL with {"text": "..."} entries
dataset = load_dataset("json", data_files="/workspace/data/train.jsonl", split="train")

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    args=SFTConfig(
        output_dir="/workspace/checkpoints",
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=10,
        max_steps=200,
        learning_rate=2e-4,
        fp16=True,
        logging_steps=10,
        save_steps=100,
    ),
)
trainer.train()

# Save the merged model to persistent storage:
model.save_pretrained_merged("/workspace/llama-3.1-8b-finetuned", tokenizer)

On an RTX 4090 (24 GB), this runs at roughly 3–4 it/s for 2048-token sequences. 200 steps takes about 10 minutes at $0.69/hr — under $0.12 in compute cost.

RunPod API and CLI Automation

Managing pods manually via the web UI doesn't scale. The RunPod CLI and Python SDK let you automate provisioning from scripts or CI pipelines:

# Install runpodctl:
wget -q https://github.com/runpod/runpodctl/releases/latest/download/runpodctl-linux-amd64 
  -O runpodctl
chmod +x runpodctl
sudo mv runpodctl /usr/local/bin/runpodctl

# Authenticate:
runpodctl config --apiKey YOUR_API_KEY_HERE

# List running pods:
runpodctl get pod

# Start a stopped pod:
runpodctl start pod abc123def456

# Stop a pod (stops compute billing, keeps storage):
runpodctl stop pod abc123def456

For programmatic pod creation from Python (useful for batch job orchestration):

pip install runpod

import runpod

runpod.api_key = "YOUR_API_KEY_HERE"

# Create a pod:
pod = runpod.create_pod(
    name="llm-finetune-job",
    image_name="runpod/pytorch:2.2.0-py3.11-cuda12.1.1-devel-ubuntu22.04",
    gpu_type_id="NVIDIA A40",
    cloud_type="COMMUNITY",
    gpu_count=1,
    volume_in_gb=100,
    container_disk_in_gb=20,
    ports="22/tcp",
    volume_mount_path="/workspace",
)
print(f"Pod ID: {pod['id']}, Status: {pod['desiredStatus']}")

# Stop it when done:
runpod.stop_pod(pod["id"])

RunPod Serverless is worth evaluating for inference APIs. You package your model as a handler function, push a Docker image, and RunPod autoscales workers from zero. You pay only for execution time, not idle GPU time — a significant cost difference for low-traffic inference endpoints.

Cost-

No GPU Needed?
Hostinger VPS — From €3.99/month

RunPod is ideal for GPU-intensive AI workloads, but if you only need a Linux server for hosting, APIs, or running lightweight models like Ollama with small quantizations, a standard VPS is far cheaper. Hostinger VPS starts at €3.99/month with full root access and Ubuntu 26.04 support.

• Use RunPod for: training, inference with large models (7B+), image generation
• Use Hostinger VPS for: hosting, APIs, small models (1B–3B), development
• KVM virtualization — 1 vCPU / 4 GB RAM / 50 GB NVMe from €3.99/month
• Ubuntu 26.04, Debian 12, AlmaLinux, CentOS supported

Get Hostinger VPS →

Affiliate disclosure: we earn a small commission if you sign up — at no extra cost to you.

---

Further Reading

• Best Linux VPS Hosting in 2026
• How to Install Ollama on Linux
• How to Run Stable Diffusion on Linux
• Running AI Agents on Linux Servers

$ share:XFacebookLinkedInRedditPinterestHacker NewsCopy link