Prometheus and Grafana: Linux Server Monitoring Guide

Tested on: Ubuntu 26.04 LTS · Debian 12 · Rocky Linux 10 — Last updated: June 2026

Prometheus scrapes metrics from your servers and applications on a configurable interval, stores them as time series, and exposes a powerful query language (PromQL) for analysis. Grafana connects to Prometheus as a data source and turns those queries into dashboards, graphs, and alerts. Together they form the de facto open-source monitoring stack for Linux — used everywhere from single-server homelab setups to thousand-node production clusters.

Prerequisites

• Ubuntu 26.04, Debian 12, or Rocky Linux 10 with sudo access
• Ports 9090 (Prometheus), 9100 (node_exporter), and 3000 (Grafana) open in your firewall
• wget, curl, and tar installed
• At least 2 GB RAM and 10 GB free disk for a single-server setup
• Basic familiarity with systemd and editing config files

Architecture Overview

Before running any commands, understand the data flow. Prometheus is a pull-based system — it reaches out to scrape targets rather than waiting for agents to push data in. Each target exposes a /metrics HTTP endpoint. Node exporter is the agent that exposes Linux OS metrics on that endpoint.

Prometheus Server (:9090)
├── Scrapes /metrics every 15s from:
│   ├── node_exporter :9100   → CPU, RAM, disk, network, filesystem
│   ├── nginx_exporter :9113  → request rates, status codes, upstreams
│   ├── postgres_exporter :9187 → query latency, connections, cache hits
│   └── app :8080/metrics     → custom application metrics
├── Evaluates alert rules every 15s
├── Fires alerts → Alertmanager (:9093) → Slack/PagerDuty/email
└── Stores time series in /var/lib/prometheus/ (default 15-day retention)

Grafana (:3000)
└── Queries Prometheus via HTTP API
    └── Renders dashboards, graphs, stat panels, alert annotations

Prometheus and node_exporter can run on the same server or on separate hosts. For monitoring multiple servers, install node_exporter on each target and point Prometheus at all of them from a single central instance.

Install Prometheus

Prometheus ships as a static binary — no package manager required, which means you always get the latest version without waiting for distro repositories.

# Create a dedicated system user (no login shell, no home directory)
sudo useradd --system --no-create-home --shell /usr/sbin/nologin prometheus

# Fetch the latest release version tag from GitHub API
PROM_VER=$(curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest 
  | grep '"tag_name"' | cut -d'"' -f4 | tr -d 'v')
echo "Installing Prometheus ${PROM_VER}"

# Download and extract
wget https://github.com/prometheus/prometheus/releases/download/v${PROM_VER}/prometheus-${PROM_VER}.linux-amd64.tar.gz
tar xzf prometheus-${PROM_VER}.linux-amd64.tar.gz
cd prometheus-${PROM_VER}.linux-amd64

# Install binaries
sudo cp prometheus promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

# Install config directories and console files
sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo cp -r consoles/ console_libraries/ /etc/prometheus/
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

Create the systemd unit file:

sudo tee /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus Monitoring System
Documentation=https://prometheus.io/docs/
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/prometheus 
    --config.file=/etc/prometheus/prometheus.yml 
    --storage.tsdb.path=/var/lib/prometheus/ 
    --storage.tsdb.retention.time=15d 
    --web.console.templates=/etc/prometheus/consoles 
    --web.console.libraries=/etc/prometheus/console_libraries 
    --web.listen-address=0.0.0.0:9090 
    --web.enable-lifecycle

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus

The --web.enable-lifecycle flag enables the /-/reload HTTP endpoint so you can reload configuration without restarting the process. Verify it's up:

curl -s http://localhost:9090/-/healthy
# Expected output:
Prometheus Server is Healthy.

sudo systemctl status prometheus
# Should show: Active: active (running)

Install Node Exporter

Node exporter exposes over 1,000 Linux kernel and system metrics. Install it on every server you want to monitor — including the Prometheus server itself.

sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter

NODE_VER=$(curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest 
  | grep '"tag_name"' | cut -d'"' -f4 | tr -d 'v')

wget https://github.com/prometheus/node_exporter/releases/download/v${NODE_VER}/node_exporter-${NODE_VER}.linux-amd64.tar.gz
tar xzf node_exporter-${NODE_VER}.linux-amd64.tar.gz
sudo cp node_exporter-${NODE_VER}.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

sudo tee /etc/systemd/system/node_exporter.service << 'EOF'
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
RestartSec=5s
ExecStart=/usr/local/bin/node_exporter 
    --collector.systemd 
    --collector.processes

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

Confirm the metrics endpoint is working — it should return thousands of lines:

curl -s http://localhost:9100/metrics | head -25
# Expected output (excerpt):
# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 12345.67
node_cpu_seconds_total{cpu="0",mode="user"} 234.56
# HELP node_memory_MemTotal_bytes Memory information field MemTotal_bytes.
# TYPE node_memory_MemTotal_bytes gauge
node_memory_MemTotal_bytes 8.372838e+09

Configure Prometheus to Scrape Targets

Write the main Prometheus configuration. This file controls which targets get scraped, how often, and where to find alert rules.

sudo tee /etc/prometheus/prometheus.yml << 'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: 'production'

alerting:
  alertmanagers:
    - static_configs:
        - targets: []   # add 'localhost:9093' when Alertmanager is installed

rule_files:
  - "rules/*.yml"

scrape_configs:
  # Prometheus self-monitoring
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Local server OS metrics
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
        labels:
          hostname: 'primary'

  # Additional servers — add as many as needed
  - job_name: 'remote_nodes'
    static_configs:
      - targets:
          - '192.168.1.10:9100'
          - '192.168.1.11:9100'
        labels:
          environment: 'production'
          datacenter: 'fra1'
EOF

Always validate the config before reloading — a syntax error will prevent Prometheus from starting:

promtool check config /etc/prometheus/prometheus.yml
# Expected output:
# Checking /etc/prometheus/prometheus.yml
#   SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

# Apply changes without a full restart
curl -X POST http://localhost:9090/-/reload

# Verify targets are being scraped — check the web UI or via API
curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | grep '"health"'
# "health": "up"

Install Grafana

# Import Grafana GPG key and repository
sudo apt install -y apt-transport-https software-properties-common
wget -q -O /tmp/grafana.gpg https://apt.grafana.com/gpg.key
sudo gpg --dearmor -o /usr/share/keyrings/grafana.gpg /tmp/grafana.gpg
echo "deb [signed-by=/usr/share/keyrings/grafana.gpg] https://apt.grafana.com stable main" 
  | sudo tee /etc/apt/sources.list.d/grafana.list

sudo apt update && sudo apt install grafana -y
sudo systemctl enable --now grafana-server

# Confirm it's running on port 3000
sudo ss -tlnp | grep 3000
# LISTEN 0 4096 *:3000 * users:(("grafana",pid=...))

Open http://yourserver:3000 in a browser. Default credentials are admin / admin — Grafana forces a password change on first login. To add Prometheus as a data source: navigate to Connections → Data Sources → Add data source, select Prometheus, set the URL to http://localhost:9090, and click Save & Test. You should see "Successfully queried the Prometheus API."

Import the Node Exporter Full Dashboard

Rather than building dashboards from scratch, import the community dashboard that covers every node_exporter metric. Go to Dashboards → Import, enter ID 1860, select your Prometheus data source, and click Import. You'll immediately have graphs for CPU usage, memory, disk I/O, network throughput, load average, and filesystem usage — all pre-configured.

Other useful dashboard IDs: 11074 (cleaner system overview), 9614 (Nginx), 9628 (PostgreSQL), 193 (Docker via cAdvisor).

Essential PromQL Queries

PromQL is what separates Prometheus from simpler monitoring tools. These are the queries you'll use most often — paste them directly into Grafana's query editor or the Prometheus expression browser at http://localhost:9090/graph.

# CPU usage % across all cores (per instance)
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage %
100 * (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes))

# Available memory in GiB
node_memory_MemAvailable_bytes / 1024^3

# Root filesystem usage %
100 - ((node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"} * 100)
       / node_filesystem_size_bytes{mountpoint="/",fstype!="tmpfs"})

# Disk read throughput (bytes/sec, averaged over 5m)
rate(node_disk_read_bytes_total{device="sda"}[5m])

# Network receive rate in MB/s
rate(node_network_receive_bytes_total{device="eth0"}[5m]) / 1024 / 1024

# 1-minute load average relative to CPU count
node_load1 / count without(cpu, mode) (node_cpu_seconds_total{mode="idle"})

# Number of running processes
node_procs_running

# TCP connections in ESTABLISHED state
node_netstat_Tcp_CurrEstab

Alerting with Alert Rules and Alertmanager

Create the rules directory and write your first alert rules. These fire when conditions stay true for a specified duration — the for field prevents false positives from transient spikes.

sudo mkdir -p /etc/prometheus/rules

sudo tee /etc/prometheus/rules/server_alerts.yml << 'EOF'

groups:

  - name: server_alerts

    interval: 1m

    rules:

      - alert: HighCPUUsage

        expr: >

          100 - (avg by(instance)

          (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80

        for: 5m

        labels:

          severity: warning

        annotations:

          summary: "High CPU on {{ $labels.instance }}"

          description: "CPU at {{ printf "%.1f" $value }}% for over 5 minutes."
      - alert: CriticalDiskUsage

        expr: >

          100 - ((node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"} * 100)

          / node_filesystem_size_bytes{mountpoint="/",fstype!="tmpfs"}) > 90

        for: 2m

        labels:

          severity: critical

        annotations:

          summary: "Disk almost full on {{ $labels.instance }}"

          description: "Root filesystem is {{ printf "%.1f" $value }}% full."
      - alert: MemoryPressure

        expr: >

          100 * (1 - (node_memory_


Further Reading

Linux Logs: journalctl and /var/log
Linux Process Management Guide
Docker Compose: Complete Guide
Best Linux VPS for Server Monitoring