Running AI Agents Without a GUI on Linux Servers
Running AI Agents Without a GUI on Linux Servers
AI agents are autonomous programs that use large language models to reason, plan, and execute multi-step tasks. Unlike chatbots that wait for human input, agents call tools, make decisions, and loop until a goal is complete. Running them headless on a Linux server means no browser, no desktop, no interactive terminal — just a process that runs, does work, and logs what happened.
This pattern unlocks genuinely useful automation: agents that monitor your infrastructure and file tickets, scrape and summarize content on a schedule, process incoming data pipelines, or autonomously manage cloud resources. This tutorial covers everything from environment setup through production-grade systemd services, monitoring, and cost controls.
1. What Headless AI Agents Are and Why They Belong on Servers
A headless agent has no interactive component. It receives a goal (hardcoded, passed via argument, or read from a queue), executes a reasoning loop, and terminates or waits for the next trigger. The loop typically looks like this:
- Send the current context and available tools to the LLM
- Parse the model's response for tool calls
- Execute those tool calls (bash commands, HTTP requests, database queries)
- Append results to context and repeat until the model returns a final answer
Running this on a server rather than a laptop gives you persistent execution, cron/systemd scheduling, proper log management, resource limits, and the ability to integrate with existing infrastructure like databases, message queues, and secrets managers.
2. Prerequisites: Python Environments and API Key Management
Python Virtual Environments
Never install agent frameworks into the system Python. Use a dedicated virtualenv per project:
sudo apt update && sudo apt install -y python3.11 python3.11-venv python3-pip
mkdir -p /opt/agents/myagent
cd /opt/agents/myagent
python3.11 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
For a service account that runs agents, create a dedicated system user:
sudo useradd --system --shell /bin/bash --home /opt/agents --create-home agentuser
sudo chown -R agentuser:agentuser /opt/agents
API Key Management
Never hardcode keys in scripts. On a server, use one of these approaches:
Option A: Environment file loaded by systemd (recommended)
sudo mkdir -p /etc/agents
sudo tee /etc/agents/myagent.env > /dev/null <<'EOF'
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
TELEGRAM_BOT_TOKEN=...
TELEGRAM_CHAT_ID=...
EOF
sudo chmod 600 /etc/agents/myagent.env
sudo chown agentuser:agentuser /etc/agents/myagent.env
Option B: HashiCorp Vault for secrets at scale
# Retrieve secrets at runtime using the Vault CLI
export ANTHROPIC_API_KEY=$(vault kv get -field=api_key secret/agents/anthropic)
Option C: AWS Secrets Manager (if running on EC2)
pip install boto3
python3 -c "
import boto3, json
client = boto3.client('secretsmanager', region_name='us-east-1')
secret = json.loads(client.get_secret_value(SecretId='prod/agents/keys')['SecretString'])
print(secret['ANTHROPIC_API_KEY'])
"
3. CrewAI for Multi-Agent Task Orchestration
CrewAI lets you define multiple specialized agents that collaborate on a task. It is well-suited for pipelines where you want a researcher agent, a writer agent, and a critic agent working in sequence.
Installation
source /opt/agents/myagent/.venv/bin/activate
pip install crewai crewai-tools
Defining a Headless Crew
Create /opt/agents/myagent/infra_report_crew.py:
#!/opt/agents/myagent/.venv/bin/python
"""
Headless CrewAI agent that checks server health and writes a report.
Run as: python infra_report_crew.py
"""
import os
import sys
from datetime import datetime
from crewai import Agent, Task, Crew, Process
from crewai_tools import tool
# --- Custom Tools ---
@tool("run_shell_command")
def run_shell_command(command: str) -> str:
"""Execute a safe read-only shell command and return stdout."""
import subprocess
ALLOWED = ["df", "free", "uptime", "top", "ps", "netstat", "ss", "cat /proc"]
if not any(command.startswith(a) for a in ALLOWED):
return f"Command not in allowlist: {command}"
result = subprocess.run(command, shell=True, capture_output=True, text=True, timeout=10)
return result.stdout[:2000] # Truncate to avoid context bloat
# --- Agents ---
sysadmin = Agent(
role="Linux Systems Administrator",
goal="Gather accurate system metrics from the server",
backstory="Experienced sysadmin who knows every useful /proc file and CLI tool.",
tools=[run_shell_command],
verbose=False, # Set False for clean headless logs
llm="anthropic/claude-3-5-haiku-20241022",
)
analyst = Agent(
role="Infrastructure Analyst",
goal="Interpret system metrics and identify problems",
backstory="You turn raw metrics into actionable insights.",
verbose=False,
llm="anthropic/claude-3-5-haiku-20241022",
)
# --- Tasks ---
gather_task = Task(
description="Run commands to check disk usage, memory, CPU load, and top processes. Collect raw output.",
expected_output="Raw metrics output from df, free -h, uptime, and ps aux --sort=-%cpu | head -10",
agent=sysadmin,
)
report_task = Task(
description="Analyze the collected metrics. Flag anything concerning. Produce a 5-bullet executive summary.",
expected_output="A concise health report with severity indicators (OK / WARN / CRIT).",
agent=analyst,
context=[gather_task],
)
# --- Crew ---
crew = Crew(
agents=[sysadmin, analyst],
tasks=[gather_task, report_task],
process=Process.sequential,
verbose=False,
)
if __name__ == "__main__":
print(f"[{datetime.utcnow().isoformat()}] Starting infrastructure report crew...")
result = crew.kickoff()
report_path = f"/var/log/agents/infra-report-{datetime.utcnow().strftime('%Y%m%d-%H%M')}.txt"
os.makedirs("/var/log/agents", exist_ok=True)
with open(report_path, "w") as f:
f.write(str(result))
print(f"[{datetime.utcnow().isoformat()}] Report written to {report_path}")
print(result)
Run it manually first to verify it works before scheduling:
cd /opt/agents/myagent
source .venv/bin/activate
ANTHROPIC_API_KEY=sk-ant-... python infra_report_crew.py
4. Agno Framework: Lightweight Agents with Tools
Agno (formerly Phidata) takes a simpler approach — single-file agent definitions with built-in tool integrations for web search, databases, shell access, and APIs. It is ideal when you want a capable agent without the overhead of defining a full crew.
Installation
pip install agno anthropic duckduckgo-search
Defining a Headless Agno Agent
#!/opt/agents/myagent/.venv/bin/python
"""
Agno agent that monitors a URL and summarizes changes.
"""
import os
import sys
import logging
from agno.agent import Agent
from agno.models.anthropic import Claude
from agno.tools.duckduckgo import DuckDuckGoTools
from agno.tools.shell import ShellTools
logging.basicConfig(
filename="/var/log/agents/agno-monitor.log",
level=logging.INFO,
format="%(asctime)s %(levelname)s %(message)s",
)
def build_agent() -> Agent:
return Agent(
model=Claude(id="claude-3-5-haiku-20241022"),
tools=[
DuckDuckGoTools(),
ShellTools(run_shell_commands=True),
],
instructions=[
"You are a monitoring agent running on a Linux server.",
"When asked to check something, use your tools, then report findings concisely.",
"Always terminate after completing the task. Do not ask clarifying questions.",
],
markdown=False,
show_tool_calls=False,
)
def run_headless(prompt: str) -> str:
agent = build_agent()
response = agent.run(prompt, stream=False)
return response.content
if __name__ == "__main__":
task = sys.argv[1] if len(sys.argv) > 1 else (
"Search for the latest CVEs for OpenSSH and summarize any critical ones from the past 30 days."
)
logging.info(f"Task: {task}")
result = run_headless(task)
logging.info(f"Result: {result[:500]}")
print(result)
# Test with a custom task
python agno_monitor.py "Check disk space and summarize which directories are largest under /var"
5. Custom Lightweight Agent with the Anthropic SDK
Sometimes you do not need a framework. Here is a complete agentic loop in under 50 lines using the Anthropic SDK directly. This gives you full control and zero framework overhead.
#!/opt/agents/myagent/.venv/bin/python
"""
Minimal agentic loop with tool use. Under 50 lines of logic.
"""
import os, json, subprocess
import anthropic
client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
MODEL = "claude-3-5-haiku-20241022"
TOOLS = [
{
"name": "bash",
"description": "Run a read-only bash command. Only df, free, uptime, cat, ls, ps, ss are allowed.",
"input_schema": {
"type": "object",
"properties": {"command": {"type": "string", "description": "The bash command to run"}},
"required": ["command"],
},
}
]
ALLOWLIST = ("df ", "free", "uptime", "cat /proc", "ls ", "ps ", "ss ", "hostname", "uname")
def run_bash(command: str) -> str:
if not command.strip().startswith(ALLOWLIST):
return "Error: command not in allowlist"
r = subprocess.run(command, shell=True, capture_output=True, text=True, timeout=15)
return (r.stdout + r.stderr)[:3000]
def agent_loop(goal: str) -> str:
messages = [{"role": "user", "content": goal}]
for _ in range(10): # Hard cap on iterations
response = client.messages.create(
model=MODEL, max_tokens=1024, tools=TOOLS, messages=messages
)
messages.append({"role": "assistant", "content": response.content})
if response.stop_reason == "end_turn":
return next((b.text for b in response.content if hasattr(b, "text")), "")
tool_results = []
for block in response.content:
if block.type == "tool_use":
output = run_bash(block.input["command"])
tool_results.append({"type": "tool_result", "tool_use_id": block.id, "content": output})
messages.append({"role": "user", "content": tool_results})
return "Max iterations reached"
if __name__ == "__main__":
import sys
goal = sys.argv[1] if len(sys.argv) > 1 else "Summarize current server health in 3 bullet points."
print(agent_loop(goal))
pip install anthropic
python minimal_agent.py "Check if any filesystem is over 80% full and report it"
6. Running Agents as systemd Services
For agents that should run on a schedule or restart on failure, systemd is far more robust than cron for anything beyond simple one-shot tasks.
One-Shot Service (runs and exits)
sudo tee /etc/systemd/system/infra-report.service > /dev/null <<'EOF'
[Unit]
Description=Infrastructure Report AI Agent
After=network-online.target
Wants=network-online.target
[Service]
Type=oneshot
User=agentuser
Group=agentuser
WorkingDirectory=/opt/agents/myagent
EnvironmentFile=/etc/agents/myagent.env
ExecStart=/opt/agents/myagent/.venv/bin/python /opt/agents/myagent/infra_report_crew.py
StandardOutput=journal
StandardError=journal
SyslogIdentifier=infra-report-agent
TimeoutStartSec=300
# Security hardening
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/var/log/agents /tmp
PrivateTmp=yes
EOF
Timer (replaces cron)
sudo tee /etc/systemd/system/infra-report.timer > /dev/null <<'EOF'
[Unit]
Description=Run infrastructure report agent every 6 hours
Requires=infra-report.service
[Timer]
OnBootSec=5min
OnUnitActiveSec=6h
AccuracySec=1min
Persistent=true
[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now infra-report.timer
sudo systemctl list-timers infra-report.timer
Long-Running Agent Service (persistent daemon)
sudo tee /etc/systemd/system/agno-monitor.service > /dev/null <<'EOF'
[Unit]
Description=Agno Monitoring Agent Daemon
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=agentuser
Group=agentuser
WorkingDirectory=/opt/agents/myagent
EnvironmentFile=/etc/agents/myagent.env
ExecStart=/opt/agents/myagent/.venv/bin/python /opt/agents/myagent/agno_monitor_daemon.py
Restart=on-failure
RestartSec=30s
StandardOutput=journal
StandardError=journal
SyslogIdentifier=agno-monitor
# Resource limits
MemoryMax=512M
CPUQuota=50%
# Security
NoNewPrivileges=yes
PrivateTmp=yes
ProtectKernelTunables=yes
ProtectControlGroups=yes
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now agno-monitor.service
7. Monitoring and Logging Agent Runs
Querying Logs with journalctl
# Follow live output
journalctl -u infra-report-agent -f
# Last run output
journalctl -u infra-report-agent --since "1 hour ago"
# Show only errors
journalctl -u agno-monitor -p err
# Export to file for analysis
journalctl -u infra-report-agent --since "7 days ago" --output=json > /tmp/agent-logs.json
Structured Python Logging
import logging
import json
from datetime import datetime
class JSONFormatter(logging.Formatter):
def format(self, record):
return json.dumps({
"ts": datetime.utcnow().isoformat(),
"level": record.levelname,
"agent": record.name,
"msg": record.getMessage(),
"tokens_used": getattr(record, "tokens_used", None),
"cost_usd": getattr(record, "cost_usd", None),
})
logger = logging.getLogger("myagent")
handler = logging.FileHandler("/var/log/agents/myagent.jsonl")
handler.setFormatter(JSONFormatter())
logger.addHandler(handler)
logger.setLevel(logging.INFO)
Telegram Notifications for Agent Completion
import os, requests
def notify_telegram(message: str, level: str = "INFO"):
token = os.environ["TELEGRAM_BOT_TOKEN"]
chat_id = os.environ["TELEGRAM_CHAT_ID"]
icons = {"INFO": "ℹ️", "WARN": "⚠️", "CRIT": "🚨", "OK": "✅"}
icon = icons.get(level, "🤖")
payload = {"chat_id": chat_id, "text": f"{icon} *Agent Alert*nn{message}", "parse_mode": "Markdown"}
try:
requests.post(f"https://api.telegram.org/bot{token}/sendMessage", json=payload, timeout=10)
except requests.RequestException as e:
logger.warning(f"Telegram notification failed: {e}")
# Usage after agent run:
notify_telegram(f"Infra report completed.n{result[:500]}", level="OK")
8. Security Considerations: Sandboxing Agents
Agents that can execute tools are a privilege escalation risk if compromised via prompt injection or a malicious tool result. Apply these mitigations in layers.
Restrict Tool Execution with an Allowlist
ALLOWED_COMMANDS = {
"df": ["df", "-h"],
"free": ["free", "-h"],
"uptime": ["uptime"],
"ps_top": ["ps", "aux", "--sort=-%cpu"],
}
def safe_execute(tool_name: str) -> str:
if tool_name not in ALLOWED_COMMANDS:
return f"Tool '{tool_name}' is not permitted."
result = subprocess.run(
ALLOWED_COMMANDS[tool_name],
capture_output=True, text=True, timeout=10,
env={"PATH": "/usr/bin:/bin"} # Minimal PATH
)
return result.stdout[:2000]
Run Inside a Bubblewrap Sandbox
sudo apt install -y bubblewrap
# Wrap the agent process itself
bwrap
--ro-bind /usr /usr
--ro-bind /etc /etc
--ro-bind /lib /lib
--ro-bind /lib64 /lib64
--bind /var/log/agents /var/log/agents
--tmpfs /tmp
--unshare-net
--unshare-pid
--die-with-parent
/opt/agents/myagent/.venv/bin/python /opt/agents/myagent/minimal_agent.py
Network Namespace Isolation (for agents that don't need internet)
# In the systemd unit file, add:
# PrivateNetwork=yes # Completely isolated network
# RestrictAddressFamilies=AF_INET AF_INET6 # Or restrict address families
# IPAddressAllow=192.168.1.0/24 10.0.0.0/8 # Allow only internal ranges
AppArmor Profile
sudo tee /etc/apparmor.d/opt.agents.myagent <<'EOF'
#include
/opt/agents/myagent/.venv/bin/python {
#include
#include
/opt/agents/myagent/** r,
/var/log/agents/** rw,
/tmp/** rw,
/proc/meminfo r,
/proc/cpuinfo r,
network inet stream,
deny /etc/shadow r,
deny /root/** rw,
}
EOF
sudo apparmor_parser -r /etc/apparmor.d/opt.agents.myagent
9. Cost Monitoring for Long-Running Agents
Agents in agentic loops can burn through token budgets quickly. Build cost tracking in from the start.
Token and Cost Tracking with Anthropic SDK
import json
from pathlib import Path
from datetime import date
COST_PER_1K = {
"claude-3-5-haiku-20241022": {"input": 0.00025, "output": 0.00125},
"claude-3-5-sonnet-20241022": {"input": 0.003, "output": 0.015},
}
COST_LOG = Path("/var/log/agents/costs.jsonl")
def log_cost(model: str, usage, agent_name: str):
input_tokens = usage.input_tokens
output_tokens = usage.output_tokens
rates = COST_PER_1K.get(model, {"input": 0, "output": 0})
cost = (input_tokens / 1000 * rates["input"]) + (output_tokens / 1000 * rates["output"])
record = {
"date": date.today().isoformat(),
"agent": agent_name,
"model": model,
"input_tokens": input_tokens,
"output_tokens": output_tokens,
"cost_usd": round(cost, 6),
}
with COST_LOG.open("a") as f:
f.write(json.dumps(record) + "n")
return cost
def daily_cost_summary():
today = date.today().isoformat()
total = 0.0
if not COST_LOG.exists():
return 0.0
for line in COST_LOG.read_text().splitlines():
record = json.loads(line)
if record["date"] == today:
total += record["cost_usd"]
return total
def enforce_daily_budget(budget_usd: float = 1.0):
spent = daily_cost_summary()
if spent >= budget_usd:
raise RuntimeError(f"Daily budget of ${budget_usd} exceeded. Spent: ${spent:.4f}. Agent halted.")
return spent
Integrate Budget Enforcement into the Agent Loop
DAILY_BUDGET_USD = float(os.environ.get("AGENT_DAILY_BUDGET_USD", "2.0"))
for iteration in range(10):
# Check budget before each LLM call
enforce_daily_budget(DAILY_BUDGET_USD)
response = client.messages.create(model=MODEL, max_tokens=1024, tools=TOOLS, messages=messages)
cost = log_cost(MODEL, response.usage, agent_name="infra-monitor")
logger.info(f"Iteration {iteration}: cost=${cost:.5f}, total_today=${daily_cost_summary():.4f}")
Weekly Cost Report via Cron
sudo tee /opt/agents/cost_report.sh <<'EOF'
#!/bin/bash
source /opt/agents/myagent/.venv/bin/activate
python3 - <<'PYTHON'
import json
from pathlib import Path
from collections import defaultdict
log = Path("/var/log/agents/costs.jsonl")
totals = defaultdict(float)
for line in log.read_text().splitlines():
r = json.loads(line)
totals[r["agent"]] += r["cost_usd"]
print("=== Weekly Agent Cost Report ===")
for agent, cost in sorted(totals.items(), key=lambda x: -x[1]):
print(f" {agent}: ${cost:.4f}")
print(f" TOTAL: ${sum(totals.values()):.4f}")
PYTHON
EOF
chmod +x /opt/agents/cost_report.sh
# Add to crontab for agentuser
(crontab -l 2>/dev/null; echo "0 9 * * 1 /opt/agents/cost_report.sh | mail -s 'Agent Costs' ops@yourcompany.com") | crontab -
With this stack — proper virtualenvs, secret management, framework selection, systemd service definitions, structured logging, sandboxing, and cost guardrails — you have a production-grade foundation for running AI agents as first-class server workloads. Start with the minimal Anthropic SDK loop to understand the mechanics, add Agno or CrewAI when you need built-in tools or multi-agent coordination, and always wrap everything in the systemd and security layers from day one rather than retrofitting them later.