How to Set Up SSH on Linux: Complete Guide with Key Authentication

How to Set Up SSH on Linux: Complete Guide with Key Authentication

✅ Tested on Ubuntu 24.04, Debian 12, Arch Linux — Last updated: June 2026

SSH (Secure Shell) is the standard protocol for securely connecting to remote Linux servers, and setting it up is one of the first things any sysadmin or developer needs to do. This guide covers everything: installing and configuring the SSH server, connecting from a client, setting up SSH key authentication, hardening the server against attacks, and using SSH tunnels and port forwarding.

Table of Contents

How SSH Works

SSH creates an encrypted tunnel between two computers. When you connect to a remote server, SSH:

  1. Establishes a TCP connection to port 22 (default) on the server
  2. Negotiates an encryption algorithm (usually ChaCha20 or AES-256-GCM in 2026)
  3. Authenticates you (via password or cryptographic keys)
  4. Opens an encrypted channel for all communication

Two components are involved:

  • SSH Server (sshd): Runs on the machine you want to connect TO. On Linux, this is OpenSSH Server (openssh-server)
  • SSH Client: Runs on your machine. On Linux and Mac, this is the ssh command. On Windows, it's PowerShell's built-in ssh or PuTTY

Install the SSH Server

Ubuntu / Debian / Linux Mint

sudo apt update
sudo apt install openssh-server -y

Fedora / RHEL / CentOS

sudo dnf install openssh-server -y

Arch Linux

sudo pacman -S openssh

Start and Enable the SSH Service

sudo systemctl enable --now ssh         # Ubuntu/Debian (service is named 'ssh')
sudo systemctl enable --now sshd        # Fedora/Arch (service is named 'sshd')

Check SSH is Running

sudo systemctl status ssh

# Expected output:
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled)
     Active: active (running) since ...
   Main PID: 1234 (sshd)

Allow SSH Through the Firewall

# UFW (Ubuntu/Debian/Mint):
sudo ufw allow ssh
sudo ufw enable
sudo ufw status

# firewalld (Fedora/RHEL):
sudo firewall-cmd --permanent --add-service=ssh
sudo firewall-cmd --reload

Connect via SSH

From any Linux, Mac, or Windows terminal (PowerShell 7+):

# Basic connection
ssh username@hostname_or_ip

# Examples:
ssh jm@192.168.1.100           # connect to LAN machine
ssh root@203.0.113.10          # connect to VPS (not recommended — use non-root user)
ssh jm@myserver.example.com    # connect with hostname

# Connect on a non-standard port:
ssh -p 2222 jm@myserver.example.com

# Verbose mode (for debugging):
ssh -v jm@myserver.example.com

The first time you connect to a new server, SSH will show you the server's fingerprint and ask you to verify it:

The authenticity of host '203.0.113.10 (203.0.113.10)' can't be established.
ED25519 key fingerprint is SHA256:abc123def456...
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

Type yes and the fingerprint is saved to ~/.ssh/known_hosts. Future connections to the same server will be verified against this fingerprint automatically.

SSH Key Authentication (Recommended)

Password authentication is convenient but vulnerable to brute-force attacks. SSH key authentication uses cryptographic key pairs: a private key (stays on your machine) and a public key (placed on the server). It's both more secure and more convenient — no typing passwords.

Step 1: Generate an SSH Key Pair

On your LOCAL machine (not the server):

# Modern ed25519 key (recommended):
ssh-keygen -t ed25519 -C "your_email@example.com"

# Or RSA 4096 if you need compatibility with older systems:
ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

# The command will ask:
# Enter file in which to save the key: [press Enter for default ~/.ssh/id_ed25519]
# Enter passphrase: [recommended — protects the key if your laptop is stolen]

This creates two files:

  • ~/.ssh/id_ed25519 — Private key. Never share this. Never upload it anywhere.
  • ~/.ssh/id_ed25519.pub — Public key. This goes on the server.

Step 2: Copy Your Public Key to the Server

# Easiest method (if you have password access):
ssh-copy-id username@server_ip

# If on a non-standard port:
ssh-copy-id -p 2222 username@server_ip

# Manual method (when ssh-copy-id isn't available):
cat ~/.ssh/id_ed25519.pub | ssh username@server_ip "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys && chmod 700 ~/.ssh"

Step 3: Test Key Authentication

ssh username@server_ip

# If a passphrase was set, you'll be asked for it.
# If no passphrase, you connect immediately without a password prompt.

# Verbose mode shows which key was used:
ssh -v username@server_ip 2>&1 | grep "Offering|Accepted"

Step 4: Disable Password Authentication (Optional but Recommended)

Once key auth works, disable password login to prevent brute-force attacks. Edit /etc/ssh/sshd_config on the SERVER:

sudo nano /etc/ssh/sshd_config

# Find and change these lines:
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes   # keep this yes

# Apply changes:
sudo systemctl restart ssh

⚠️ Before disabling passwords: Make absolutely sure your key works and you have another way in (console access) in case you lock yourself out.

SSH Client Configuration

The SSH config file (~/.ssh/config) lets you create shortcuts for servers you connect to frequently. Create it if it doesn't exist:

nano ~/.ssh/config
# ~/.ssh/config example

# Personal VPS
Host myvps
    HostName 203.0.113.10
    User jm
    Port 22
    IdentityFile ~/.ssh/id_ed25519

# Hostinger server on non-standard port
Host hostinger
    HostName 147.93.93.225
    User u617605593
    Port 65002
    IdentityFile ~/.ssh/id_ed25519

# GitHub
Host github.com
    User git
    IdentityFile ~/.ssh/id_ed25519

# Keep connections alive (prevents timeouts)
Host *
    ServerAliveInterval 60
    ServerAliveCountMax 3
chmod 600 ~/.ssh/config   # required — SSH refuses to use config files with loose permissions

Now instead of ssh -p 65002 u617605593@147.93.93.225, you type just ssh hostinger.

Harden Your SSH Server

Any server exposed to the internet is constantly being probed by bots attempting to log in. These settings significantly reduce your attack surface.

Recommended /etc/ssh/sshd_config Settings

sudo nano /etc/ssh/sshd_config
# Change default port (reduces automated scanning):
Port 2222

# Disable root login:
PermitRootLogin no

# Key auth only:
PasswordAuthentication no
PubkeyAuthentication yes

# Limit login attempts:
MaxAuthTries 3
LoginGraceTime 30

# Disable unused features:
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no    # Enable this only if you use SSH tunnels

# Allow only specific users:
AllowUsers jm deploy

# Show last login info:
PrintLastLog yes
# Test config before restarting (to catch typos):
sudo sshd -t

# Restart only if no errors:
sudo systemctl restart ssh

Fail2ban — Block Brute Force Attacks

Fail2ban monitors your SSH logs and automatically bans IPs that have too many failed login attempts.

sudo apt install fail2ban -y

# Create local config (don't edit the default):
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sudo nano /etc/fail2ban/jail.local

# Find [sshd] section and set:
[sshd]
enabled = true
port = 2222        # change to your SSH port
maxretry = 5
bantime = 3600     # ban for 1 hour (3600 seconds)
findtime = 600     # within 10 minutes

sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd   # check banned IPs

SSH Tunnels and Port Forwarding

SSH isn't just for terminal access. It can forward ports, creating encrypted tunnels for other services.

Local Port Forwarding

Forward a remote port to your local machine. Useful to access services on a remote server that aren't exposed publicly (like a database or local web app):

# Access remote MySQL (port 3306) on your local port 3307:
ssh -L 3307:localhost:3306 jm@myserver.example.com

# Then connect locally:
mysql -u root -p -h 127.0.0.1 -P 3307

# Access a Jupyter notebook running on a remote server:
ssh -L 8888:localhost:8888 jm@myserver.example.com

# Keep the tunnel open without opening a shell (-N), run in background (-f):
ssh -fN -L 8888:localhost:8888 jm@myserver.example.com

Remote Port Forwarding

Expose a local port through the remote server. Useful for demos or webhooks when you don't have a public IP:

# Make your local port 3000 accessible on remote port 8080:
ssh -R 8080:localhost:3000 jm@myserver.example.com

SOCKS Proxy

Route your browser traffic through a remote server (simple VPN-like behavior):

ssh -D 1080 jm@myserver.example.com

# Then configure your browser to use SOCKS5 proxy at 127.0.0.1:1080

Useful SSH Tips

SSH Agent — Avoid Typing Passphrase Repeatedly

# Start the agent and add your key:
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

# On most desktop Linux systems, the agent starts automatically at login.
# Check if a key is loaded:
ssh-add -l

Copy Files with SCP and rsync

# SCP — simple file copy:
scp local_file.txt jm@server:/remote/path/          # upload
scp jm@server:/remote/file.txt /local/path/         # download
scp -r local_dir/ jm@server:/remote/                # copy directory
scp -P 2222 file.txt jm@server:/tmp/                # non-standard port

# rsync — faster, only copies changes:
rsync -avz local_dir/ jm@server:/remote/dir/        # sync directory
rsync -avz --delete local_dir/ jm@server:/remote/   # mirror (delete remote extras)
rsync -avz -e "ssh -p 2222" local/ jm@server:/tmp/  # non-standard port

Run a Single Command Without Opening a Shell

ssh jm@server "df -h"
ssh jm@server "sudo systemctl restart nginx"
ssh jm@server "cat /var/log/nginx/error.log | tail -50"

# Run a script remotely:
ssh jm@server "bash -s" < local_script.sh

SSH Jump Hosts (ProxyJump)

Connect to a server that's only accessible through another server (bastion host):

ssh -J bastion.example.com jm@internal-server.private

# Or in ~/.ssh/config:
Host internal
    HostName internal-server.private
    User jm
    ProxyJump bastion.example.com

Keep Sessions Alive with tmux or screen

If your SSH connection drops, any running commands are killed. Use tmux or screen to create persistent sessions:

ssh jm@server
tmux new -s mysession         # start tmux session
# run your long command
# disconnect (Ctrl+B then D)

# Later, reconnect:
ssh jm@server
tmux attach -t mysession      # your session is still there

Troubleshooting

Connection refused

# Check SSH is running on the server:
sudo systemctl status ssh

# Check firewall:
sudo ufw status

# Verify the port is listening:
ss -tulnp | grep :22

# Check from client which port you're trying:
ssh -v username@server 2>&1 | grep "connect to"

Permission denied (publickey)

# Check permissions on server (must be exactly this):
ls -la ~/.ssh/
# drwx------ 2 user user 4096 ... .ssh
# -rw------- 1 user user ... authorized_keys

# Fix if wrong:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

# Check the public key is in authorized_keys on server:
cat ~/.ssh/authorized_keys

# On client, check which key is being offered:
ssh -vv username@server 2>&1 | grep "Offering|offered|Accepted"

Connection times out

# Test basic connectivity:
ping server_ip
telnet server_ip 22    # or: nc -zv server_ip 22

# Possible causes:
# - Wrong IP or hostname
# - Firewall on server blocking port 22
# - Cloud provider security group blocking port 22
# - SSH running on different port

Host key verification failed

This happens when the server's key has changed (server reinstalled, or different server on the same IP). If you're sure it's legitimate:

ssh-keygen -R server_ip_or_hostname   # remove old key from known_hosts
ssh username@server                    # reconnect and accept new key

⚠️ If you didn't reinstall the server, a changed host key could indicate a man-in-the-middle attack. Investigate before accepting.

Frequently Asked Questions

What's the difference between SSH and a VPN?

SSH creates an encrypted connection to a specific server for terminal access or tunneling specific ports. A VPN routes all your traffic through a remote server. SSH is lighter, requires no VPN client software, and is more appropriate for server administration. VPNs are better for routing all internet traffic or accessing an entire internal network.

Which key type should I use in 2026 — RSA or Ed25519?

Use Ed25519. It's faster, more secure, and produces shorter keys. RSA 4096 is still fine but only needed for compatibility with very old SSH implementations (which are rare in 2026). For everything modern, Ed25519 is the right choice.

Should I change the SSH port from 22?

It reduces automated scanning noise — attackers constantly probe port 22, and most bots don't try non-standard ports. It's not a security measure by itself (security through obscurity), but combined with key-only auth and fail2ban, it keeps your logs clean. Choose any unused port above 1024.

How do I generate a different key for each server?

ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_myserver -C "myserver_access"

# Then reference it in ~/.ssh/config:
Host myserver
    IdentityFile ~/.ssh/id_ed25519_myserver

Can SSH be used for Git?

Yes. GitHub, GitLab, and Gitea all support SSH authentication. Add your public key to your GitHub account under Settings → SSH Keys, then clone with SSH instead of HTTPS: git clone git@github.com:user/repo.git.

Now you have a secure, well-configured SSH setup. If you're using your server to run AI models or other services, check out our guide on installing Ollama to run LLMs locally or on a remote VPS — combined with SSH tunneling, you can access a private AI server from anywhere.

Further Reading

Go up