grep, sed and awk: Linux Text Processing Complete Guide

Tested on: Ubuntu 26.04 LTS · Debian 12 · Arch Linux (GNU grep 3.11, GNU sed 4.9, gawk 5.3) — Last updated: June 2026

grep, sed, and awk are the backbone of Linux text processing. grep finds lines matching a pattern, sed transforms text in-place or in streams, and awk is a full programming language for structured data. Together they handle the vast majority of log analysis, config management, and data extraction tasks directly in the shell — no editor required.

Prerequisites

• Basic terminal familiarity — you know how to open a shell and run commands
• GNU coreutils installed (default on all major Linux distros)
• For grep -P (PCRE), ensure grep was compiled with PCRE support: grep --version | grep -i pcre
• macOS users: examples use GNU tools. Install via brew install grep gnu-sed gawk and invoke as ggrep, gsed, gawk — or adjust the PATH

grep: Search Files for Patterns

grep (Global Regular Expression Print) reads input line by line and prints every line that matches a pattern. It's the fastest way to filter text — from log files to source code to command output.

# Basic usage
grep "error" /var/log/syslog           # lines containing "error"
grep "error" file1.txt file2.txt       # search multiple files
grep "error" *.log                     # glob expansion across .log files
grep -r "TODO" ~/projects/             # recursive directory search

# Case control
grep -i "error" logfile.txt            # case-insensitive (matches ERROR, Error, error)
grep -v "DEBUG" app.log                # invert: print lines NOT matching

# Counting and file discovery
grep -c "error" logfile.txt            # count matching lines (not occurrences)
grep -l "error" *.log                  # list files that contain the pattern
grep -L "error" *.log                  # list files that do NOT contain it

# Line numbers and context
grep -n "error" logfile.txt            # prefix each match with its line number
grep -A 3 "error" logfile.txt          # 3 lines After the match
grep -B 3 "error" logfile.txt          # 3 lines Before the match
grep -C 3 "error" logfile.txt          # 3 lines of Context (before + after)

# Extract only the matching portion (not the whole line)
grep -o "[0-9]+.[0-9]+" logfile.txt

# Fixed-string mode — no regex, much faster for literal matches
grep -F "192.168.1.1" access.log

# Whole-word match: "test" but not "testing" or "protest"
grep -w "test" file.txt

# Silent mode for scripting: exit 0 if found, 1 if not
grep -q "error" logfile.txt && echo "errors present"

grep with Regular Expressions

grep supports three regex flavors. Basic (BRE) is the default. Extended (ERE) with -E adds +, ?, |, and grouping without backslashes. Perl-compatible (PCRE) with -P adds lookaheads, backreferences, and d / w shortcuts.

# BRE (default) — metacharacters need backslash for grouping/alternation
grep "^error" logfile.txt              # lines starting with "error"
grep "error$" logfile.txt              # lines ending with "error"
grep "err.r" logfile.txt               # dot matches any single character
grep "err*" logfile.txt                # zero or more 'r' after "er"

# ERE with -E
grep -E "error|warning|critical" logfile.txt     # alternation
grep -E "error+" logfile.txt                     # one or more 'r'
grep -E "(WARN|ERROR): .+" logfile.txt           # grouped prefix

# Match IPv4 addresses
grep -E "b([0-9]{1,3}.){3}[0-9]{1,3}b" access.log

# Match email addresses
grep -E "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+.[a-zA-Z]{2,}" contacts.txt

# PCRE with -P (most powerful)
grep -P "d{4}-d{2}-d{2}Td{2}:d{2}:d{2}" app.log    # ISO 8601 timestamps
grep -P "(?<=user=)w+" auth.log                            # positive lookbehind
grep -P "^(?!#)" /etc/fstab                                 # lines not starting with #

# Recursive search filtered by file type
grep -r --include="*.py" "import requests" ~/projects/
grep -r --exclude-dir=".git" "password" /etc/

sed: Stream Editor

sed reads input line by line, applies one or more editing commands, and writes the result to stdout. Without -i it never touches the original file — always test a transformation first, then add -i to apply it in place.

# Print specific lines (-n suppresses default output)
sed -n '5p' file.txt                   # print line 5
sed -n '5,10p' file.txt                # print lines 5 through 10
sed -n '/error/p' file.txt             # print matching lines (like grep)

# Delete lines
sed '5d' file.txt                      # delete line 5
sed '5,10d' file.txt                   # delete lines 5-10
sed '/^#/d' file.txt                   # delete comment lines
sed '/^$/d' file.txt                   # delete blank lines
sed '/^[[:space:]]*$/d' file.txt       # delete lines containing only whitespace

sed Substitution

The s command is the workhorse of sed: s/pattern/replacement/flags.

# Replace first occurrence per line
sed 's/foo/bar/' file.txt

# Replace ALL occurrences per line (g = global)
sed 's/foo/bar/g' file.txt

# Case-insensitive replace (GNU sed)
sed 's/foo/bar/gI' file.txt

# In-place editing — modify the actual file
sed -i 's/foo/bar/g' file.txt

# In-place with automatic backup
sed -i.bak 's/foo/bar/g' file.txt      # original saved as file.txt.bak

# Replace only on lines matching a pattern
sed '/^server/s/localhost/127.0.0.1/g' nginx.conf

# Use alternate delimiter (avoid escaping slashes in paths/URLs)
sed 's|http://example.com|https://example.com|g' links.txt
sed 's|/usr/local/bin|/opt/bin|g' script.sh

# Capture groups — backreferences as 1, 2
sed 's/(hello) (world)/2 1/' file.txt         # BRE: "hello world" → "world hello"
sed -E 's/(hello) (world)/2 1/' file.txt           # ERE: same result, cleaner syntax

# Practical cleanup operations
sed 's/[[:space:]]*$//' file.txt        # strip trailing whitespace
sed 's/^[[:space:]]*//' file.txt        # strip leading whitespace
sed 's/^/    /' file.txt                # indent every line with 4 spaces
sed 's/$/ /' file.txt                 # add line continuation marker

Advanced sed Operations

# Insert and append text
sed '/error/i--- ERROR FOUND ---' file.txt       # insert line BEFORE match
sed '/error/a--- REVIEW REQUIRED ---' file.txt   # append line AFTER match

# Replace entire matching line
sed '/^DEPRECATED/c# Line removed — see v2 docs' config.txt

# Chain multiple commands with -e or semicolons
sed -e 's/foo/bar/g' -e '/^#/d' -e '/^$/d' file.txt
sed 's/foo/bar/g; /^#/d; /^$/d' file.txt   # equivalent

# Apply substitution to a specific line range only
sed '10,50s/old/new/g' file.txt

# Range by pattern: apply commands between two markers
sed '/^BEGIN/,/^END/ s/foo/bar/g' file.txt

# Insert contents of another file after a matching line
sed '/^# INSERT_CONFIG_HERE$/r extra.conf' main.conf

# Write matching lines to a separate file (without -n, also prints to stdout)
sed -n '/ERROR/w errors.txt' app.log

# Print line count (alternative to wc -l)
sed -n '$=' file.txt

awk: Text Processing Language

awk processes input record by record (default: one line = one record) and splits each record into fields by a separator (default: whitespace). Its structure is pattern { action } — omit the pattern to run the action on every line, omit the action to print matching lines.

# Field variables
# $0 = entire line, $1 = first field, $2 = second, $NF = last field
awk '{print $1}' file.txt              # first field of every line
awk '{print $1, $3}' file.txt          # first and third, space-separated
awk '{print $1 "-" $3}' file.txt       # custom delimiter between fields
awk '{print $NF}' file.txt             # last field
awk '{print $(NF-1)}' file.txt         # second to last

# Change the field separator with -F
awk -F: '{print $1, $3}' /etc/passwd          # username and UID
awk -F',' '{print $2}' data.csv               # CSV second column
awk -F't' '{print $1, $4}' report.tsv        # TSV columns 1 and 4
awk -F'[,;]' '{print $1}' mixed.txt           # multiple delimiters (regex)

# Set FS in a BEGIN block (same effect as -F)
awk 'BEGIN {FS=":"} {print $1, $7}' /etc/passwd

# Built-in variables
# NR = current record (line) number, NF = number of fields in current record
awk '{print NR, $0}' file.txt          # prefix every line with its number
awk 'NR==5' file.txt                   # print only line 5
awk 'NR>=10 && NR<=20' file.txt        # print lines 10-20
awk 'NF > 3' file.txt                  # only lines with more than 3 fields

Patterns, BEGIN/END, and Aggregation

# Pattern matching
awk '/error/' logfile.txt                        # lines containing "error"
awk '!/error/' logfile.txt                       # lines NOT containing it
awk '$3 > 100' data.txt                          # numeric field comparison
awk '$1 == "WARN" || $1 == "ERROR"' app.log      # field value match

# BEGIN runs once before input, END runs once after
awk 'BEGIN {print "=== Report ==="} {print} END {print "=== Done ==="}' file.txt

# Count matching lines
awk '/error/ {count++} END {print count, "errors found"}' logfile.txt

# Sum a column
awk '{sum += $4} END {printf "Total bytes: %dn", sum}' access.log

# Min/max
awk 'NR==1 {min=max=$1} {if ($1max) max=$1} END {print "Min:", min, "Max:", max}' numbers.txt

# Average with formatted output
awk '{sum += $1; n++} END {printf "Average: %.2fn", sum/n}' values.txt

# Range pattern: print lines between two markers (inclusive)
awk '/^START/,/^END/' file.txt

awk as a Programming Language

# String functions
awk '{print length($0)}' file.txt                          # line length
awk '{print toupper($1), tolower($2)}' file.txt            # case conversion
awk '{print substr($1, 1, 8)}' file.txt                    # substring (1-indexed)
awk '{gsub(/[0-9]+/, "NUM"); print}' file.txt              # global substitution
awk '{n = split($3, parts, ":"); print parts[1]}' file.txt # split field into array

# printf for formatted output
awk '{printf "%-20s %8.2fn", $1, $2}' prices.txt

# Associative arrays — count occurrences per key
awk '{count[$1]++} END {for (k in count) printf "%5d %sn", count[k], k}' logfile.txt

# Two-pass equivalent in one: arrays accumulate, END reports
awk '{bytes[$1] += $10} END {for (ip in bytes) print bytes[ip], ip}' access.log | sort -rn | head -10

# Conditional logic
awk '{
  if ($5 >= 500) status = "CRITICAL"
  else if ($5 >= 400) status = "ERROR"
  else if ($5 >= 300) status = "REDIRECT"
  else status = "OK"
  print $1, $5, status
}' access.log

# Multi-file processing — FNR resets per file, NR does not
awk 'FNR==1 {print "--- File:", FILENAME}' file1.txt file2.txt

Regular Expressions Quick Reference