grep, sed, and awk: the Linux text processing trio

Most Linux administration happens in text files: configs, logs, scripts. Three tools handle the majority of text processing tasks: grep for finding, sed for replacing, and awk for extracting and transforming.

grep: finding text

grep searches for patterns in files or input.

grep "error" /var/log/syslog
grep -i "error" /var/log/syslog          # case insensitive
grep -r "password" /etc/                 # recursive search
grep -n "error" /var/log/syslog          # show line numbers
grep -c "error" /var/log/syslog          # count matches
grep -v "debug" /var/log/syslog          # invert — show non-matching lines

Regular expressions in grep:

grep "^2024" /var/log/syslog             # lines starting with 2024
grep "failed$" /var/log/auth.log         # lines ending with "failed"
grep "err[oe]r" /var/log/syslog          # matches "error" or "errer"
grep "disk[0-9]" /var/log/syslog         # disk0, disk1, etc.

For extended regex (alternation, +, ?, groups), use grep -E or egrep:

grep -E "error|warning|critical" /var/log/syslog
grep -E "disk[0-9]+" /var/log/syslog     # one or more digits

grep is fast. For searching large files or directory trees, it’s almost always the right first step.

sed: stream editor

sed processes text line by line, applying transformations.

Substitute (the most common use):

sed 's/old/new/' file.txt                # replace first occurrence per line
sed 's/old/new/g' file.txt               # replace all occurrences per line
sed -i 's/old/new/g' file.txt            # edit file in place

Delete lines:

sed '/pattern/d' file.txt                # delete lines matching pattern
sed '3d' file.txt                        # delete line 3
sed '1,5d' file.txt                      # delete lines 1 through 5

Print specific lines:

sed -n '10p' file.txt                    # print line 10
sed -n '10,20p' file.txt                 # print lines 10-20

Multiple operations:

sed -e 's/foo/bar/g' -e 's/baz/qux/g' file.txt

Capture groups:

echo "hello world" | sed 's/\(.*\) \(.*\)/ 
/'
# output: world hello

With extended regex (-E):

echo "hello world" | sed -E 's/(.*) (.*)/ 
/'

awk: pattern scanning and processing

awk is a full programming language for text processing. It’s overkill for simple substitutions but invaluable for structured data.

Basic structure:

awk 'pattern { action }' file.txt

Print specific columns (default delimiter is whitespace):

awk '{print $1}' file.txt                # first column
awk '{print $1, $3}' file.txt            # first and third columns
awk -F: '{print $1}' /etc/passwd         # custom delimiter (:)

Filter rows:

awk '$3 > 100 {print $1, $3}' file.txt   # rows where column 3 > 100
awk '/error/ {print}' /var/log/syslog    # lines matching pattern

Built-in variables:

awk '{print NR, $0}' file.txt            # NR = line number
awk 'END {print NR}' file.txt            # total line count
awk -F: '{print NF, $0}' /etc/passwd     # NF = number of fields

Calculations:

awk '{sum += $3} END {print sum}' file.txt
awk '{sum += $3} END {print sum/NR}' file.txt  # average

Formatting output:

awk -F: '{printf "%-20s %s
", $1, $7}' /etc/passwd

Combining them

The real power comes from piping them together:

# Find all failed SSH attempts, extract IPs, count and sort
grep "Failed password" /var/log/auth.log |   awk '{print $(NF-3)}' |   sort | uniq -c | sort -rn | head -10

# Find large files mentioned in logs
grep -i "disk" /var/log/syslog |   sed 's/.*file //' |   awk '{print $1}'

# Extract usernames from /etc/passwd with their shells
awk -F: '$7 != "/usr/sbin/nologin" {printf "%-15s %s
", $1, $7}' /etc/passwd

When to use which

grep — “Show me lines matching this pattern.” Finding things.

sed — “Replace this with that.” Editing in place or transforming.

awk — “Extract columns, filter, calculate.” Structured data processing.

For simple searches, grep. For substitutions, sed. For anything involving columns, arithmetic, or formatted output, awk. For complex logic, maybe Python — but these three cover 90% of what you’ll need on a server.

Common mistakes

Forgetting -i with sed. Without it, sed only replaces the first match per line. Almost always use /g.

Not quoting patterns. Shell globbing can expand your pattern before grep sees it. Always quote: grep "pattern", not grep pattern.

Using awk for simple greps. If you just need lines matching a string, grep is faster and simpler than awk.

Remarks

These three tools have been part of Unix since the beginning and they’re still the fastest way to process text on a server. Learn grep first (finding things), then sed (replacing things), then awk (structured processing). The investment pays off every time you SSH into a box.