Log management with journalctl and rsyslog

When something breaks, logs are where you find out why. Linux has two logging systems that coexist: journald (systemd’s binary log) and rsyslog (the traditional text-based logger). Most systems run both.

journalctl basics

View all logs:

journalctl

Follow logs in real time (like tail -f):

journalctl -f

Filter by service:

journalctl -u nginx
journalctl -u nginx -u php-fpm    # multiple services

Filter by time:

journalctl --since today
journalctl --since "2024-01-15 10:00:00"
journalctl --since "1 hour ago"
journalctl --since yesterday --until today

Filter by priority:

journalctl -p err           # errors and above
journalctl -p warning       # warnings and above

Priorities: emerg, alert, crit, err, warning, notice, info, debug.

Combining filters

journalctl -u nginx -p err --since today

This shows nginx errors from today. Filters are ANDed together.

Binary vs text logs

Journalctl stores logs in a binary format at /var/log/journal/ (if persistent) or /run/log/journal/ (temporary). The binary format supports structured metadata — you can filter by PID, UID, boot ID, and more.

journalctl _PID=1234
journalctl _UID=1000
journalctl -b -1              # previous boot
journalctl -b                 # current boot

Making journal persistent

By default, journald may not persist logs across reboots. To enable:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

Control journal size in /etc/systemd/journald.conf:

[Journal]
SystemMaxUse=500M
SystemMaxFileSize=50M
RuntimeMaxUse=200M

Then restart journald.

rsyslog: traditional logging

Rsyslog writes text log files in /var/log/. Key files:

• /var/log/syslog — main system log
• /var/log/auth.log — authentication
• /var/log/kern.log — kernel messages
• /var/log/daemon.log — daemon messages

Configuration is in /etc/rsyslog.conf and /etc/rsyslog.d/.

Remote logging

For centralized logging, rsyslog can send logs to a remote server.

On the receiving server, enable UDP or TCP reception in /etc/rsyslog.conf:

module(load="imudp")
input(type="imudp" port="514")

module(load="imtcp")
input(type="imtcp" port="514")

On the sending server, add to /etc/rsyslog.d/remote.conf:

*.* @logserver.example.com:514       # UDP
*.* @@logserver.example.com:514      # TCP

One @ is UDP (faster, fire-and-forget). Two @@ is TCP (reliable, ordered).

Log rotation

Logrotate prevents logs from filling your disk. Configuration is in /etc/logrotate.conf and /etc/logrotate.d/.

A typical logrotate config for a custom application:

/var/log/myapp/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 appuser appuser
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}

• daily — rotate every day
• rotate 14 — keep 14 old copies
• compress — gzip old logs
• postrotate — run a command after rotation (usually reloading the service to reopen log files)

Test your config:

sudo logrotate -d /etc/logrotate.d/myapp

Force a rotation:

sudo logrotate -f /etc/logrotate.d/myapp

Searching logs

For text logs, grep is your friend:

grep "error" /var/log/syslog
grep -i "failed" /var/log/auth.log

For journalctl, use the built-in filters:

journalctl -u nginx --grep="error"

For complex searches across multiple log files, consider zgrep for gzipped rotated logs:

zgrep "error" /var/log/syslog.2.gz

Common mistakes

Not checking both journald and rsyslog. Some services log to one, some to the other. If you can’t find what you’re looking for, check both.

Letting logs fill the disk. Logrotate exists for a reason. Set it up for every application that writes logs.

Ignoring rotated logs. The problem might have happened yesterday. Check /var/log/syslog.1 or the gzipped versions.

Remarks

Logging on Linux is split between journald (structured, filterable) and rsyslog (text files, greppable). journalctl is great for filtering by time and service. rsyslog is great for grep and piping. Set up log rotation from the start — discovering a full disk at 3 AM because of log growth is a bad way to learn this lesson.