13. System Monitoring & Logging

13.1 System Performance Monitoring

# vmstat — virtual memory statistics (refresh every 2 seconds)
vmstat 2 5

# iostat — IO statistics (requires sysstat)
iostat -xz 2 3

# sar — system activity report (requires sysstat)
sar -u 1 5         # CPU usage
sar -r 1 5         # Memory usage
sar -b 1 5         # IO statistics

# uptime — system load
uptime

# dmesg — kernel messages
dmesg | tail -20
dmesg | grep -i error

$ vmstat 2 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs  us sy id wa st
 1  0 247808 156340  23456 999232    0    0     2    15   89  156   5  1 93  1  0
 0  0 247808 155216  23456 1000356   0    0     0    42  102  178   3  1 96  0  0
 2  0 247808 154092  23456 1001480   0    0     0    28   95  168   4  1 95  0  0

$ uptime
 10:30:45 up 15:23,  2 users,  load average: 0.52, 0.48, 0.35

# load average meaning: 1-minute/5-minute/15-minute average load
# For an 8-core CPU, a load of 0.52 means the system is very idle

13.2 Logging System

Linux logs are a goldmine for troubleshooting. Most logs are stored in the /var/log/ directory.

Log File	Content
/var/log/syslog (Debian)	General system log
/var/log/messages (RHEL)	General system log
/var/log/auth.log	Authentication log (logins, sudo)
/var/log/kern.log	Kernel log
/var/log/dmesg	Hardware detection log
/var/log/apt/	APT package management log
/var/log/nginx/	Nginx access and error logs
/var/log/mysql/	MySQL logs

# View system logs in real-time
tail -f /var/log/syslog

# Search for failed logins
grep "Failed password" /var/log/auth.log

# View SSH login records
journalctl -u ssh --since today

# View recent kernel errors
dmesg | grep -i "error\|fail" | tail -10

# View logs for a specific time period
grep "Jun 20 10:" /var/log/syslog | head -20

$ grep "Failed password" /var/log/auth.log | tail -5
Jun 20 08:15:23 server sshd[5678]: Failed password for invalid user admin from 45.33.32.156 port 48312 ssh2
Jun 20 08:15:25 server sshd[5678]: Failed password for invalid user admin from 45.33.32.156 port 48312 ssh2
Jun 20 09:22:14 server sshd[6789]: Failed password for root from 104.236.228.48 port 37216 ssh2
Jun 20 09:22:16 server sshd[6789]: Failed password for root from 104.236.228.48 port 37216 ssh2
Jun 20 10:05:30 server sshd[7890]: Failed password for invalid user test from 185.143.223.67 port 54512 ssh2

⚠️ Note: ⚠️ Security tip: If you see a large number of "Failed password" entries from unknown IPs, your server is being brute-forced. Install fail2ban immediately to block these attacks.

13.3 logrotate — Log Rotation

Log files grow continuously. logrotate handles automatic compression, rotation, and deletion of old logs.

# View logrotate configuration
cat /etc/logrotate.conf

# View configuration for a specific application
cat /etc/logrotate.d/nginx

# Manual test (without actually executing)
sudo logrotate -d /etc/logrotate.d/nginx

# Force rotation
sudo logrotate -f /etc/logrotate.d/nginx

Nginx Log Rotation Configuration Example:

/var/log/nginx/*.log {
    daily                   # Rotate daily
    missingok               # Don't error if log doesn't exist
    rotate 14               # Keep 14 backups
    compress                # Compress old logs
    delaycompress           # Delay compression by one rotation
    notifempty              # Don't rotate empty logs
    create 0640 www-data adm # Permissions for new logs
    sharedscripts           # Run script only once after all logs are rotated
    postrotate
        [ -s /run/nginx.pid ] && kill -USR1 $(cat /run/nginx.pid)
    endscript
}