// Security · Incident Response
Incident Response Using Logs
When a breach is suspected or confirmed, your logs are the primary forensic record. This page covers the log-based investigation workflow: preserve first, then reconstruct the attack timeline, identify the entry point, determine scope, and document for remediation. Speed matters — act before logs rotate.
Step 1 — Preserve Logs Immediately
Logs rotate. On a busy server, today's access log may be overwritten within hours. The first action is always to copy all relevant logs off the compromised server to read-only storage before doing anything else.
bash
# Create timestamped evidence archive TS=$(date +%Y%m%d_%H%M%S) DEST=/tmp/ir_evidence_$TS mkdir -p $DEST # Copy all logs (compress large ones) cp -r /var/log/nginx /var/log/apache2 /var/log/auth.log /var/log/syslog \ /var/log/mail.log /var/log/mysql/ $DEST/ 2>/dev/null journalctl --no-pager -o json > $DEST/journal_full.jsonl # Hash everything for integrity verification find $DEST -type f -exec sha256sum {} \; > $DEST/hashes.sha256 tar czf /tmp/evidence_$TS.tar.gz $DEST/ # Transfer to external storage immediately scp /tmp/evidence_$TS.tar.gz analyst@safe-server:/evidence/ echo "Evidence archived. Hash: $(sha256sum /tmp/evidence_$TS.tar.gz)"
Step 2 — Establish the Timeline
Build a chronological sequence of events. Start with the known indicator (defacement, malware alert, Google warning) and work backward to find the initial access, then forward to understand what happened after.
bash
LOG=/var/log/nginx/access.log SUSPECT_IP=185.220.101.45 SUSPECT_FILE="/wp-content/uploads/2025/03/image.php" # All activity from suspect IP (chronological) grep $SUSPECT_IP $LOG | sort -k4 # First time suspect IP appeared in logs grep $SUSPECT_IP $LOG | head -1 # All requests to the shell file grep $SUSPECT_FILE $LOG # What happened just before and after first shell access? FIRST_HIT=$(grep "$SUSPECT_FILE" $LOG | head -1 | awk '{print $4}' | tr -d '[') grep $SUSPECT_IP $LOG | awk -v t="$FIRST_HIT" '$4 > "["t && $4 < "["t+5000' # Cross-reference auth.log for SSH around same time grep "Apr 10 03:" /var/log/auth.log # adjust date/hour
Step 3 — Find the Entry Point
Entry points are usually one of: file upload vulnerability, SQLi leading to file write, compromised credentials (admin login, FTP, SSH), or a vulnerable plugin. Check each vector.
| Entry Vector | Log to Check | What to Look For |
|---|---|---|
| File upload exploit | access.log | POST to upload endpoints before shell first appeared. Check the response size — 200 + small response = upload confirmed. |
| Admin panel compromise | access.log | Successful POST to wp-login.php / admin login from attacker IP before any shell activity. |
| FTP compromise | vsftpd.log / proftpd.log | Login from new IP followed by file uploads (direction=i, status=c in xferlog). |
| SSH compromise | auth.log | Accepted publickey or Accepted password from attacker IP. Cross-reference timestamp with first web shell access. |
| SQLi → file write | access.log + MySQL general log | UNION SELECT … INTO OUTFILE or LOAD DATA INFILE in MySQL log. Web log shows SQLi payloads before shell appeared. |
| Vulnerable plugin | access.log | POST to specific plugin PHP file (e.g. /wp-content/plugins/xyz/upload.php) before shell exists. |
Step 4 — Determine Scope
bash
LOG=/var/log/nginx/access.log ATTACKER=185.220.101.45 # All files accessed by attacker with 200 response (what did they read/execute?) grep $ATTACKER $LOG | awk '$9=="200" {print $7}' | sort -u # Data exfiltration: large outbound responses to attacker grep $ATTACKER $LOG | awk '$10+0 > 50000 {print $10, $7}' | sort -rn # Did attacker pivot to other IPs? (shell may have been shared) grep "$SUSPECT_FILE" $LOG | awk '{print $1}' | sort -u # Files created/modified in webroot during the attack window find /var/www/html -newer /tmp/before_attack_reference_file -ls 2>/dev/null # (create the reference file before starting: touch -t 202504100200 /tmp/before_attack_reference_file) # Check for new cron jobs, users, SSH keys added by attacker getent passwd | awk -F: '$3 >= 1000 && $3 < 65534' # new OS users cat /root/.ssh/authorized_keys cat /home/*/.ssh/authorized_keys 2>/dev/null
IR Checklist — Quick Reference
| Phase | Action | Command / Location |
|---|---|---|
| Contain | Block attacker IP at firewall | ufw deny from [IP] |
| Preserve | Archive all logs with hashes | See Step 1 above |
| Preserve | Snapshot the disk or VM | Cloud console / dd |
| Identify | First attacker IP appearance | grep [IP] access.log | head -1 |
| Identify | Entry vector (upload/login/SQLi/SSH) | See Step 3 table |
| Scope | All files accessed by attacker | grep [IP] access.log | awk '$9=="200"' |
| Scope | Malicious files on disk | find /var/www -name "*.php" -newer [ref] |
| Scope | New OS users or SSH keys added | getent passwd, cat ~/.ssh/authorized_keys |
| Scope | Cron jobs added for persistence | crontab -l -u www-data |
| Eradicate | Remove all malicious files | Delete shells, restore from clean backup |
| Eradicate | Rotate all credentials | DB password, admin password, SSH keys, FTP |
| Recover | Restore clean state from backup | Verify backup predates attack timeline |
| Improve | Patch the entry point vulnerability | Plugin update, server config, WAF rule |
🚨 Do Not Reuse a Compromised Server
Reinstalling or restoring from backup on the same server without patching the entry point will result in re-compromise, often within hours. The attacker already knows your server's address and the vulnerability. Patch first, then restore.