App returns 500. SSH still works. df -h shows 100% on /. Reflex: rm -rf /var/log/*.log. Frees 200 MB — disk refills in two hours because nobody saw MySQL binlog or Docker logs eating 40 GB.
A full disk is symptom, not diagnosis.
Quick df/du read
df -hT, df -i, du -xh / --max-depth=1. Compare df vs du — gap means deleted open files or hidden mount.
Frequent culprits
journald, Docker overlay/logs, MySQL binlog, app logs, cores, local backups. Timeline with find large recent files.
Deleted but open
lsof +L1 | grep deleted. Fix with reload/restart, not rm alone.
Ordered emergency
Identify top dirs, vacuum journald, truncate known logs, docker prune after volume review, extend cloud volume before surgery.
Lasting prevention
Monitor space + inodes, test logrotate, cap journald and docker logs, alert runbook per mount.
15-minute runbook
Minute 0–3: df -hT + df -i + du -xh / --max-depth=1.
Minute 3–6: journalctl --disk-usage, docker system df if applicable.
Minute 6–9: lsof +L1 | head if df/du gap.
Minute 9–12: targeted action (vacuum, truncate rotated log, listed docker prune).
Minute 12–15: verify df + monitoring alert ack + root cause ticket.
Forbidden: rm -rf without top 3 dirs identified. Mandatory: post-mortem note if >80% prod impact.
Typical post-mortem
Timeline: ignored 80% alert → 95% → 100% → outage. Root cause: debug logging enabled Friday without rotation. Action: prod log level + 85% alert + runbook.
Capacity: forecast log growth MB/day × retention.
Cloud autoscale disk if available — grow filesystem script post-resize.
Daily auto cloud snapshots without lifecycle: snapshot pile = indirectly full parent disk — review snapshot policy.
Postfix/exim mail queue blocked on full disk: huge backlog on recover — watch queue length metric.
Predictable growth
Access logs ×10 during promo: pre-provision hourly rotation temporarily or disk upgrade before campaign.
MySQL binlog expiration: expire_logs_days or max_binlog_size — forgotten binlog = guaranteed full disk under write-heavy.
ZFS/btrfs snapshots count space — do not snapshot an already full disk in a loop.
Operational summary
Full disk on Linux production is almost always predictable with 80/90% alerts, tested log rotation, and known consumers (journald, Docker, binlog, local backups). Emergency: diagnose with df, du, lsof before rm.
Cloud: extend volume safer than blind purge if legit growth. Mandatory post-mortem if customer impact — often forgotten debug logging enabled.
Production readiness
Game day fill disk staging safely. Team executes runbook timed. Gaps updated same week.
Cloud ops note
Auto-scaling disk without alerting still produces outage during growth lag — monitor derivative of used space, not only threshold.
Kubernetes ephemeral storage limits kill pods before node disk full — different symptom same root cause log explosion.
Capacity planning note
Forecast disk growth using linear regression on weekly used GB — simple spreadsheet beats surprise 100% during marketing campaign. Include snapshot storage growth if snapshots live on same volume.
Separate volumes for logs, data, and backups when provider allows — blast radius reduction worth small cost increase.
Run game day quarterly: intentionally fill staging disk and execute runbook timed. Gaps found in game day are cheaper than gaps found in Black Friday.
Keep a dated runbook, before/after metrics, post-incident review — cumulative discipline beats Friday night panic.
Keep a dated runbook, before/after metrics, post-incident review — cumulative discipline beats Friday night panic.
Keep a dated runbook, before/after metrics, post-incident review — cumulative discipline beats Friday night panic.
Keep a dated runbook, before/after metrics, post-incident review — cumulative discipline beats Friday night panic.
Keep a dated runbook, before/after metrics, post-incident review — cumulative discipline beats Friday night panic.
Keep a dated runbook, before/after metrics, post-incident review — cumulative discipline beats Friday night panic.
Post-mortem
Friday debug log without rotation. K8s ephemeral storage kills pods before node full.
Deep dive linux-disk-full
Keep before/after metrics for each change, the last successful drill date, and documented gaps between host marketing and field measurement. A quarterly review with business teams avoids repeating the same mistakes. To compare infrastructure and read more field notes, browse our directory, comparison tool, and technical blog guides.:::note Key takeaway. Free space without destroying evidence; find the recurring consumer before closing the ticket. :::
Decide and move forward without blind spots
- 15-minute runbook —
df -hT,df -i, top-leveldu, journalctl --disk-usage, docker system df. - Find deleted open files —
lsof +L1if df/du gap. - Targeted action — journal vacuum, truncate rotated log, listed docker prune; never rm -rf without top 3 dirs.
- Alert on derivative — GB/day growth, not only 90% threshold.
- Post-mortem if prod impact — root cause documented same week.
Monitor disk and inodes on VPS via our comparison tool and directory.
Frequently asked questions
df 100% but du finds nothing?
Deleted but open files. lsof +L1 or restart the holding process.
What to purge first in emergency?
journalctl --vacuum, old rotated logs, /tmp — never /var/lib blindly (DB, docker).
Docker fills where?
/var/lib/docker/overlay2, unlimited json-file logs, orphan volumes.
Prevention?
Alerts at 80/90%, logrotate, journald limits, docker log max-size.
Next 100% disk: run du and lsof +L1 before any rm — future you will thank you.