The log that ate the disk

The volume was 38 GB. Six weeks after a service started logging every request at debug level, it was full, and the database stopped accepting writes at exactly the wrong hour.

What I check first now

journalctl --disk-usage
du -sh /var/log /var/cache /var/lib/apt/lists

The last one surprises people. An apt cache that nobody has cleared since the machine was built is routinely a couple of hundred megabytes.

The two settings

A cap on the journal, and a cap per service. Both belong in configuration management rather than in your memory of having run a command once.

[Journal]
SystemMaxUse=200M
MaxRetentionSec=1month

Neither is clever. That is the point: the interesting version of this problem only arrives once you have ruled out the boring one.