Systemd writes messages to a binary log stored in /var/log/journal this cannot be viewed directly it can be viewed using the journalctl command. Without any options the entire log is viewed:
Messages since the last boot can be viewed using:
Messages since a point in time can be viewd using the -S or –since options. For example message today since 10:00 as displayed using:
Message since 11:00am on 10/04/2023 can be displayed using (quotes are required since there is a space between the date and the time):
journalctl --since="2023-04-10 11:00"
To list the amount of disk space used by the journal files:
Remove old entries (e.g. greater than 10 days) from
sudo journalctl --vacuum-time=10d
Or to restrict the journal to a specified size use:
sudo journalctl --vacuum-size=100M
/etc/systemd/journald.conf is the master configuration for the
systemd journal - changes to this may be overwritten by an
upgrade so it’s better to add a file to the
The latest salt states add a configuration setting to
journal.conf.d/ to restrict the time messages are retained
to 10 days.
The salt states add a file which performs the vacuum command automatically:
The contents of that file are:
Changes will take effect from next time the systemd-journald service is restarted, Either at system boot or using:
sudo systemctl restart systemd-journald
We’ve noticed that a typical system with 10 days retention will have from 32M to 150Mb. If the log is larger than 150Mb, it may indicate an issue on the system that is causing messages to be logged.
Look for repeated error messages in the log. e.g. recently we found the following messages repeatedly in the log for a system:
Apr 09 00:00:06 SYSTEMNAME systemd: logrotate.service: Main process exited, code=exited, status=1/FAILURE Apr 09 00:00:06 SYSTEMNAME systemd: logrotate.service: Failed with result 'exit-code'. Apr 09 00:00:06 SYSTEMNAME systemd: Failed to start Rotate log files.
On a healthy system this would read:
Apr 09 00:00:02 SYSTEMNAME systemd: logrotate.service: Succeeded.
Using this information, one can investigate the logrotate service and find the rule that is causing the failure.