systemd
*******



Systemd logging
===============

Systemd writes messages to a binary log stored in /var/log/journal this cannot
be viewed directly it can be viewed using the `journalctl` command.  Without
any options the entire log is viewed::

  journalctl

Messages since the last boot can be viewed using::

  journalctl -b

Messages since a point in time can be viewd using the `-S` or --since options.
For example message today since 10:00 as displayed using::

  journalctl --since=10:00

Message since 11:00am on 10/04/2023 can be displayed using (quotes are required
since there is a space between the date and the time)::

  journalctl --since="2023-04-10 11:00"

To list the amount of disk space used by the journal files::

  journalctl --disk-usage

Remove old entries (e.g. greater than 10 days) from ``systemd`` journal::

  sudo journalctl --vacuum-time=10d

Or to restrict the journal to a specified size use::

  sudo journalctl --vacuum-size=100M

.. _sys-systemd-journal-configuration:

Journal Configuration
---------------------

.. warning:: ``/etc/systemd/journald.conf`` is the master configuration for the
             ``systemd`` journal - changes to this may be overwritten by an
             upgrade so it's better to add a file to the
             ``/etc/systemd/journald.conf.d`` directory.

             The latest salt states add a configuration setting to
             ``journal.conf.d/`` to restrict the time messages are retained 
             to 10 days.

The salt states add a file which performs the vacuum command automatically::

  /etc/systemd/journald.conf.d/journald.conf

The contents of that file are::

  [Journal]
  MaxRetentionSec=10d

Changes will take effect from next time the systemd-journald service is
restarted, Either at system boot or using::

  sudo systemctl restart systemd-journald


Investigating issues
--------------------

We've noticed that a typical system with 10 days retention will have from 32M
to 150Mb.  If the log is larger than 150Mb, it may indicate an issue on the
system that is causing messages to be logged.

Look for repeated error messages in the log.  e.g. recently we found the
following messages repeatedly in the log for a system::

  Apr 09 00:00:06 SYSTEMNAME systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
  Apr 09 00:00:06 SYSTEMNAME systemd[1]: logrotate.service: Failed with result 'exit-code'.
  Apr 09 00:00:06 SYSTEMNAME systemd[1]: Failed to start Rotate log files.

On a healthy system this would read::

  Apr 09 00:00:02 SYSTEMNAME systemd[1]: logrotate.service: Succeeded.

Using this information, one can investigate the logrotate service and find the 
rule that is causing the failure.