Monitor - Grafana (Loki), Elastic and Raygun
********************************************

.. highlight:: python

Grafana / Loki / Promtail
=========================

- For help and support, try the `Grafana Labs Community Forum`_

Grafana
-------

*Alerting*, *Contact points*, edit ``grafana-default-email`` e.g.

.. image:: ./misc/monitor/2022-06-07-grafana-alert-email.png

Promtail
--------

To add logging to a server, add ``config.promtail`` to the ``top.sls`` file in
the pillar e.g::

  base:
    '*':
      - global.users
  'server':
    - config.promtail
    - config.search
    - config.workflow

Where ``config.promtail`` contains the IP address of the Loki host e.g::

  promtail:
    loki_host: 10.10.2.41

To check the status of the Promtail service on a server::

  journalctl -u promtail.service -f

Raygun
======

In ``settings/production.py``::

  # raygun middleware added in ``production.py``
  # See note dated 12/02/2021 on
  # https://www.kbsoftware.co.uk/crm/ticket/124/
  MIDDLEWARE = ("raygun4py.middleware.django.Provider",) + MIDDLEWARE

  RAYGUN4PY_CONFIG = {
      "api_key": get_env_variable("RAYGUN4PY_API_KEY"),
      "ignored_exceptions": ["EmptyPage", "Http404", "PermissionDenied"],
      "transmit_environment_variables": False,
  }

To get the name of the exception to add to ``ignored_exceptions``
e.g. for ``EmptyPage``::

  django-admin.py shell                                                                                                                       07:44:28
  >>> from django.core.paginator import EmptyPage
  >>> EmptyPage.__name__
  'EmptyPage'

In ``settings/base.py`` (just a comment)::

  MIDDLEWARE = (
      # raygun middleware added in ``production.py``
      # See note dated 12/02/2021 on
      # https://www.kbsoftware.co.uk/crm/ticket/124/
      # "raygun4py.middleware.django.Provider",

In ``.env.fish``::

  set -x RAYGUN4PY_API_KEY "my-raygun-api-key"

In ``.gitlab-ci.yml```::

  test:
    script:
    - export RAYGUN4PY_API_KEY="my-raygun-api-key"

Testing
-------

::

  www.hatherleigh.info.sh shell
  >>> from raygun4py.cli import send_test_exception
  >>> from django.conf import settings
  >>> send_test_exception(settings.RAYGUN4PY_CONFIG["api_key"])

Elastic - Hosted Server
=======================

.. tip:: I think we need to install X-Pack to get the APM dashboards.
         (from `elastic, Doubling Down on Open`_,
         *X-Pack features - ... dedicated APM UIs*)

30/12/2020
We are using the Elastic hosted service, so no need to setup a server
(for now).

Connectors
----------

To configure Sparkpost:

.. image:: ./misc/elastic-sparkpost-connector.png

.. tip:: For more information, see
         https://www.sparkpost.com/blog/setup-sparkpost-smtp-relay/

Server Configuration
====================

To create a server acting as a ``monitor`` using Elastic APM, add the following
to the ``top.sls`` file of your pillar:

.. code-block:: yaml

  base:
    '*':
      - global.users
    'elastic-*':
      - config.firewall.monitor
      - config.kibana
      - config.search

.. tip:: My monitor servers all start with a ``hostname`` of ``elastic-``.

``config/kibana.sls``:

.. code-block:: yaml

  kibana:
    True

``config/search.sls``:

.. code-block:: yaml

  search:
    True

``config/firewall/monitor.sls``:

.. code-block:: yaml

  firewall:
    monitor:
      # My server
      - 55.144.55.144
      # My other server
      - 133.55.133.55

Re-start services (it might be a good idea to ``tail`` the log files first)::

  service elasticsearch restart
  service kibana restart
  service apm-server restart

Check the log files::

  tail -f /var/log/syslog
  tail -f /var/log/elasticsearch/elasticsearch.log
  tail -f /var/log/apm-server/apm-server

.. warning:: If you re-start ``apm-server``, the log file will be re-created,
             so you will need to re-start the ``tail``.

Tips
====

To run the APM server in the foreground:

.. code-block:: bash

  service apm-server stop
  apm-server -e

To view the web interface of the monitor server on your workstation:

.. code-block:: bash

  ssh -N -L 8888:localhost:5601 monitor.yourbiz.co.uk

.. note:: This command tunnels the HTTP service from port 5601 on the monitor
          server (Kibana) to your local workstation on port 8888.

Browse to: http://localhost:8888/ to view the dashboards.

To make this easier, add the following to ``~/.ssh/config``::

  Host elasticapm
      Hostname monitor.yourbiz.co.uk
      LocalForward 8888 localhost:5601

Now you can type ``ssh -N elasticapm`` to run it in the foreground and
``ssh -f -N elasticapm`` to run it in the background.

You could then add the following to your ``~/.bashrc`` which means that the
monitoring is available when you turn on your computer
(it's wrapped in an ``if`` statement so the ssh tunnel is only set up the first
time the bash startup file is executed)::

  if [ "$(ss --listen | grep 8888 | head -n 1)" = "" ]; then
      ssh -f -N elasticapm
  fi

Site Configuration
==================

Add the following to ``requirements/base.txt``::

  elastic-apm==

.. tip:: See :doc:`dev-requirements` for the current version.

In ``settings/base.py``::

  MIDDLEWARE = (
      'elasticapm.contrib.django.middleware.TracingMiddleware',

  THIRD_PARTY_APPS = (
      'elasticapm.contrib.django',

.. note:: Make sure that ``TracingMiddleware`` is the first ``MIDDLEWARE`` in
          the list.

In ``settings/base.py``, add the following to ``LOGGING``
(`Elastic APM and Django Logging`_)::

  "handlers": {
      "elasticapm": {
          "level": "WARNING",
          "class": "elasticapm.contrib.django.handlers.LoggingHandler",
      },

  "loggers": {
      # Log errors from the Elastic APM module to the console (recommended)
      "elasticapm.errors": {
          "level": "ERROR",
          "handlers": ["console", "logfile"],
          "propagate": False,
      },

In ``settings/local.py``::

  ELASTIC_APM = {"DISABLE_SEND": True}

In ``settings/production.py``::

  DOMAIN = get_env_variable('DOMAIN')
  DATABASE = DOMAIN.replace('.', '_').replace('-', '_')

  ELASTIC_APM = {
      "DISABLE_SEND": get_env_variable_bool("MONITOR_DISABLE_SEND"),
      "SECRET_TOKEN": get_env_variable("MONITOR_SECRET_TOKEN"),
      "SERVER_URL": get_env_variable("MONITOR_SERVER_URL"),
      "SERVICE_NAME": DATABASE_NAME,
  }

.. note:: Refer to your company :doc:`checklist` and replace
          ``monitor.yourbiz.co.uk`` with the name of your server.

.. note:: ``DOMAIN`` and ``DATABASE`` should already exist in ``production.py``.

.. _monitor_firewall_config:

Deploy
------

Find the external IP address of your workstation::

  dig +short myip.opendns.com @resolver1.opendns.com

Add it to ``config/firewall/monitor.sls`` in your pillar e.g:

.. code-block:: yaml

  firewall:
    monitor:
      # web server
      - 13.16.24.14
      # yourbiz
      - 91.85.167.37

.. tip:: Please add a comment to show which server / workstation has the IP
         address.

Copy the pillar to your Salt master and run a ``state.apply`` to update the
firewall on your monitor server e.g:

.. code-block:: bash

  salt 'elastic-monitor' state.apply --state-verbose=False

Check you have included ``config/monitor.sls`` in your pillar e.g:

.. code-block:: yaml

  monitor:
    server_url: http://monitor.yourbiz.co.uk:8200

To check your configuration, run::

  python manage.py elasticapm check

To send a test exception using the current settings, run::

  python manage.py elasticapm test

To disable monitoring for your test (or temporary) site:

.. code-block:: yaml

  # sites/my.sls
  sites:
    hatherleigh_info:
      monitor: False

Curator
=======

Delete / clean APM monitor indexes...

Install::

  wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

  # update
  # /etc/apt/sources.list.d/curator.list
  # and add
  deb [arch=amd64] https://packages.elastic.co/curator/5/debian9 stable main

  apt update
  apt install elasticsearch-curator

List ``apm`` indexes::

  curator_cli --host localhost show_indices --filter_list '[{"filtertype":"pattern","kind":"prefix","value":"apm-"}]'

Clean indexes older than 60 days::

  curator_cli --host localhost delete_indices --filter_list '[{"filtertype":"pattern","kind":"prefix","value":"apm-7.2.0-"}, {"filtertype":"age","source":"name","timestring":"%Y.%m.%d","unit":"days","unit_count":1,"direction":"older"}]'

.. tip:: Update the ``value`` to match the indexes you want to delete.

Old Notes
=========

- :doc:`old/monitor`


.. _`Elastic APM and Django Logging`: https://www.elastic.co/guide/en/apm/agent/python/current/django-support.html#django-logging
.. _`elastic, Doubling Down on Open`: https://www.elastic.co/blog/doubling-down-on-open
.. _`Grafana Labs Community Forum`: https://community.grafana.com/c/grafana-loki/41