Monitor - Elastic APM

Tip

I think we need to install X-Pack to get the APM dashboards. (from elastic, Doubling Down on Open, X-Pack features - … dedicated APM UIs)

Server Configuration

To create a server acting as a monitor using Elastic APM, add the following to the top.sls file of your pillar:

base:
  '*':
    - global.users
  'elastic-*':
    - config.firewall.monitor
    - config.kibana
    - config.search

Tip

My monitor servers all start with a hostname of elastic-.

Re-start services (it might be a good idea to tail the log files first):

service elasticsearch restart
service kibana restart
service apm-server restart

Check the log files:

tail -f /var/log/syslog
tail -f /var/log/elasticsearch/elasticsearch.log
tail -f /var/log/apm-server/apm-server

Warning

If you re-start apm-server, the log file will be re-created, so you will need to re-start the tail.

Tips

To run the APM server in the foreground:

service apm-server stop
apm-server -e

To view the web interface of the monitor server on your workstation:

ssh -N -L 8888:localhost:5601 monitor.yourbiz.co.uk

Note

This command tunnels the HTTP service from port 5601 on the monitor server (Kibana) to your local workstation on port 8888.

Browse to: http://localhost:8888/ to view the dashboards.

To make this easier, add the following to ~/.ssh/config:

Host elasticapm
    Hostname monitor.yourbiz.co.uk
    LocalForward 8888 localhost:5601

Now you can type ssh -N elasticapm to run it in the foreground and ssh -f -N elasticapm to run it in the background.

You could then add the following to your ~/.bashrc which means that the monitoring is available when you turn on your computer (it’s wrapped in an if statement so the ssh tunnel is only set up the first time the bash startup file is executed):

if [ "$(ss --listen | grep 8888 | head -n 1)" = "" ]; then
    ssh -f -N elasticapm
fi

Site Configuration

Add the following to requirements/base.txt:

elastic-apm==

Tip

See Requirements for the current version.

In settings/base.py:

MIDDLEWARE = (
    'elasticapm.contrib.django.middleware.TracingMiddleware',

THIRD_PARTY_APPS = (
    'elasticapm.contrib.django',

Note

Make sure that TracingMiddleware is the first MIDDLEWARE in the list.

In settings/local.py:

ELASTIC_APM = {"DISABLE_SEND": True}

In settings/production.py:

DOMAIN = get_env_variable('DOMAIN')
DATABASE = DOMAIN.replace('.', '_').replace('-', '_')

ELASTIC_APM = {
    'DISABLE_SEND': get_env_variable_bool('MONITOR_DISABLE_SEND'),
    'SERVER_URL': get_env_variable('MONITOR_SERVER_URL'),
    'SERVICE_NAME': DATABASE,
}

Note

Refer to your company Checklist and replace monitor.yourbiz.co.uk with the name of your server.

Note

DOMAIN and DATABASE should already exist in production.py.

Deploy

Find the external IP address of your workstation:

dig +short myip.opendns.com @resolver1.opendns.com

Add it to config/firewall/monitor.sls in your pillar e.g:

firewall:
  monitor:
    # web server
    - 13.16.24.14
    # yourbiz
    - 91.85.167.37

Tip

Please add a comment to show which server / workstation has the IP address.

Copy the pillar to your Salt master and run a state.apply to update the firewall on your monitor server e.g:

salt 'elastic-monitor' state.apply --state-verbose=False

Check you have included config/monitor.sls in your pillar e.g:

monitor:
  server_url: http://monitor.yourbiz.co.uk:8200

To check your configuration, run:

python manage.py elasticapm check

To send a test exception using the current settings, run:

python manage.py elasticapm test

To disable monitoring for your test (or temporary) site:

# sites/my.sls
sites:
  hatherleigh_info:
    monitor: False

Old Notes