Monitor - Elastic APM

Tip

I think we need to install X-Pack to get the APM dashboards. (from elastic, Doubling Down on Open, X-Pack features - … dedicated APM UIs)

Server Configuration

To create a server acting as a monitor using Elastic APM, add the following to the top.sls file of your pillar:

base:
  '*':
    - global.users
  'elastic-*':
    - config.firewall
    - config.kibana
    - config.search

Tip

My monitor servers all start with a hostname` of elastic-.

Install X-Pack in ElasticSearch:

cd /usr/share/elasticsearch
bin/elasticsearch-plugin install x-pack

service elasticsearch restart
bin/x-pack/setup-passwords auto

# you will get a response like this
#
# Changed password for user kibana
# PASSWORD kibana = AbCdEfGhIjK
#
# Changed password for user logstash_system
# PASSWORD logstash_system = A1B2C3D4E5
#
# Changed password for user elastic
# PASSWORD elastic = XaYbZcPdRe

Install X-Pack in Kibana:

cd /usr/share/kibana/
sudo -u kibana bin/kibana-plugin install x-pack

Note

This will take some time…

In your pillar, update config/kibana.sls with the password for user elastic (see above) e.g:

kibana:
  elastic_username: elastic
  elastic_password: XaYbZcPdRe

Re-start services (it might be a good idea to tail the log files first):

service elasticsearch restart
service kibana restart
service apm-server restart

Check the log files:

tail -f /var/log/syslog
tail -f /var/log/elasticsearch/elasticsearch.log
tail -f /var/log/apm-server/apm-server

Tips

To run the APM server in the foreground:

service apm-server stop
apm-server -e

To view the web interface of the monitor server on your workstation:

ssh -N -L 8888:localhost:5601 monitor.yourbiz.co.uk

Note

This command tunnels the HTTP service from port 5601 on the monitor server (Kibana) to your local workstation on port 8888.

Browse to: http://localhost:8888/ to view the dashboards.

Site Configuration

Add the following to requirements/base.txt:

elastic-apm==

Tip

See Requirements for the current version.

In settings/base.py:

MIDDLEWARE = (
    'elasticapm.contrib.django.middleware.TracingMiddleware',

THIRD_PARTY_APPS = (
    'elasticapm.contrib.django',

Note

Make sure that TracingMiddleware is the first MIDDLEWARE in the list.

In settings/production.py:

DOMAIN = get_env_variable('DOMAIN')
DATABASE = DOMAIN.replace('.', '_').replace('-', '_')

ELASTIC_APM = {
    'DISABLE_SEND': get_env_variable_bool('MONITOR_DISABLE_SEND'),
    'SERVER_URL': get_env_variable('MONITOR_SERVER_URL'),
    'SERVICE_NAME': DATABASE,
}

Note

Refer to your company Checklist and replace monitor.yourbiz.co.uk with the name of your server.

Note

DOMAIN and DATABASE should already exist in production.py.

Deploy

Find the IP address of your server. Add it to config/firewall.sls in your pillar e.g:

firewall:
  monitor:
    # server
    - 139.162.245.14
    # yourbiz
    - 91.85.167.37

Tip

Please add a comment to show which server / workstation has the IP address.

Copy the pillar to your Salt master and run a highstate to update the firewall on your monitor server e.g:

salt 'elastic-monitor' state.highstate --state-verbose=False

Check you have included config/monitor.sls in your pillar e.g:

monitor:
  server_url: http://monitor.yourbiz.co.uk:8200

To check your configuration, run:

python manage.py elasticapm check

To send a test exception using the current settings, run:

python manage.py elasticapm test

To disable monitoring for your test (or temporary) site:

# sites/my.sls
sites:
  hatherleigh_info:
    monitor: False

Old Notes