mail
****

.. highlight:: python

https://gitlab.com/kb/mail

Icon::

  <i class="fa fa-envelope"></i>

We have other mail related apps:

- :doc:`app-mailchimp`
- :doc:`app-mandrill`. Do **NOT** use this app.
  The code was moved to the ``mail`` app.
- :doc:`sys-postfix`

Prerequisites
=============

Setup :doc:`celery`...

The mail app supports sending mail using Mandrill_ or :doc:`sys-sparkpost`

Requirements
============

Add the mail app to ``requirements/local.txt``::

  -e ../../app/mail

- `Mandrill Requirements`_
- `Sparkpost Requirements`_

Management Commands
===================

check-mail-size
---------------

Uses ``file_size_as_base64`` to check the size of file attachments when
encoded to base64?::

  django-admin check-mail-size 100

.. tip:: The parameter is the ``pk`` of the ``Mail`` model
         (source code in ``mail/management/commands/``).

email-has-clean-history
-----------------------

Check the email address to see if it has a clean history
i.e. has **not** been rejected for spam, bouncing etc::

  django-admin email-has-clean-history code@pkimber.net

.. tip:: The source code is in the ``mail`` app in
         ``mail/management/commands/email-has-clean-history.py``

email-init-rejected
-------------------

To reject an email address after checking Mandrill.
Has the following parameters:

1. email address (case is ignored)
2. Reason for rejection, ``hard-bounce``, ``soft-bounce`` or ``spam``
3. Name of the template (should be an optional parameter)

::

  django-admin email-init-rejected code@pkimber.net soft-bounce notify-weekly

.. tip:: The source code is in the ``mail`` app in
         ``mail/management/commands/email-init-rejected.py``

mail_send
---------

Send all pending mail messages::

  django-admin mail_send

.. tip:: Check ``MailManager``, ``get_mail_to_send`` for selection criteria.

Settings
========

In your ``settings/base.py``, add ``mail`` to your apps e.g::

  LOCAL_APPS = (
      'project',
      ...
      'mail',

And add the default from address::

  DEFAULT_FROM_EMAIL = 'patrick@hatherleigh.info'

Add the template type::

  MAIL_TEMPLATE_TYPE = get_env_variable("MAIL_TEMPLATE_TYPE")
  # if you are not using third party templates, you can use:
  MAIL_TEMPLATE_TYPE = "django"

.. warning:: Don't forget to set the ``EMAIL_BACKEND`` to use Mandrill or
             Sparkpost as appropriate.

In ``project/management/commands/start_scheduler.py``
(see :doc:`dev-apscheduler` for more information)::

  # process_mail
  scheduler.add_job(
      "mail.tasks:schedule_process_mail",
      "interval",
      minutes=60,
      id="schedule_process_mail",
      max_instances=1,
      replace_existing=True,
  )

- `Mandrill Settings`_
- `Sparkpost Settings`_

Deploy
======

- `Mandrill Deploy`_
- `Sparkpost Deploy`_

A ``host_name`` for your site will automatically be generated by Salt.  If you
need to override this, then you can set the ``host_name`` in the pillar.  For
more information, see :ref:`pillar_host_name`.

.. note:: Our standard is to use ``host_name`` rather than the Django
          ``build_absolute_uri`` method for adding links to email messages.

Usage
=====

.. important:: The ``slug`` value for the template name should always be in
               lower case.  We will make all Mandrill template names lower case
               - using underscores instead of a space.

Create a Template
-----------------

::

  from django.conf import settings
  from mail.models import MailTemplate

  # slug for the email template
  PAYMENT_THANKYOU = 'payment_thankyou'

  MailTemplate.objects.init_mail_template(
      PAYMENT_THANKYOU,
      'Thank you for your payment',
      (
          "You can add the following variables to the template:\n"
          "{{ NAME }} name of the customer.\n"
          "{{ DATE }} date of the transaction.\n"
          "{{ DESCRIPTION }} transaction detail.\n"
          "{{ TOTAL }} total value of the transaction."
      ),
      False,
      MailTemplate.MANDRILL,
      subject='Thank you for your payment',
      description="We will send you the course materials.",
  )

Queue an email
--------------

.. note:: In the examples below, ``self.object`` is an object which the email
          will be linked to.

To queue an email without using a template::

  from mail.models import Notify
  from mail.service import queue_mail_message

  email_addresses = [n.email for n in Notify.objects.all()]
  if email_addresses:
      queue_mail_message(
          self.object,
          email_addresses,
          subject,
          message,
      )
  else:
      logging.error(
          "Cannot send email notification of payment.  "
          "No email addresses set-up in 'mail.models.Notify'"
      )

To add attachments, add a list of file names to the ``queue_mail_message``
function e.g::

  from mail.service import queue_mail_message

  queue_mail_message(
      self.object,
      ['web@pkimber.net'],
      subject,
      message,
      attachments=['/temp/note.doc'],
      # attachments to be a simple path rather than a copy of the 'FileField'
      # attachments_use_path=True,
  )

.. note:: Attachments are only available on the ``queue_mail_message``
          function.  They do not work with templates (``queue_mail_template``).

Queue an email template
-----------------------

::

  from mail.service import queue_mail_template

  context = {
      'test@pkimber.net': {
          "DATE": created.strftime("%d-%b-%Y %H:%M:%S"),
          "DESCRIPTION": description,
          "NAME": "Re: {}".format(subject),
          "TOTAL": "123.34",
      },
  }
  queue_mail_template(
      self.object,
      'enquiry_acknowledgement',
      context,
  )

A dictionary or list can be added to the context if required e.g::

  context = {
      'test@pkimber.net': {
          "cake": ['Carrot', 'Ginger'],
          "fruit": {'a': 'Apple'},
      },
  }

.. note:: The data will be serialized and deserialized using ``json.dumps`` and
          ``json.loads``, so not all data types have their types preserved (for
          some unknown reason) e.g. ``datetime`` becomes a string.

Send queued emails
------------------

::

  from django.db import transaction
  from mail.tasks import process_mail

  # with a transaction
  transaction.on_commit(lambda: process_mail.send())

  # without a transaction
  process_mail.send()

To send email, you can use the ``mail_send`` management command e.g:

.. code-block:: bash

  django-admin mail_send

Rejected Mail
=============

If you look for rejected email addresses using Mandrill::

  # mandrill - find rejected email
  # https://www.kbsoftware.co.uk/crm/ticket/5388/
  scheduler.add_job(
      "mail.tasks:schedule_find_rejected_email",
      "cron",
      hour=2,
      minute=30,
      id="schedule_find_rejected_email",
      max_instances=1,
      replace_existing=True,
  )

Testing
=======

Unit Testing
------------

Pick some code from the following (just a quick reference)::

  from mail.models import Message
  from mail.tests.factories import MailTemplateFactory

  MailTemplateFactory(slug=Enrol.MAIL_TEMPLATE_CHARGE)
  NotifyFactory()

  # check email template context
  assert 1 == Message.objects.count()
  message = Message.objects.first()
  assert 1 == message.mail_set.count()
  mail = message.mail_set.first()
  assert 4 == mail.mailfield_set.count()
  assert {
      'description': '2 of 3 instalments',
      'enrol_description': '1 x Apple @ 100.00',
      'name': enrol.checkout_name,
      'total': '£{:.2f}'.format(Decimal('50.00')),
  } == {f.key: f.value for f in mail.mailfield_set.all()}

Maintenance
-----------

.. warning:: Only run the following command on a test site.  It will **mark all
             emails as sent** (which you wouldn't want on a live site)!

This will mark **all emails as sent**::

  from django.utils import timezone
  from mail.models import Mail

  Mail.objects.filter(sent__isnull=True).update(sent=timezone.now())

Mandrill
========

Mandrill Requirements
---------------------

Add the following to ``requirements/base.txt``::

  djrill==<current version e.g. 1.3.0>

See :doc:`dev-requirements` for the current version...

.. note:: We have started using `django-anymail`_ for :doc:`sys-sparkpost`.
          It would probably be a good idea to replace ``djrill`` with
          `django-anymail`_ at some stage
          (for more information, see :doc:`sys-sparkpost`).

Mandrill Settings
-----------------

Add the following to ``settings/production.py``::

  # mandrill
  EMAIL_BACKEND = 'djrill.mail.backends.djrill.DjrillBackend'
  MANDRILL_API_KEY = get_env_variable('MANDRILL_API_KEY')
  MANDRILL_USER_NAME = get_env_variable('MANDRILL_USER_NAME')

.. warning:: Don't forget to set the ``EMAIL_BACKEND`` to use Mandrill.

Mandrill Development
--------------------

For Mandrill add the following to your ``.private`` file e.g:

.. code-block:: bash

  export MANDRILL_API_KEY="your-api-key"
  export MANDRILL_USER_NAME="notify@hatherleigh.info"

Mandrill Deploy
---------------

In the salt pillar ``sls`` file for your site, add the
``mandrill_api_key`` and ``mandrill_user_name`` e.g::

  sites:
    my_site:
      celery: True
      env:
        mail_template_type: <'sparkpost' | 'mandrill' | 'django'>
        mandrill_api_key: your-api-key
        mandrill_user_name: <mandrill user name>

.. _mandrill_template:

Mandrill Template
-----------------

.. tip:: The example_mail_template_ management command has some sample code
         showing the following features.

Mandrill Templates are usually designed in MailChimp and then
*Sent to Mandrill*.

When you `Queue an email template`_, you can add variables to the templates.
To add line breaks to these variables::

  context={
      "test@pkimber.net": {
          "COLOURS": "1. Red<br />2. Green<br />3. Blue",

You can also add HTML links::

  context={
      "test@pkimber.net": {
          "ACTION": '<a href="https://www.kbsoftware.co.uk/" mc:disable-tracking>Click here!</a>',

.. tip:: The ``mc:disable-tracking`` attribute prevents click tracking and makes
         the generated HTML much easier to read
         (from `Can I disable click-tracking on selected links in my email?`_).

.. warning:: For some reason HTML links do not work when loaded from
             the Mandrill *View content* option in *Outbound*, *Activity*.
             I am hoping they work when the email is live!

To disable tracking in a MailChimp template:

Click the chevron icon ``<>``.

Add ``mc:disable-tracking`` to the anchor tag e.g.

.. image:: ./misc/mailchimp-click-tracking.png

The variable name for unsubscribe is ``UNSUB``:

.. code-block:: html

  To unsubscribe from this list <a href="*|UNSUB|*" mc:disable-tracking="">click here...</a><br />

SMTP
====

SparkPost
---------

- :ref:`mail-sparkpost-smtp`

Google Mail
-----------

`SMTP settings to send mail from a printer, scanner, or app`_:

.. image:: ./misc/mail-google-smtp-relay-service.png

.. note:: I failed to get this working for several days.
          It finally worked when I set *Require TLS encryption:* to **No**.
          I then realised that ``EMAIL_USE_TLS`` was not in setttings, so it was
          defaulting to ``False``.

Add the following to ``settings/production.py``::

  EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
  EMAIL_HOST = 'smtp-relay.gmail.com'
  EMAIL_HOST_USER = get_env_variable('EMAIL_HOST_USER')
  EMAIL_PORT = 587
  EMAIL_USE_TLS = True

Standard SMTP
-------------

::

  EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
  EMAIL_HOST = get_env_variable('EMAIL_HOST')
  EMAIL_HOST_USER = get_env_variable('EMAIL_HOST_USER')
  EMAIL_PORT = int(get_env_variable('EMAIL_PORT'))
  EMAIL_USE_TLS = get_env_variable_bool('EMAIL_USE_TLS')

SparkPost
=========

.. tip:: We have started using `django-anymail`_ instead of ``sparkpost`` to
         `allow use of the EU account`_
         and to
         `add track options`_.

Links

- :ref:`sparkpost_blacklist`

SparkPost Requirements
----------------------

Add the following to ``requirements/base.txt``::

  sparkpost==<current version e.g. 1.3.0>

See :doc:`dev-requirements` for the current version.

SparkPost Settings
------------------

Add the following to ``settings/base.py``::

  # sparkpost
  EMAIL_BACKEND = 'sparkpost.django.email_backend.SparkPostEmailBackend'
  SPARKPOST_API_KEY = get_env_variable('SPARKPOST_API_KEY')

  SPARKPOST_OPTIONS = {
    'track_opens': False,  # or True as required
    'track_clicks': False,  # or True as required
    'transactional': True,
  }

SparkPost Development
---------------------

Add the following to your ``.private`` file e.g:

.. code-block:: bash

  export SPARKPOST_API_KEY="your-api-key"

SparkPost Deploy
----------------

In the salt pillar ``sls`` file for your site add the ``sparkpost_api_key``
e.g::

  sites:
    my_site:
      celery: True
      env:
        mail_template_type: <'sparkpost' | 'mandrill' | 'django'>
        sparkpost_api_key: your-api-key

.. tip:: :doc:`sys-sparkpost` for API, Domains and SMTP

Convert from Mandrill
---------------------

In ``requirements/base.txt`` include the line::

  sparkpost==<current sparkpost version>

e.g. at the time of writing the latest version is 1.3.0::

  sparkpost==1.3.0

If mandrill was previously used remove the line::

  djrill==2.1.0

In your ``.private`` file for the project include the lines::

  export SPARKPOST_API_KEY='<your sparkpost api key>'

  unset MANDRILL_API_KEY
  unset MANDRILL_USER_NAME

If mandrill was previously included in your project remove the lines::

  export MANDRILL_API_KEY='<your mandrill api key>'
  export MANDRILL_USER_NAME='<your mandrill username>'

In ``settings/base.py`` include the line::

  DEFAULT_FROM_EMAIL = 'user@example.com'

  EMAIL_BACKEND = 'sparkpost.django.email_backend.SparkPostEmailBackend'

  SPARKPOST_API_KEY = get_env_variable('SPARKPOST_API_KEY')

  SPARKPOST_OPTIONS = {
    'track_opens': False,  # or True as required
    'track_clicks': False,  # or True as required
    'transactional': True,
  }

If mandrill was previously included in the project remove the lines (search
for ``EMAIL_BACKEND`` in the settings directory to make sure you get all
occurances)::

  EMAIL_BACKEND = 'djrill.mail.backends.djrill.DjrillBackend'

  MANDRILL_API_KEY = get_env_variable('MANDRILL_API_KEY')
  MANDRILL_USER_NAME = get_env_variable('MANDRILL_USER_NAME')

In ``settings/base.py`` alter the ``THIRD_PARTY_APPS``
(included in ``INSTALLED_APPS``) to include the ``sparkpost`` app for example::

  THIRD_PARTY_APPS = (
     'easy_thumbnails',
     'reversion',
     'captcha',
     'sparkpost',
  )

If mandrill was previously used in a the project remove the following line
from ``THIRD_PARTY_APPS``::

  'djrill',

Tips
====

Celery (or cron)
----------------

If you are **not** using Celery add the the ``mail_send`` cron command e.g::

  sites:
    my_site:
      celery: True
      cron:
        mail_send:
          schedule: "*/5    *       *       *       *"
      env:
        mail_template_type: <'sparkpost' | 'mandrill' | 'django'>
        sparkpost_api_key: your-api-key

Password Reset
--------------

When testing the password reset workflow, make sure you use a valid email
address for a user.  On the standard demo data, this will be ``web@pkimber.net``


.. _`add track options`: https://gitlab.com/kb/mail/commit/ada579bcce2d164281ce1b86e9253ecd80ae2c06
.. _`allow use of the EU account`: https://gitlab.com/kb/mail/commit/caca04b65bb5d6a6218337880807aa981a951b77
.. _`Can I disable click-tracking on selected links in my email?`: https://mandrill.zendesk.com/hc/en-us/articles/205582927-Can-I-disable-click-tracking-on-selected-links-in-my-email-
.. _`django-anymail`: https://github.com/anymail/django-anymail
.. _`SMTP settings to send mail from a printer, scanner, or app`: https://support.google.com/a/answer/176600?hl=en
.. _example_mail_template: https://gitlab.com/kb/mail/blob/master/example_mail/management/commands/example_mail_template.py