mail

https://gitlab.com/kb/mail

Prerequisites

Setup Celery (using Redis)

The mail app supports sending mail using Mandrill or Sparkpost

SparkPost

Using http://www.mail-tester.com/ we found our domain was blacklisted. The mail app has a facility for sending a test email: Navigate to /mail/create/.

We emailed SparkPost, and they sent the following advice:

Thank you for reporting the blacklist issue!

If you look at the raw reason provided:

522 email sent from spmailtechn.com found on industry URI blacklists
on 2017/05/10 18:11:12 BST, please contact your mail sending service or use
an alternate email service to send your email.
Guide for bulk senders www.bt.com/bulksender

… it is the domain spmailtechn.com that has been blacklisted and not the IP. Your messages are containing spmailtechn.com in the headers because you haven’t set-up a custom bounce domain yet. In order to do so, you can find more information through following link: https://support.sparkpost.com/customer/portal/articles/2371794

Adding a custom bounce domain to your account will also remove spmailtechn.com from your headers and help fully brand all of your emails and also protect you from other senders on SparkPost.

We did the following:

Added a DMARC record to our Digital Ocean account:

TXT / _dmarc.kbsoftware.co.uk / returns v=DMARC1; p=none"
_images/digital-ocean-sparkpost.png

Added a CNAME record to our Digital Ocean account:

mail.kbsoftware.co.uk / is an alias of sparkpostmail.com

Added mail.kbsoftware.co.uk as a Bounce Domain in Sparkpost:

_images/sparkpost-bounce-domain.png

Requirements

Add the following to requirements/base.txt:

sparkpost==<current version e.g. 1.3.0>

or:

djrill==<current version e.g. 1.3.0>

Tip

See Requirements for the current version.

Add the mail app to requirements/local.txt:

-e ../../app/mail

Settings

In your settings/base.py, add mail to your apps e.g:

LOCAL_APPS = (
    'project',
    ...
    'mail',

And add the default from address:

DEFAULT_FROM_EMAIL = 'patrick@hatherleigh.info'

Add the template type (this is optional)

MAIL_TEMPLATE_TYPE = get_env_variable("MAIL_TEMPLATE_TYPE")

Warning

Don’t forget to set the EMAIL_BACKEND to use Mandrill or Sparkpost as appropriate.

For Mandrill, add the following to settings/base.py:

# mandrill
EMAIL_BACKEND = 'djrill.mail.backends.djrill.DjrillBackend'
MANDRILL_API_KEY = get_env_variable('MANDRILL_API_KEY')
MANDRILL_USER_NAME = get_env_variable('MANDRILL_USER_NAME')

For Sparkpost, add the following to settings/base.py:

# sparkpost
EMAIL_BACKEND = 'sparkpost.django.email_backend.SparkPostEmailBackend'
SPARKPOST_API_KEY = get_env_variable('SPARKPOST_API_KEY')

SPARKPOST_OPTIONS = {
  'track_opens': False,  # or True as required
  'track_clicks': False,  # or True as required
  'transactional': True,
}

In settings/production.py (after CELERY_DEFAULT_QUEUE) (see Celery (using Redis) for more information):

from celery.schedules import crontab
CELERYBEAT_SCHEDULE = {
    'process_mail': {
        'task': 'mail.tasks.process_mail',
        'schedule': crontab(minute='1', hour='*/1'),
    },
}

Tip

Tip

When testing the password reset workflow, make sure you use a valid email address for a user. On the standard demo data, this will be web@pkimber.net

Google Mail

Add the following to settings/production.py:

EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
EMAIL_HOST = 'smtp-relay.gmail.com'
EMAIL_HOST_USER = get_env_variable('EMAIL_HOST_USER')
EMAIL_PORT = 587
EMAIL_USE_TLS = True

Mandrill

Requirements

Add the following to requirements/base.txt:

djrill

Add the mail app to requirements/local.txt:

-e ../../app/mail

Settings

Add the following to settings/production.py:

# mandrill
EMAIL_BACKEND = 'djrill.mail.backends.djrill.DjrillBackend'
MANDRILL_API_KEY = get_env_variable('MANDRILL_API_KEY')
MANDRILL_USER_NAME = get_env_variable('MANDRILL_USER_NAME')

Warning

Don’t forget to set the EMAIL_BACKEND to use Mandrill.

Development

For Mandrill add the following to your .private file e.g:

export MANDRILL_API_KEY="your-api-key"
export MANDRILL_USER_NAME="notify@hatherleigh.info"

For Sparkpost add the following to your .private file e.g:

export SPARKPOST_API_KEY="your-api-key"

Deploy

In the salt pillar sls file for your site, if using mandrill add the mandrill_api_key and mandrill_user_name, if you’re using sparkpost add the ‘sparkpost_api_key’. If you are not using Celery add the the mail_send cron command e.g:

sites:
  my_site:
    celery: True
    cron:
      mail_send:
        schedule: "*/5    *       *       *       *"
    env:
      mail_template_type: <'sparkpost' | 'mandrill' | 'django'>
      sparkpost_api_key: your-api-key

if using Mandrilli replace the sparkpost_api_key key with these:

mandrill_api_key: your-api-key
mandrill_user_name: <mandrill user name>

A host_name for your site will automatically be generated by Salt. If you need to override this, then you can set the host_name in the pillar. For more information, see Host Name.

Note

Our standard is to use host_name rather than the Django build_absolute_uri method for adding links to email messages.

Usage

Important

The slug value for the template name should always be in lower case. We will make all Mandrill template names lower case - using underscores instead of a space.

Create a mail template:

from django.conf import settings
from mail.models import MailTemplate

# slug for the email template
PAYMENT_THANKYOU = 'payment_thankyou'

MailTemplate.objects.init_mail_template(
    PAYMENT_THANKYOU,
    'Thank you for your payment',
    (
        "You can add the following variables to the template:\n"
        "{{ NAME }} name of the customer.\n"
        "{{ DATE }} date of the transaction.\n"
        "{{ DESCRIPTION }} transaction detail.\n"
        "{{ TOTAL }} total value of the transaction."
    ),
    False,
    MailTemplate.MANDRILL,
    subject='Thank you for your payment',
    description="We will send you the course materials.",
)

Queue the email:

Note

In the examples below, self.object is an object which the email will be linked to.

To queue an email without using a template:

from mail.models import Notify
from mail.service import queue_mail_message

email_addresses = [n.email for n in Notify.objects.all()]
if email_addresses:
    queue_mail_message(
        self.object,
        email_addresses,
        subject,
        message,
    )
else:
    logging.error(
        "Cannot send email notification of payment.  "
        "No email addresses set-up in 'mail.models.Notify'"
    )

To add attachments, add a list of file names to the queue_mail_message function e.g:

queue_mail_message(
    self.object,
    ['web@pkimber.net'],
    subject,
    message,
    attachments=['/temp/note.doc']
)

Note

Attachments are only available on the queue_mail_message function. They do not work with templates (queue_mail_template).

To queue an email template:

from mail.service import queue_mail_template

context = {
    'test@pkimber.net': {
        "DATE": created.strftime("%d-%b-%Y %H:%M:%S"),
        "DESCRIPTION": description,
        "NAME": "Re: {}".format(subject),
        "TOTAL": "123.34",
    },
}
queue_mail_template(
    self.object,
    'enquiry_acknowledgement',
    context,
)

A dictionary or list can be added to the context if required e.g:

context = {
    'test@pkimber.net': {
        "cake": ['Carrot', 'Ginger'],
        "fruit": {'a': 'Apple'},
    },
}

Note

The data will be serialized and deserialized using json.dumps and json.loads, so not all data types have their types preserved (for some unknown reason) e.g. datetime becomes a string.

To send queued emails:

from mail.tasks import process_mail

# with a transaction
transaction.on_commit(lambda: process_mail.delay())

# without a transaction
process_mail.delay()

To send email, use the mail_send management command e.g:

django-admin.py mail_send

Testing

Unit Testing

Pick some code from the following (just a quick reference):

from mail.models import Message

MailTemplateFactory(slug=Enrol.MAIL_TEMPLATE_CHARGE)
NotifyFactory()

# check email template context
assert 1 == Message.objects.count()
message = Message.objects.first()
assert 1 == message.mail_set.count()
mail = message.mail_set.first()
assert 4 == mail.mailfield_set.count()
assert {
    'description': '2 of 3 instalments',
    'enrol_description': '1 x Apple @ 100.00',
    'name': enrol.checkout_name,
    'total': '£{:.2f}'.format(Decimal('50.00')),
} == {f.key: f.value for f in mail.mailfield_set.all()}

Maintenance

Warning

Only run the following command on a test site. It will mark all emails as sent (which you wouldn’t want on a live site)!

This will mark all emails as sent:

from django.utils import timezone
from mail.models import Mail

Mail.objects.filter(sent__isnull=True).update(sent=timezone.now())

Convert a project from Mandrill to Sparkpost

Requirements

In requirements/base.txt include the line:

sparkpost==<current sparkpost version>

e.g. at the time of writing the latest version is 1.3.0:

sparkpost==1.3.0

If mandrill was previously used remove the line:

djrill==2.1.0

In your .private file for the project include the lines:

export SPARKPOST_API_KEY='<your sparkpost api key>'

unset MANDRILL_API_KEY
unset MANDRILL_USER_NAME

If mandrill was previously included in your project remove the lines:

export MANDRILL_API_KEY='<your mandrill api key>'
export MANDRILL_USER_NAME='<your mandrill username>'

In settings/base.py include the line:

DEFAULT_FROM_EMAIL = 'user@example.com'

EMAIL_BACKEND = 'sparkpost.django.email_backend.SparkPostEmailBackend'

SPARKPOST_API_KEY = get_env_variable('SPARKPOST_API_KEY')

SPARKPOST_OPTIONS = {
  'track_opens': False,  # or True as required
  'track_clicks': False,  # or True as required
  'transactional': True,
}

If mandrill was previously included in the project remove the lines (search for EMAIL_BACKEND in the settings directory to make sure you get all occurances):

EMAIL_BACKEND = 'djrill.mail.backends.djrill.DjrillBackend'

MANDRILL_API_KEY = get_env_variable('MANDRILL_API_KEY')
MANDRILL_USER_NAME = get_env_variable('MANDRILL_USER_NAME')

In settings/base.py alter the THIRD_PARTY_APPS (included in INSTALLED_APPS) to include the sparkpost app for example:

THIRD_PARTY_APPS = (
   'easy_thumbnails',
   'reversion',
   'captcha',
   'sparkpost',
)

If mandrill was previously used in a the project remove the following line from THIRD_PARTY_APPS

'djrill',

SMTP

Google Mail:

SMTP settings to send mail from a printer, scanner, or app:

_images/mail-google-smtp-relay-service.png

SparkPost:

server        smtp.sparkpostmail.com
user          SMTP_Injection
password      <sparkpost-api-key>
port          587
security      STARTTLS