We are Using for backups.


To enable backups for a server, add the following to the pillar config for your server e.g:

# top.sls
  - global.gpg

To disable backups for a site, add the following to the pillar e.g:

# sites/my.sls
    profile: django
    backup: False


Backup is auto-enabled, so you don’t need to add backup: True to enable backups.


Initialise the repository using fabric:

fab init_backup:backup
fab init_backup:files

Older Notes


The process below can easily be adapted for use with any linux based storage system because we use Duplicity, ssh keys and gpg for encryption.

Strategy (draft)

Using the following commands:


We will run one backup which does weekly full, daily incremental, deleting full backups over 4 weeks, and incremental over 7 days.

We will run another (separate) monthly full backup which is deleted after 3 months.

For database backups:

# put the database backups into a 'backup' folder on the cloud server e.g.

# cron task will remove the previous days backups after making todays e.g.
rm /home/web/repo/backup/
rm /home/web/repo/backup/

# duplicity will back this up to
duplicity full --encrypt-key="ABCD0001" \
  scp:// \
duplicity full --encrypt-key="ABCD0001" \
  scp:// \

# duplicity will verify the backup
duplicity verify --encrypt-key="ABCD0001" \
  scp:// \
duplicity verify --encrypt-key="ABCD0001" \
  scp:// \

What can we do?

Delete all backups older than x days (weeks or months)

Remove all backups older than count full backups

Remove all incrementals older than count full backups

F   1st Sept
F   15th Sept
F   1st Oct

Getting Started

Install Duplicity 0.6


We are using Duplicity 0.6 because the latest version of Duplicity on Ubuntu 16.04 doesn’t work nicely with the Duplicity on our 14.04 servers. When our servers are using 16.04, then we can probably go back to installing the standard distro version i.e. sudo apt-get install duplicity python-paramiko

You will receive your account details from Please refer to the Checklist and fill in your own details.


Create an SSH key on your laptop:

ssh-keygen -o -a 100 -t ed25519


Do not enter a password here.


The old command line was ssh-keygen -t rsa. We updated on the advice of this article, Upgrade your SSH keys!

Upload your key to the server:

cat ~/.ssh/ | ssh 'dd of=.ssh/authorized_keys oflag=append conv=notrunc'

If you are setting up the server for the first time ever:


Do not run the following command unless you are the first person (or computer) to use this server. If you upload another key using this command, they will overwrite the first key.

scp ~/.ssh/

Test your ssh login to the server:

ssh ls

These instructions are copied from Generating SSH Keys for Automated Backups (up to and including Testing Your Passwordless Login).


To encrypt the backups we need a gpg key. This key will be shared with all the web servers and with any laptops which need to decrypt (and restore) the data.

To create the gpg key:

gpg --gen-key

# defaults...
Please select what kind of key you want:
(1) RSA and RSA (default)
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Please specify how long the key should be valid.
0 = key does not expire
You need a Passphrase to protect your secret key.

Accept the defaults (as above) and enter a passphrase for your gpg key.

List the keys, and make a note of the key number (in this example, the key is ABCD1234):

gpg --list-keys
# --------------------------------
# pub   2048R/ABCD1234 2014-10-30

Export the public and private keys and add them to your pillar:

cd ~/repo/dev/module/deploy/pillar/
gpg --armor --export ABCD1234 >> global/gpg.sls
gpg --armor --export-secret-key ABCD1234 >> global/gpg.sls

Edit the global/gpg.sls file so it is in the following format e.g:

    user: 123
    key: ABCD1234
    public: |
      Version: GnuPG v1

      -----END PGP PUBLIC KEY BLOCK-----
    private: |
      Version: GnuPG v1



Multiline strings in YAML files are started with the | character and are indented two characters.

Cloud Server

Log into the Salt master and update your cloud server. Salt will do the following tasks:

  • create an ssh key
  • copy the GPG keys to the ~/repo/temp/ folder.
  • create a backup script for each site on the server
  • create a cron script for each site on the server

Add the ssh keys to the server:

ssh server
sudo -i -u web
cat ~/.ssh/ | ssh \
  'dd of=.ssh/authorized_keys oflag=append conv=notrunc'
# enter your password

Check that you can connect to the server without a password:

ssh ls -la

Import the GPG keys:

ssh server
sudo -i -u web
gpg --import ~/repo/temp/pub.gpg
gpg --allow-secret-key-import --import ~/repo/temp/sec.gpg

List the keys, and then mark the key as trusted:

gpg --list-keys
gpg --edit-key ABCD1234
> trust
# Select option 5 = I trust ultimately
> q

Do an initial full backup. The Salt states will create a backup script in the /home/web/opt/ folder e.g:: /home/web/opt/

To initialise the backup run the script with the full argument e.g:

/home/web/opt/ full





To install an earlier version of Duplicity, see Install Duplicity 0.6

To list the files on

# database backup (and any files in the backup folder)
ssh ls -la
# files backup
ssh ls -la

To list backup dates:

duplicity collection-status ssh://

To list the backups:

duplicity list-current-files ssh://
duplicity list-current-files ssh://

Duplicity makes restoring easy. You can restore by simply reversing the remote and local parameters.


You will probably see Operation not permitted errors. This is Duplicity attempting to restore owner and group permissions on the files.

To restore a folder:

PASSPHRASE="gpg-password" \
  duplicity \
  --file-to-restore \
  "path/to/folder/" \
  ssh:// \


When restoring a folder, /path/to/restore/folder/ must not exist. It will be created by Duplicity.

To restore a single file (in this example we are restoring from a Dropbox backup):

PASSPHRASE="gpg-password" \
  duplicity \
  --file-to-restore \
  "Dropbox/Contact/Cycle Policy.docx" \
  ssh:// \
  "/path/to/restore/Cycle Policy.docx"


When restoring a single file, /path/to/restore/policy.docx is the file name NOT the folder name.

To restore by date or time:

To restore a full set of files from 2 days ago (note you can omit the restore):

duplicity restore -t 2D ssh://

To restore a full set of files from a specific time (note you can omit the restore):

The --time format is YYYYMMDDTHHMMSSZ and Duplicity will pull the restore from the next backup older than the time entered e.g. collection-status shows a backup dated 20141125T112710Z and one dated 20141125T122710Z enter --time as 20141125T113000Z and retrieve data from 20141125T112710Z:

duplicity restore --time 20141125T113000Z \
  ssh:// \


Restoring from back to any location other than the original location will result in an Error '[Errno 1] Operation not permitted: prefix to each restored file although the files will restore and be available… this is a known bug with Duplicity to do with permissions. See: Why do I get an ‘Operation not permitted’


To check the usage on your account, follow the Usage instructions…

Install Duplicity 0.6

Here are some instructions for installing 0.6 version of duplicity. I’ve installed it in /opt/duplicity so that it is not overwritten by a new version when we upgrade ubuntu.

Remove the installed version of duplicity:

apt remove duplicity

Download the latest version of the 0.6 tarball from:

Create a directory for Duplicity (you will need to sudo for this):

# sudo
mkdir /opt/duplicity

Extract the archive to a temporary folder.

Change into the folder and run:

# sudo
python install --prefix=/opt/duplicity

You might need to:

sudo apt install librsync-dev python-dev python-lockfile python-paramiko

Create a script (in your home folder) to run duplicity as follows:

# vim ~/bin/duplicity

# exit immediately if a command exits with a nonzero exit status.
set -e
# treat unset variables as an error when substituting.
set -u
export PYTHONPATH='/opt/duplicity/lib/python2.7/site-packages/'
/opt/duplicity/bin/duplicity $@

Make it executable:

chmod a+x ~/bin/duplicity

Test it using:

~/bin/duplicity --version

Then uninstall the distro version:

sudo apt-get remove duplicity