¶ Duplicity Backup with Dropbox
This guide can be used to setup Duplicity backups that are encrypted using GPG and uploaded to Dropbox.
Recently, Dropbox made a change to the access tokens which breaks Duplicity (the tokens are only valid for 4 hours). Due to this, rclone will be used to upload the backups to Dropbox.
¶ Initial Setup
The initial setup is in three parts - the Dropbox setup, the rclone setup and the general server setup.
For Debian servers, the following packages need to be installed:
apt -y install \
duplicity rclone
¶ Dropbox Setup
These steps assume that an application scoped API token will be used (rather than providing access to entire Dropbox account).
Log into Dropbox and from the Developer Tools section create a new application. Fill out the page with these details:
On the next page, select the Permissions heading at the top. The following permissions need to be defined:
account_info.readsharing.writesharing.readfiles.metadata.writefiles.metadata.readfiles.content.writefiles.content.read
Make sure these permissions are defined before continuing.
Back in the settings tab, add the OAuth 2 redirect URI http://localhost:53682/:
Record the "App Key" and "App Secret" values - they will be needed for the rclone setup.
¶ rclone Setup
Run the rclone configuration command to add a new backend for Dropbox:
rclone config
I will be naming the backend for rclone dropbox. After entering the type of storage, you will be prompted for the client_id and client_secret. This should be the "App Key" and "App Secert" values from Dropbox.
rclone will then prompt you to check if you would like to auto configure. As this is a headless remote server, I select n and follow the steps to authorize the application remotely.
rclone should then confirm the remote backend has been configured.
¶ Server Setup
On the server that will be running the backups generate a GPG key. The GPG key will be used to encrypt the backups. Run the following command to generate the key:
gpg --full-gen-key --expert
During the key generation, select the following options:
- Please select what kind of key you want:
9 - Please select which elliptic curve you want:
1 - Please specify how long the key should be valid:
0
You will then be prompted for the user details for the key:
- Real name:
<server hostname> - Email address: (can be left blank)
- Comment:
<server hostname> - Duplicity Backups
Press the O key to then confirm the key generation:
When prompted, define a secure password for the private key and record it.
It is critical that a backup of the GPG (private) key is stored in a secure location. If you lose the private key you will not be able to restore any of the backups. Also ensure you do not lose the password for the private key, otherwise it will be useless.
To backup the key it can be exported in ASCII format which makes it easy to copy (in this case the key ID that was generated is B0D763E0A6D5069DC60F0D99937AF361A9647ACA):
gpg --output privkey.txt --armor --export-secret-key B0D763E0A6D5069DC60F0D99937AF361A9647ACA
The GPG and Dropbox configuration can now be placed into a file; the file will be sourced by the backup script to ensure the passwords are not passed as a command line argument. The following variables need to be set:
GPG_KEY: The GPG key ID used for encryptionPASSPHRASE: The password for the GPG private key
The file must be protected due to the token and passphrase being present. Since the backups in this case will be ran by the root user, I will use the file name /root/.duplicity_env with 0400 permissions:
cat << EOF > /root/.duplicity_env
## GPG Config
export GPG_KEY="B0D763E0A6D5069DC60F0D99937AF361A9647ACA"
export PASSPHRASE="MY LONG GPG PASSPHRASE"
EOF
chmod 0400 /root/.duplicity_env
The basic server setup is now complete and the backup script can be written.
¶ Backup Script
A script needs to be created which will actually run the backup. For my case, I would like to backup both Docker volumes as well as various system configuration, how ever I want them to backup to separate folders in Dropbox.
- Docker volumes (
/var/lib/docker/XXXX) will be backed up into the Dropbox folder called "Docker" - System configuration will be backed up into the Dropbox folder called "System"
The backup script looks like this (I have created it in /opt/backup.sh):
#!/bin/bash
## Terminate if any command fails
set -e
## Source variables
source /root/.duplicity_env
## Remove old backups for Docker volumes
duplicity \
remove-older-than 360D \
--force \
rclone://dropbox:/Docker
## Backup Docker volumes
duplicity \
--verbosity info \
--encrypt-sign-key="$GPG_KEY" \
--full-if-older-than 90D \
--log-file "/var/log/duplicity.log" \
--asynchronous-upload \
--include '/var/lib/docker/volumes/volume1' \
--include '/var/lib/docker/volumes/volume2' \
--include '/var/lib/docker/volumes/volume3' \
--include '/var/lib/docker/volumes/volume4' \
--include '/var/lib/docker/volumes/volume5' \
--include '/var/lib/docker/volumes/volume6' \
--include '/var/lib/docker/volumes/volume7' \
--exclude '**' \
/ \
rclone://dropbox:/Docker
## Cleanup Docker volume backups
duplicity \
cleanup \
--force \
rclone://dropbox:/Docker
## Remove old backups for system files
duplicity \
remove-older-than 900D \
--force \
rclone://dropbox:/System
## Backup system files
duplicity \
--verbosity info \
--encrypt-sign-key="$GPG_KEY" \
--full-if-older-than 14D \
--log-file "/var/log/duplicity.log" \
--asynchronous-upload \
--include '/etc/iptables' \
--include '/etc/docker' \
--include '/opt/backup.sh' \
--include '/etc/systemd/system/duplicity.service' \
--include '/etc/systemd/system/duplicity.timer' \
--exclude '**' \
/ \
rclone://dropbox:/System
## Cleanup system backups
duplicity \
cleanup \
--force \
rclone://dropbox:/System
## Unset variables for safety
unset GPG_KEY
unset PASSPHRASE
Make sure the backup script is executable:
chmod +x /opt/backup.sh
To ensure the backup is working, execute the script. Review the Duplicity log after it has completed to make sure it looks as expected.
¶ Scheduling
I use systemd timers/services to schedule tasks rather than cron.
First create the timer file /etc/systemd/system/duplicity.timer:
cat << EOF > /etc/systemd/system/duplicity.timer
[Unit]
Description=Call the Duplicity backup script each day
RefuseManualStart=no
RefuseManualStop=no
[Timer]
OnCalendar=daily
Persistent=true
RandomizedDelaySec=3600
Unit=duplicity.service
[Install]
WantedBy=timers.target
EOF
In this case the backup will run each day with a randomized delay.
Create the service file that the timer file calls (/etc/systemd/system/duplicity.service):
cat << EOF > /etc/systemd/system/duplicity.service
[Unit]
Description=Execute the Duplicity backup script
[Service]
Type=simple
ExecStart=/opt/backup.sh
TimeoutSec=3600
TimeoutStartSec=3600
TimeoutStopSec=3600
StandardOutput=null
StandardError=null
[Install]
WantedBy=default.target
EOF
Enable and start the timer:
systemctl enable duplicity.timer
systemctl start duplicity.timer