Monitored Encrypted Backups on a Raspberrypi with a Sleepy Disk

22 February 2022
#matrix #raspberrypi #self-hosting #backup

I self-host a few services. It’s easy to put services online, but self-hosting properly in the long run is difficult. A part of self-hosting properly is having backups and monitoring. In this article I’m going to show you how I make sure to get off-site backups, with a Raspberry Pi at home pulling data from my VPS. There’s room for improvement, and if you have constructive comments on how I can do better I’d be happy to hear them!

The Threat

The threat I’m working against is data loss. It can happen because my VPS gets hacked, in which case I want to destroy the VPS to prevent it from bein exploited further without having to think twice. If my data is leaked, I don’t want the additional burden of not having access to it myself

It can also happen because the datacenter my server lives in catches fire (it happened for real, to a large French provider), in which case I want to be able to set up a new VPS somewhere else

Given the raspberry pi is at my home and burglaries are not uncommon here, I want to make sure my data won’t be in the wild, so I encrypt the disk on which my backups are stored.

The Setup

I got myself a Western Digital WD Elements of 4TB, as I wanted a large disk that can be powered by the Raspberry Pi only. The less cables, the better!

I wanted to be able to plug my backup drive and to plug a “decryption usb key” to decrypt the backup drive without having to type a password. This way the Raspberry Pi can run in headless mode, without ever being attached to a keyboard and screen. Long Steve’s USB Hard Drive Encryption on a Raspberry Pi guide is a nice resource to set-up an encrypted disk drive and make it decryptable with a key file instead of a password. My set-up is heavily inspired from this guide, plus a few things to make it more reliable

The main steps are:

  • preparing the encrypted disk and decryption usb key
  • making sure it’s mounted before starting the backups (more on why later)
  • getting the backups done daily
  • getting paged when things get wrong.

Just Do It

Prepare the Disk

First, plug the disk and run dmesg | tail. You should see something like the following.

root@raspberrypi:/home/pi# dmesg | tail
[1755509.305600] usb 2-1: New USB device found, idVendor=1058, idProduct=2620, bcdDevice=10.18
[1755509.305622] usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[1755509.305641] usb 2-1: Product: Elements 2620
[1755509.305659] usb 2-1: Manufacturer: Western Digital
[1755509.305678] usb 2-1: SerialNumber: redacted
[1755509.308926] usb-storage 2-1:1.0: USB Mass Storage device detected
[1755509.309829] scsi host0: usb-storage 2-1:1.0
[1755510.314584] scsi 0:0:0:0: Direct-Access     WD       Elements 2620    1018 PQ: 0 ANSI: 6
[1755510.315368] sd 0:0:0:0: Attached scsi generic sg0 type 0
[1755510.319313] sd 0:0:0:0: [sda] Spinning up disk...

It looks like our disk is called sda here. Let’s run lsblk to see what is the name of the device. You should have something similar to:

root@raspberrypi:/home/pi# lsblk
NAME                                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                                   8:0    0  3.6T  0 disk 
`-sda1                                8:1    0  3.6T  0 part

If we have more than one partition, we then want to partition the disk to one large partition of ext4, using fdisk /dev/sda

Let’s then format the partition:

root@raspberrypi:/home/pi# mkfs.ext4 /dev/sda1

Encrypt the disk with cryptsetup:

root@raspberrypi:/home/pi# cryptsetup --verify-passphrase luksFormat /dev/sda1 -c aes -s 256 -h sha256

Run ls -alh /dev/disks/by-uuid/ to retrieve the UUID of your disk. Unique identifiers (which are by definition unique) are more reliable than the device file (e.g. sda) udev allocated to it, which can change depending on when it was plugged, and on which USB port.

Open the encrypted partition and map it to /dev/mapper/securebackup:

root@raspberrypi:/home/pi# cryptsetup open /dev/disk/by-uuid/{your-backup-disk-uuid} securebackup

Format the encrypted partition:

root@raspberrypi:/home/pi# mkfs -t ext4 -m 1 /dev/mapper/securebackup

Create a mount point and mount the decrypted partition:

root@raspberrypi:/home/pi# mkdir /media/secure
root@raspberrypi:/home/pi# mount /dev/mapper/securebackup /media/secure/

Give ownership to the pi user:

root@raspberrypi:/home/pi# chown pi:pi /media/secure/

Prepare the Decryption USB Key

Here some of the steps are similar. You need to plug the key and run dmesg | tail to identify it, make it single partition, and format it with mkfs.ext4 as previously. Let’s consider the partition of the decryption key is called /dev/sdb1 in the rest of the article.

You then need to edit /etc/fstab so the key is mounted automatically at boot. Let’s create a mountpoint and find the key’s UUID:

root@raspberrypi:/home/pi# mkdir /media/key
root@raspberrypi:/home/pi# ls -alh /dev/disk/by-uuid | grep sdb1
lrwxrwxrwx 1 root root  10 Feb 14 16:17 redacted-reda-cted-reda-f78438a85f4f -> ../../sdb1

Add the following entry to /etc/fstab:

/dev/disk/by-uuid/{your-usb-key-uuid}	/media/key	ext4	defaults,rw	0	0

You can now mount the key:

root@raspberrypi:/home/pi# mount /media/key

Let’s generate a random file on it. This will be the actual decryption key file:

root@raspberrypi:/home/pi# dd if=/dev/urandom of=/media/key/file bs=1024 count=4

You can now make the file readable by root exclusively

root@raspberrypi:/home/pi# chmod 400 /media/key/file

And you can edit your /etc/fstab entry so the decryption key is mounted as a read-only device in the future (change rw to ro):

/dev/disk/by-uuid/{your-usb-key-uuid}	/media/key	ext4	defaults,rw	0	0

Lets add the key file on your decryption usb key as a way to decrypt the backup partition:

root@raspberrypi:/home/pi# cryptsetup luksAddKey /dev/disk/by-uuid/{your-backup-disk-uuid} /media/key/file

Finally add this entry to /etc/crypttab so the encrypted disk is mounted at boot, if possible:

# <target name> <source device> <key file>      <options>
securebackup    /dev/disk/by-uuid/{your-usb-key-uuid}   /media/key/file luks

Ensure The Disk Is Mounted

  • One thing I didn’t know when I bought this disk is that you can’t prevent it from sleeping even with hdparm when it hasn’t been used in a while
  • This means sometimes the disk goes to sleep, and isn’t automounted
  • When I leave the house and get my usb decryption key with me, plugging it back doesn’t automatically remount the decrypted backup disk

I wrote this short mount_backup_disk.sh script that I can call before triggering a backup. It takes one argument, which is the file to which I should append logs. It tries to mount the encrypted backup and exits in error if it can’t.

#!/bin/bash

# Is /media/secure already mounted?
findmnt "/media/secure" 
if [ $? -ne 0 ]
then
        echo "[$(date +%F\ %T)] Secure drive was not mounted." >> $1
        # Is security key present?
        findmnt "/media/key" 
        if [ $? -ne 0 ]
        then
               echo "[$(date +%F\ %T)] Security key was not mounted. Mounting" >> $1
               mount /media/key
               if [ $? -ne 0 ]
               then
                       echo "[$(date +%F\ %T)] Security key was not present. Exiting with error" >> $1
                       exit 1
               else
                       echo "[$(date +%F\ %T)] Security key mounted" >> $1
               fi
        fi
        # Open disk only if not already open
        if [ ! -e /dev/mapper/securebackup ]
        then
                echo "[$(date +%F\ %T) Disk was not decrypted, decrypting.]" >> $1
                cryptsetup open /dev/disk/by-uuid/{your-disk-uuid} securebackup -d /media/key/file >> $1 
                if [ $? -ne 0 ]
                then
                        echo "[$(date +%F\ %T)] Failed to decrypt secure drive. Exiting with error" >> $1
                        exit 1
                fi
        fi
        mount /dev/mapper/securebackup /media/secure
        if [ $? -ne 0 ]
        then
                echo "[$(date +%F\ %T)] Failed to mount from mapper. Exiting with error" >> $1
                exit 1
        else
                echo "[$(date +%F\ %T)] Secure disk successfully mounted" >> $1
        fi
else
        echo "[$(date +%F\ %T)] Secure drive already mounted, nothing to do." >> $1
        exit 0
fi

Back It Up

To make sure I’m not going to fill my poor Raspberry Pi’s SD card if the secure backup is not mounted, the actual backup script is on the encrypted partition itself. Here is the content of the backup script in charge of making a Synapse backup, in /media/secure/scripts/backup_synapse.sh:

#!/bin/bash

declare -i FAILURE=0

echo "[$(date +%F\ %T)] Stopping synapse" >> $1
ssh [email protected] "sudo docker stop infra_synapse_1" >> $1 2>&1
echo "[$(date +%F\ %T)] Synapse stopped" >> $1
rsync -az -q --delete backup[email protected]:/var/lib/docker/volumes/infra_nginx_conf /media/secure/netcup/
if [ $? -eq 0 ]
then
        echo "[$(date +%F\ %T)] Nginx backup successful" >> $1
else
        echo "[$(date +%F\ %T)] Nginx backup failed" >> $1
        FAILURE=1
fi
rsync -az -q --delete backup_a[email protected]:/var/lib/docker/volumes/infra_synapse_data /media/secure/netcup/
if [ $? -eq 0 ]
then
        echo "[$(date +%F\ %T)] Synapse backup successful" >> $1
else
        echo "[$(date +%F\ %T)] Synapse backup failed" >> $1
        FAILURE=1
fi
rsync -az -q --delete backup_agent@myobviousvps:/var/lib/docker/volumes/infra_pg_synapse_data /media/secure/netcup/
if [ $? -eq 0 ]
then
        echo "[$(date +%F\ %T)] Synapse database backup successul" >> $1
else
        echo "[$(date +%F\ %T)] Synapse database backup failed" >> $1
        FAILURE=1
fi
echo "[$(date +%F\ %T)] Procedure complete. Starting Synapse" >> $1
ssh [email protected] "sudo docker start infra_synapse_1"
echo "[$(date +%F\ %T)] Synapse started" >> $1

if [ $FAILURE -eq 0 ]
then
	curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/{some-uuid-we-will-talk-about-next}
fi

This script is called by a helper in /root/backup/backup_synapse.sh:

#!/bin/bash

LOGFILE=/var/log/backup_synapse_$(date +%F).log
/root/backup/mount_encrypted_disk.sh $LOGFILE && /media/secure/scripts/backup_synapse.sh $LOGFILE

Finally, the helper itself is called by a cron task at 3am every day in /etc/cron.d/backup_synapse:

0 3 * * * root /root/backup/backup_synapse.sh

Page Me If It Doesn’t Work

healthchecks.io is a nice open source service that allows you to monitor up to 20 jobs for free. You can declare on the website when a job should run, how long it has to run before the alarm is rung, and how you want to be notified if it hasn’t run.

For each job declared, they give you a URL containing a UUID that your job needs to call to notify it ran correctly using curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/{the-job-uuid}

Even better: healthchecks.io has a Matrix integration! This means I can set-up a room, invite the healthchecks.io bot, turn on noisy notifications in that room, and whenever a backups isn’t done properly I’ll be notified on my phone!

Room For Improvement

Log Rotation

One of the most simple improvements here is log rotation: I add a log file per job per day to my /var/log directory… but never purge them. Eventually this is going to fill up the SD card if I don’t tidy things up regularly.

This can be done fairly simply either from the backup scripts themselves, or more cleanly by a dedicated cron job to clean-up log files older than n days.

Using Borg for Versioned Backups

The rsync used to backup docker volumes is pretty vulnerable as is: if the last backup is botched, then you effectively have no backup. Borg backup is a nice utility that can version backups, prune old versions, and make sure you at least have one usable version. How to use it in complement to this set-up is something I’ll probably cover in a follow-up article.

Borg supports using distant repositories over ssh in push mode (i.e. the server on which the data lives is pushing the data to the distant backup host). There is also a hack to make it work in pull mode, but that relies on chroot which is not possible from my Raspberry Pi (ARM) to my production VPS (x86_64).

Pulling Backups or Pushing Backups?

The borg backup way of working also opens the question: which is best between a push or pull backups architecture anyway? I don’t have a general answer to this, but here are the key ideas behind why I decided to go for a pull architecture:

  • My home network needs to be kept safe. This is where we have all our devices, and even if most of the trafic is encrypted, I wouldn’t be comfortable with someone eavesdropping on what requests we’re sending.
  • If my server is compromised, I’m already screwed. I’ve lost it, it needs to be burned to the ground and I need to start over entirely. I don’t want my compromised server to be able to compromise my Raspberry Pi at home too, which could in turn compromise other machines on the netwrok.

Fryberry Pi

The Raspberry Pi is well known for frying SD cards. Or SD cards are well known not to be reliable disks. Whoever the principal culprit is, the end result is that we should expect our SD card to fry anytime when used in a Raspberry Pi. Getting a Raspberry Pi to do daily backups is fine, but what if the Raspberry Pi itself becomes unusable? Then our RPO skyrockets, which is not good.

To get a Raspberry Pi back in functionning state, two solutions are possible:

  • the inelegant but most efficient (by far) one: making an image of the SD card to be able to flash its content to a brand new in case the current one fries
  • the overengineered fun one: writing Ansible playbooks to be able to reinstall and configure everything on a brand new image

Snapshots

Finally, this is a very naive approach to backups, doing it at the application/container volume level. To get proper backups I need to stop the application (in my example: Synapse) during the time I perform the backup. This means the application is not available for the whole duration of the backup.

While this is okay for a small scale deployment, as I have a few minutes of downtime at 3am, it doesn’t scale really well when your services grow. It also means I only get a single backup a day because I can’t afford to get downtime during the day, when I rely on those services.

Moving to a snapshot-based backup strategy would allow me to have no downtime, and to perform several snapshots and backups per day (e.g. hourly ones). In other words: if I lost my server, I would only lose the new files/activity in the last hour, instead of everything since 3am.

I’ll try to set-up this strategy in the next few week/months depending on the free time I have, and wrap it all up in a blog post!