My server can burn, my services will run

5 October 2023
#self-hosting #hosting #ansible

When I was a kid, our house got burgled three times. Our most valuable belongings were taken, our safe place put upside down, and our minds were scarred. This had a deep impact on my relationship with belongings: I want as little of them as possible, and I want them to be easy to replace.

Something expensive or hard to replace is a stress-inducing liability to me. This doesn’t apply to everything or people of course: my family and the house I spent time renovating are both dear to me and hard to replace. But those principles work particularly well in two different aspects of my life: let’s see how it can stretch form my bicycle to my self-hosted setup.

A disposable bike, made to last

Cycling is an important aspect of my life. It’s a gentle form of exercise, which allows me to do it regularly and painlessly. It’s also the most efficient transportation in terms of distance travelled per body weight. It doesn’t run on fuel, which makes it both cheap to use and helps me keep Earth liveable for the current species inhabiting it (including humans).

My bicycle is a low end one: it’s cheap enough to buy, making it not interesting to steal. It’s extremely basic: it has V-Brakes, it doesn’t have any suspension, it only has a derailleur on the back wheel. The less moving parts, the sturdier. That’s also making it cheap & easy to repair.

Disposable servers, resilient services

When it comes to self-hosting, the same principles apply: I want my services to be sturdy, cheap & easy to maintain. I want very few moving parts, and I treat the hardware as disposable and unreliable. The data center hosting my servers can have issues. This is not a view of the mind or a very old event: OVH’s Strasbourg data center caught fire twice in 2021.

Since I put an emphasis on the self in self-hosting, scalability is not one of my issues beyond good old fashioned capacity planning, which means I need to have a good idea of what is happening on my server. Even if I host services exclusively for myself, I still take pleasure in doing things right. I like to set up a pre-production/testing environment that matches my production before deploying services for real. But I want that environment to be disposable, since I don’t want to burn resources and money on it when I don’t need to roll out new software.

I also want to be able to move services between servers, and I want restoring from backups to be as simple and documented as reasonably possible in my spare time.

Servers recipes with Ansible

Ansible relies on the concept of playbook: a recipe you write to perform actions on one or several servers. Ansible runs on your laptop or desktop, reads how to deploy things, where to deploy them, and relies on ssh (by default) to perform them on your behalf. It also reads host variables (secrets) to allow your playbooks to be generic enough and be stored in a standard git repo, all while not leaking anything sensitive.

In Ansible’s jargon, it is common to group the various steps or tasks of “how to deploy a specific service” in a role. Those roles can be imported into the recipe, which is called a playbook. The list of hosts against which services can be deployed is the inventory. And finally the secrets can be configured in host variables. If you want to encrypt those, you will need an ansible vault.

You might think “but this could be a bash script”, and more often than not it could. But then you’d miss out on collections: specialised modules to perform very specific actions, reviewed and polished by hundreds of others.

Ansible is not about “doing things”, it’s about “putting things in a certain state”. You’re not instructing it to “deploy a Postgres container”, you’re telling it “there must be one Postgres container deployed”. This means the end result will be the same whether a task is called once or ten times, for most tasks.

I also exclusively run container-based services out of convenience: I have one or several containers providing the actual service, and one or several volumes attached to them to store the actual data.

All of this makes the hardware very disposable: as long as you have your playbooks, secrets and backups of the data, you can re-create the infrastructure on any host.

Reliability with podman and services

systemd is the most widespread services manager on Linux. A service manager is a guardian on your system, that will start and stop services to match its configuration. If service B needs to be started, and if it depends on service A, the service manager will start service A, wait for it to be ready, and start service B.

A service is roughly software that is started and stopped by the service manager and runs in the background. It can be any kind of software, but more particularly it can be a container.

To create a service we need to describe what software should run, what are the commands to run to make it start or stop, what it relies on to run (e.g. does it need the network to be operational), what to do if it crashes, and potentially other options.

All of this is described in a service file, which can look like the following

[Unit]
Description=Podman container-traefik.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/run/containers/storage

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=always
TimeoutStopSec=70
ExecStart=/usr/bin/podman start traefik
ExecStop=/usr/bin/podman stop  \
	-t 10 traefik
ExecStopPost=/usr/bin/podman stop  \
	-t 10 traefik
PIDFile=/run/containers/storage/overlay-containers/067f6276d6c68ee04f1185512dff640083467fae25fd534b5a3245f113dad787/userdata/conmon.pid
Type=forking

[Install]
WantedBy=default.target

We’re not going to dive into systemd internals, but when the server (re)boots, systemd is going to try to match what is described in the files. A service is not started but should be? It’s going to start it.

systemd services are a great way to make containers start at boot or reboot, whether it was intentional or not. But the services files can be a bit tedious to write.

Podman is a container engine that is a drop-in replacement for docker in most cases. It also provides a lot of fantastic built-in utilities including one to generate services files from existing containers.

Podman also supports Pods: a concept that works well to group (and isolate) containers for one service together, which is a great incentive to keep deployments clean.

There is an ansible collection to manage podman pods and containers which allows us to describe what containers should be deployed on our servers, what pods they should be running in, and to generate the service files for them.

My setup

Two servers and a bucket

I rely on two primary servers on the same site: one beefy server called VPS1 to host my services themselves, one smaller host called VPS2 to do the monitoring and alerting if anything goes wrong. Both VPS are hosted at Netcup, a cheap and reliable German provider.

All the services run in containers, and the data is persisted in volumes on the host itself. I perform daily backups, and push them to a S3-compatible bucket at Scaleway.

The first host (VPS1), dedicated to services, runs:

  • Traefik, a reverse-proxy automatically fetching certificates for the domains configured for it
  • Nextcloud to store my files, its MariaDB plus a Redis instance and a Cron launcher
  • Synapse and its Postgres
  • Hedgedoc and its Postgres
  • Authentik and its Postgres plus a Worker
  • Shell scripts stopping the containers, using Kopia to dump the content of the volumes into a S3-compatible bucket at Scaleway, and ping healthchecks.io on finish so I’m notified if the backup didn’t happen.

The second host (VPS2), dedicated to monitoring, is connected to VPS1 using wireguard and runs:

  • Prometheus to collect data from the primary host (VPS1) itself and the services running on it
  • Grafana to plot the data
  • Alertmanager (part of Prometheus) to warn me if there’s anything I should be concerned about

While VPS1 is open on the internet so I can access my services freely from anywhere, VPS2 can only be reached out to through a wireguard VPN. It can seem counter-intuitive that the server containing the less sensitive data is the most locked down, but there’s a reason for it.

If I could, I would protect both servers behind a VPN, but I can’t. VPS1 is indeed running two services that need to be accessible from the public internet:

  • My Matrix instance Synapse needs to be reachable publicly so it can federate properly with the other homeservers of the public federation
  • My Nextcloud instance needs to be reachable publicly so I can share files or folders with friends, family or even businesses without requiring them to set-up a VPN.

Preserving the playbook, the secrets, and the data

I don’t want my laptop to become a single point of failure. I store my playbooks on GitHub in a private repo, with the secrets encrypted via ansible-vault. I would not necessarily recommend it to everyone, but at my scale and given my threat model this is perfectly acceptable: the secrets are still encrypted, and it’s more likely that I lose my laptop than that a GitHub employee or intelligence agency try to brute force the secrets from that private repository.

The password of the ansible vault is stored in my personal password manager. I have the recovery key of that password manager stored on paper in a physical vault in a safe place.

Finally as said earlier, the data is backed up off-site in a S3 bucket at Scaleway in case anything goes terrible wrong at netcup.

Staying up to date

The two critical aspects to staying up to date for me are:

  1. Being aware that there is an update so I can check the changelog and upgrade notes
  2. Making the upgrade seamless

For the first point, the easiest route for me has been RSS. Most of the software I rely on either is developed on GitHub or is mirrored there. The releases section of a GitHub repo has RSS autodiscovery, which means you can copy the link to your RSS reader and it will figure out how to get the atom feed from it.

For example, to follow Synapse releases, I can paste https://github.com/matrix-org/synapse/releases into my reader, which will figure out that the atom feed is at https://github.com/matrix-org/synapse/releases.atom.

As for the upgrade process, I never use the latest tag for the containers powering my servies. I instead use a variable for the version number. I rely on the recommended structure for ansible playbooks and more particularly I create a roles/myservice/defaults/main.yml file for each role, in which I set the version number of the container I want to use. I could use roles/myservice/vars/main.yml as well, but I like the fact that the defaults folder is low priority and can be overriden by host variables if needed.

This can be used in addition to podman’s auto-update feature to update the minor versions, assuming the containers provider has a sane tagging scheme. Podman indeed has an auto updater service you can enable. It runs every day at midnight by default and checks if there’s a new image for the tag you’re following. If there’s an image, it will pull it and update your container, and rollback if there are issues on restart. For example with Nextcloud, you set the tag you follow to 27-apache, which is the major version. Whenever a minor version is released, podman is going to attempt to update it (e.g. bump from 27.0.0-apache to 27.0.1-apache).

Most of the time, an upgrade is as simple as changing the tag number in roles/myservice/default/main.yml and running ansible-playbook -i staging myplaybook.yml to test, and ansible-playbook -i production myplaybook.yml to deploy in production.

Room for improvement

This setup allows me to self-host services with a relative peace of mind. I’m very aware of the fragile nature of the hardware my services run on, and this playbook allows me to have decent Recovery Time and Recovery Point Objectives.

There is still room for improvement. My services need to stop running during the time of the backups. Those backups are also very efficient in case of incident, but very fragile in case of attack. Finally, while this deployment’s security is reasonable but can still be improved.

No downtime backups

I need to take services offline during the time of the backup. This is a very naive approach that works decently in a self-hosted environment but that is not acceptable at scale. It can be improved with LVM snapshots to avoid downtime entirely. I will set this up and describe it in a future blog post.

More resilient backups

The backups happen in push mode: VPS1 is using the kopia client to push the data to the S3 bucket. Because of the de-duplication abilities of Kopia, it needs to be able to delete data on the bucket. This means that a malicious person who manages to compromise VPS1 could erase the backups too.

S3 compatible storage has a feature called Object Locks. It’s a feature that allows an object to be written but not deleted for a given amount of time. The client software has to understand what this means. Fortunately for me, kopia supports Object and has generally good awareness of the ransomeware protection needs.

Rootless containers

Right now containers run in rootful mode. Podman supports running containers in rootless mode. While this can improve the security in case of container escape, it also means traefik’s conveniences via the docker socket would no longer be available.

Caddy is a good reverse proxy, and famous for being easy to configure. Running rootless containers can come with its own lot of tradeoffs and fragile manual configuration between the reverse proxy and the other containers. It doesn’t necessarily mean that it’s more secure all things considered.