Once upon a time, I was sitting in front of my laptop with an IDE where hundreds of different Kubernetes manifests were opened and I was thinking: “how come I am in this moment? I need to add a new service, but to do it, I have to keep in my mind zillions of details and configure dozens of files instead promised by marketing ’little changes’”.

It wasn’t my first frustrating time: I have always been thinking that the current practice of how to set up an environment is slightly more complicated than needed for micro/small businesses especially for pet projects. Yes, even in pet-projects we want to have automatic TLS, backups, databases, security, and other useful stuff. The system should be scalable. While those reasons sound like the truth, it also feels like the IT road turned somewhere wrong.

My first attempt to escape from the overloaded, bulky, heavy setups was the trusted-cgi project, where I followed the idea, that we shall not pay for what we are not using. It is a working approach but has some limits such as not very flexible deployment and complete denial of modern and industry standards of shipping in containers (though supported by manifest with docker run inside).

Disclaimer, after my first attempt I decided to invest in the local servers (self-hosting) instead of cloud hosting (except gateways), so all my next attempts were performed in less limited environments.

The second (and less public) attempt was to move environment setup complications to the deployment phase via Ansible. The idea was: “I will write this boring and complex playbook once and for all”. While this approach worked quite well for several years, I ended up with hundreds of roles, playbooks, and custom modules. The limitations of Ansible playbooks (specifically conditions and exceptions) even made me look into Fabric for a minute.

The third attempt was to use lightweight Kubernetes builds like K3S. Honestly, this approach worked surprisingly well and was stable. Problems started when I wanted to set up persistent (stateful) applications with regular backups. The standard approach here is to delegate persistent volumes and backups/restores to the provider side. However, for self-hosting it was a huge mess. The solution was daily whole VMs (I used Proxmox) backup to somewhere else. While it did solve the problem, it also caused a huge unnecessary storage overhead. The last and less understandable point of Kubernetes - it does require maintenance. A lot of articles from people much smarter and more experienced than me may have different perceptions of the right use case for the Kuberentes, but they will share the same point on self-hosting K8S: do not manage Kubernetes by yourself unless there is an absolute need for this.

Given all of this, I decided to make a fourth attempt.

# The idea

First of all, let’s split users into three different groups:

solo/micro/small business

• few (<7) number of nodes, all nodes may have very different specs and networks
• downtime is acceptable for a reasonable period (no need for HA)
• data can fit one server (no sharding)
• data loss between regular backups is acceptable

I would like to add one point: for the sake of cost optimization, it should be easy to move the environment from one node to another.

The gap between big and small businesses is so big, so my original attempts were based on a fundamentally wrong assumption that there is only one solution that fits all cases.

In reality, it’s barely possible to use enterprise solutions in medium companies or solo projects, because resource (humans, money) overhead will make no sense. At least two (let’s not forget about bus-factor) full-time, highly qualified engineers with expertise in Cloud Computing probably will consume all profit. From a DevOps perspective, Ansible/bash is king here.

A bit more interesting situation appeared in medium businesses: there are resources to maintain their infrastructure, but it could be too costly for managed services. In DevOps, Ansible/Swarm/light-weight Kubernetes setups rule here.

For huge enterprises, head cost plays less role, but maintaining their infra pool could be so complicated, so it usually makes sense to keep a reasonably small internal infrastructure team and outsource most of the headache to managed solutions. Cloud Kubernetes/Terraform dominates here.

These companies split into these groups based on my personal experience; however, don’t think about it as a strict rule. There are big corporations that are completely denying Kubernetes/Dockers (with a good reason though), as well as, I know micro companies built on top of GKS. The mileage may vary, but there is a correlation.

## Portable stack

For my next attempt, I decided to focus on my case, which is solo/micro.

What I need is some kind of “portable stack”: a bundle with services and data, which can be relatively easily moved between servers. One day, you are using the free tier in AWS, the next day GKE, on Friday Oracle Cloud, and finally, settle in the home lab.

• For computing, we can use docker together with docker compose. Containers can give us the required portability and low overhead.
• For data, as soon as we agree with data loss between backups, we can use any cloud non-structured storage system (ie S3). However, the backup MUST be client-side encrypted, with notifications in case of failure. I found restic a very good option for this.

We can define a “universal” Ansible playbook for the deployments.

The sequence during the deployment on the new node should be:

• install the docker and docker-compose plugin
• upload compose file
• restore the backup if a backup exists and restore required
• start compose
• periodically backup data

services:
  app: # any stateful application
    image: "postgres:14"
    environment:
      POSTGRES_PASSWORD: postgres
    volumes:
    - app-data:/var/lib/postgresql/data

  restic:
    image: restic/restic:0.15.0
    restart: "no"
    command:
    - backup
    - /backup
    environment:
      RESTIC_REPOSITORY: "s3:https://${S3_ENDPOINT}/${S3_BUCKET}/${S3_PREFIX}"
      RESTIC_PASSWORD: "${S3_ENCRYPTION_KEY}"
      AWS_ACCESS_KEY_ID: "${S3_KEY_ID}"
      AWS_SECRET_ACCESS_KEY: "${S3_KEY_SECRET}"
      AWS_DEFAULT_REGION: us-west-000
    volumes:
      - app-data:/backup/app-data
volumes:
  app-data: {}

- hosts: all
  vars:
    SRCDIR: deployment
    WORKDIR: /data
    S3_ENDPOINT: s3.us-west-000.backblazeb2.com
    S3_BUCKET: "..."
    S3_PREFIX: "..."
    S3_ENCRYPTION_KEY: "..."
    S3_KEY_ID: "..."
    S3_KEY_SECRET: "..."
  tasks:
  # install docker
  - name: Install required system packages
    apt: 
      update_cache: yes
      state: present
      name: 
        - 'apt-transport-https'
        - 'ca-certificates'
        - 'curl'
        - python3-docker

  - name: Add Docker GPG apt Key
    apt_key:
      url: https://download.docker.com/linux/ubuntu/gpg
      state: present

  - name: Add Docker Repository
    apt_repository:
      repo: "deb https://download.docker.com/linux/ubuntu {{ansible_distribution_release}} stable"
      state: present

  - name: Update apt and install docker-ce
    apt:
      name: 
      - docker-ce
      - docker-compose-plugin
      update_cache: yes
      state: latest

  # copy content
  - name: Create work dir
    file:
      path: "{{WORKDIR}}"
      state: directory

  - name: Copy content
    synchronize:
      src: "{{SRCDIR}}"
      dest: "{{WORKDIR}}/"

  # restore if possible and needed
  - name: Stop docker-compose
    command: docker compose stop
    args:
      chdir: "{{WORKDIR}}"
   
  - name: Init repo
    command: docker compose run --rm restic init
    args:
        chdir: "{{WORKDIR}}"
    environment: 
        S3_ENDPOINT: "{{S3_ENDPOINT}}"
        S3_BUCKET: "{{S3_BUCKET}}"
        S3_KEY_ID: "{{S3_KEY_ID}}"
        S3_KEY_SECRET: "{{S3_KEY_SECRET}}"
        S3_PREFIX: "{{S3_PREFIX}}"
        S3_ENCRYPTION_KEY: "{{S3_ENCRYPTION_KEY}}"
    register: init_backup
    failed_when: "init_backup.rc !=0 and 'already initialized' not in init_backup.stderr"

  - name: Check if restored
    stat:
        path: "{{WORKDIR}}/.restored"
    register: restored_backup

  - name: Restore
    command: docker compose run --rm restic restore latest --target /
    args:
        chdir: "{{WORKDIR}}"
    environment: 
        S3_ENDPOINT: "{{S3_ENDPOINT}}"
        S3_BUCKET: "{{S3_BUCKET}}"
        S3_KEY_ID: "{{S3_KEY_ID}}"
        S3_KEY_SECRET: "{{S3_KEY_SECRET}}"
        S3_PREFIX: "{{S3_PREFIX}}"
        S3_ENCRYPTION_KEY: "{{S3_ENCRYPTION_KEY}}"
    register: restore_backup
    when: init_backup.rc !=0 and not restored_backup.stat.exists
    failed_when: restore_backup.rc !=0 and 'no snapshot found' not in restore_backup.stderr

  - name: Mark as restored
    file:
        path: "{{WORKDIR}}/.restored"
        state: touch

  # start everything
  - name: Start docker-compose
    command: docker compose up --remove-orphans -d
    args:
      chdir: "{{WORKDIR}}"
    environment: 
      S3_ENDPOINT: "{{S3_ENDPOINT}}"
      S3_BUCKET: "{{S3_BUCKET}}"
      S3_KEY_ID: "{{S3_KEY_ID}}"
      S3_KEY_SECRET: "{{S3_KEY_SECRET}}"
      S3_PREFIX: "{{S3_PREFIX}}"
      S3_ENCRYPTION_KEY: "{{S3_ENCRYPTION_KEY}}"

For those who are not familiar with Ansible let me explain what’s going on here:

1. install docker from the official repositories
2. stop all running services
3. initialize backup repository and ignore an error if the repository already initialized
4. check if this node has already restored
5. if not restored yet, restore the backup and ignore an error if there are no backups yet
6. mark the node as restored
7. start all services

It means that regardless of how many times we will run this playbook, we will have consistent results. Also the migration to another node (even fresh) means just changing the host address in the inventory and all data will be automatically migrated through backup.

However, the problem is that restic will backup everything only once during the launch, and we probably want to have backups on a regualr basis. It also will be nice to have some kind of notifications after each backup.

The first thing that may come to mind is to use cron and invoke docker compose run restic each time. Unfortunatley, it means that configuration becomes scattered and harder to maintain.

To narrow down the problem, we want something like docker/docker-compose native scheduler. I found a very nice project Ofelia which can do exactly what we need, but unfortunately it can not respect compose namespaces and therefore it could cause a conflict between multiple docker-compose installations.

As an alternative, I created a compose-scheduler

which basically can run services or commands in services within the same compose file with HTTP notifications.

The main features of the service are zero-configuration by-default, designed for docker compose (auto-detect, respects namespace), and with HTTP notifications.

Let’s rewrite our compose file and add regular backups.

services:
  app: # any stateful application
    image: "postgres:14"
    environment:
      POSTGRES_PASSWORD: postgres
    volumes:
    - app-data:/var/lib/postgresql/data

  restic:
    image: restic/restic:0.15.0
    restart: "no"
    command:
    - backup
    - /backup
    environment:
      RESTIC_REPOSITORY: "s3:https://${S3_ENDPOINT}/${S3_BUCKET}/${S3_PREFIX}"
      RESTIC_PASSWORD: "${S3_ENCRYPTION_KEY}"
      AWS_ACCESS_KEY_ID: "${S3_KEY_ID}"
      AWS_SECRET_ACCESS_KEY: "${S3_KEY_SECRET}"
      AWS_DEFAULT_REGION: us-west-000
    volumes:
      - app-data:/backup/app-data
    labels:
    - "net.reddec.scheduler.cron=@hourly"

  scheduler:
    image: ghcr.io/reddec/compose-scheduler:1.0.0
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
volumes:
  app-data: {}

Here:

• we added scheduler service which will scan for labels in the same compose project
• we added net.reddec.scheduler.cron=@hourly label which tells the scheduler to run this service every hour (uses Cron notation from robigfig/cron )

Sounds good, right? But we also want to have notifications in case our backup fails.

As a notification reciever, I will use Workato.

final compose file with regular backup

services:
  app: # any stateful application
    image: "postgres:14"
    environment:
      POSTGRES_PASSWORD: postgres
    volumes:
    - app-data:/var/lib/postgresql/data

  restic:
    image: restic/restic:0.15.0
    restart: "no"
    command:
    - backup
    - /backup
    environment:
      RESTIC_REPOSITORY: "s3:https://${S3_ENDPOINT}/${S3_BUCKET}/${S3_PREFIX}"
      RESTIC_PASSWORD: "${S3_ENCRYPTION_KEY}"
      AWS_ACCESS_KEY_ID: "${S3_KEY_ID}"
      AWS_SECRET_ACCESS_KEY: "${S3_KEY_SECRET}"
      AWS_DEFAULT_REGION: us-west-000
    volumes:
      - app-data:/backup/app-data
    labels:
    - "net.reddec.scheduler.cron=@hourly"

  scheduler:
    image: ghcr.io/reddec/compose-scheduler:1.0.0
    restart: unless-stopped
    environment:
      NOTIFY_URL: "https://webhooks.sg.workato.com/webhooks/rest/d17cefc7-44de-42a6-bacf-e6c923c8562a/on-backup"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
volumes:
  app-data: {}

Scheduler will send notifications after each job if NOTIFY_URL env variable set. HTTP method, attempts number, and interval between attempts can be configured. Authorization via Authorization header is also supported.

Scheduler will stop retrying if at least one of the following criteria is met:

• reached maximum number of attempts
• server returned any 2xx code (ex: 200, 201, …)

# Voilà!

We have our portable stack, which we can deploy anywhere with docker support, migrate data between nodes hassle-free, that’s without Kubernetes-like overhead, and is a transparent and clear system.

Refrence repository with portable stack