Skip to content

Draft: Use S4

Justin Russell requested to merge use-s4 into main

Storage volumes have been refactored to use Simple Self-hosted Storage Solution, or S4, for short. This change is to increase the reliability of backups. In order to support using S4, the Hive Backend and Device have had various modifications.


A major struggle with our previous storage solution was how "invisible" and fragile it was. There were times where storage volumes would simply never be replicated due to some underlying issue and due to the nature of Docker plugins, debugging was downright impossible. We realized that we needed a solution that didn't rely on external services and a complicated Docker plugin. We also needed a solution that is far more simple and maintainable.

The gist of S4 is relatively simple: minimize the complexity of data replication; while maintaining reliability. While likely not as scalable as our previous solution, we believe that S4 better fits our goal of getting more people to run their own services. S4 mimics git from a CLI and repository perspective.

Setting up an S4 volume is as simple as running:

> s4 init
S4 volume successfully created at /home/mosaic/my-data
Done. You will need to re-enter this directory `cd /home/mosaic/my-data` to continue.

> cd /home/mosaic/my-data

> s4 remote add origin <remote_target>

> s4 push
Initializing remote origin
Remote origin initialized successfully
Taking new snapshot
Create a readonly snapshot of '/home/mosaic/my-data' in '.s4/snapshots/snapshot-f937e8cd-ebe1-48a9-813f-8ac5658ee7fe'
Delete subvolume (no-commit): '/home/mosaic/my-data/.s4/snapshots/snapshot-f937e8cd-ebe1-48a9-813f-8ac5658ee7fe'

Pulling down a volume

> s4 pull
Checking with remote "origin" for new snapshots...
New changes found, syncing from remote "origin"
Latest changes synced from remote "origin"

Cloning a volume

> s4 clone borg@localhost:/my-data
Mounting /dev/loop3 at /home/mosaic/my-data
Pulling latest snapshot from borg@localhost:/volumes/my-data
Latest changes synced from remote "origin"

Even better, S4 was designed not to be coupled with Fractal Mosaic. Everything the device runs is the same as what a user would run when using S4 on its own.

In order to use and create S4 volumes, an s4-agent Docker container is used. The image includes everything it needs in order to manage S4 volumes. The s4-agent image is included as a tarball inside the Device that the Device loads on startup. While this change means that the Device image is larger, storing the agent as a Docker image means that the user only ever has to pull the Device image. An added benefit to storing the agent as a tarball means that updating S4 is as simple as updating the Device.

S4 volumes are simply BTRFS formatted loop devices that are mounted as Docker volumes. The s4-agent Docker container manages creating and replicating changes to a remote by using the s4 CLI as briefly described above. For now, this remote is a cloud deployed instance that we manage but hope to quickly allow users to run their own via the Fractal App Catalog.

App Docker Compose File Structure

As for how the addition of S4 affects Fractal Mosaic, the structure of Docker Compose files for applications have been modified to manage launching replicator s4-agents. S4 replicator agents manage keeping the app's data synchronized with the remote.

Replicator agents have a Docker healthcheck that indicate when the S4 volume is ready to use. This is so that an application is not started before the app's data has been successfully pulled down into the volume. Only once the data has been fully pulled down, will the healthcheck pass, allowing the app's containers to begin starting up.

The number of S4 volumes depends on the number of volumes in the named volumes section of the Compose File that contain the external: true property. The Device upon starting an app will parse the compose file, gather all volume names that have the external property set, and create s4 volumes for them. Each name is made unique by appending a unique identifier to the name. For example: vaultwarden-vol -> vaultwarden_vol_1b92. These names are written to the app's environment file: <NAME_OF_VOLUME>_VOLUME_NAME. Each created S4 volume is also mounted on the host at a path which is typically /var/lib/fractal/mnt/<volume_name>. Doing so allows for subvolumes to be created. Subvolumes allow for the number of replicator S4 agents to be minimized. Subvolumes are simply host bind mounts that are inside the S4 Volume.

Below is the Vaultwarden Docker Compose File to show an example:

      EXPOSE: vaultwarden:80
      # subvolume of `vaultwarden-vol` that is mounted on the host.
      # stores only the data for the link container
      - ${FRACTAL_S4_MOUNT_PATH}/${VAULTWARDEN_VOL_VOLUME_NAME}/link-data:/root/.local/share/caddy
      # ensure that container is not started until the vaultwarden-s4-agent container starts
        condition: service_healthy

    image: vaultwarden/server:latest
      # subvolume of `vaultwarden-vol` that is mounted on the host.
      # stores only the data for the vaultwarden container
      test: curl localhost:80
      interval: 10s
      timeout: 10s
      retries: 5
      # ensure that container is not started until the vaultwarden-s4-agent container starts
        condition: service_healthy

    image: s4-agent:latest # provided by the `s4-agent` tarball that the device loads
    command: s4 replicate
      - SYS_ADMIN # required in order to take BTRFS snapshots
    working_dir: /s4 # directory that should be replicated
    restart: always # allows the agent to start through reboots
    - vaultwarden-vol:/s4 # mounts the S4 Docker Volume to /s4
      # private key associated with volume. This key is how the agent authenticates
      # with the S4 remote.
      test: s4 ready /s4 # indicates that the s4 volume is ready to be used by the app
      interval: 10s
      timeout: 10s
      retries: 5

  # S4 Docker Volume that is created by the device using the S4 CLI when starting an app
    external: true
    # In order to allow for names to be unique (a user could potentially install multiple of the same app)
    # the name is created in the following format <app_name>_<unique_id>

Changes to Application Start / Stop Flow

In order to support S4, changes had to be made to how Fractal Mosaic starts and stops applications. The flow, at a high level, is described below.

Creating New S4 Volumes

  1. In the case that no encrypted storage keys were given, the device will look up the app's Docker Compose file in order to determine the number of S4 volumes to create. S4 volumes are currently determined by the external property of a volume defined in the volumes: section. Using the example Docker Compose file above, there is one volume defined: vaultwarden-vol. The Device will take that volume name, and append a unique ID to it: vaultwarden_vol_10vb. This allows for multiple of the same app to be running without any naming conflicts. For each volume, a OpenSSH ED25519 private key is generated.

  2. The Device then sends the respective private key's public keys to the Hive Backend for registration. The public keys are required in order for the app's replicator agents to push or pull an app's data.

  3. Now that keys are generated and registered with the remote, an s4-agent container is launched that initializes an S4 volume per generated private key.

  4. Once the volumes are created, the app will then be started. As mentioned in the App Docker Compose File Structure section, s4-agent healthchecks must pass before the app is allowed to start. Once passing, Docker Compose will start the other applications.

  5. Once the app is successfully started, the device finally encrypts the storage keys it generated using its configured DEVICE_PASSPHRASE, then send them to the Hive Backend. Doing so allows other Devices with the same DEVICE_PASSPHRASE to correctly decrypt the encrypted keys and pull in the app's data.

With the steps above complete, as the user uses the app, the app's data is replicated to the remote.

Pulling in an Existing S4 Volume

In the event that the user wants to switch the device their app is running on, they simply reschedule the app to the device they want.

  1. Hive sends a stop message to the first device.

  2. The first device stops the app and push up any lingering data. The volume (and its data) is left intact so that if the user ever reschedules back to this device, only the data that was added after rescheduling needs to be pulled in.

  3. Once the first device confirms that the app is stopped, the second device receives a start message from the Hive Backend that includes the encrypted storage keys in it. For brevity, a truncated example including the encrypted storage keys looks as follows:

        "encrypted_storage_apikeys": "<passphrase_encrypted_base64_blob>",
        "application": {
            "appstore": "Fractal App Store",
            "name": "BitWarden",
            "version": "latest",
            "repository": "",
            "deploy_key": null,
  4. The Device first determines if the app is a new installation by checking if encrypted storage keys ("encrypted_storage_apikeys") were provided in the start message. In this case, the message will contain encrypted storage keys.

  5. Device decrypts the encrypted_storage_apikeys. The decrypted keys are simply JSON with the following format:

        "<volume_name>": "-----BEGIN OPENSSH PRIVATE KEY-----\n..."
  6. With the now decrypted storage keys, the device launches ephemeral s4-agent Docker containers per volume that either clone or pull the latest changes for their respective <volume_name>s. A clone or pull is determined based on whether the volume exists on the Device's Docker Host or not. As previously mentioned, S4's interface is very similar to git; therefore, when getting the latest changes for an app's volume, the device runs an s4 clone if the Docker volume previously did not exist, or s4 pull when the volume already exists.

  7. Once the latest changes are synced into the S4 Docker volume, the app's docker-compose file is launched. As mentioned in Creating New S4 Volumes, an app's Docker Compose file now contains s4-agent containers whose healthchecks must pass before the app is started.

With the steps above complete, as the user uses the app, the app's data is replicated to the remote.

What's Tested

With the new changes to the device, new tests have been written to ensure the device maintains the stability it had previously. The Docker Storage Plugin is no longer installed, and the plugin's respective tasks have been disabled.

Edited by Justin Russell

Merge request reports