Migrate away for Minio to another S3 compatible storage

Decision (2026-10-05, PO)

versitygw (POSIX backend) replaces MinIO, starting in 26.10.0. Set VGW_KEEP_ALIVE=true and VGW_QUIET=true: without keep-alive, versitygw fails the latency gate.

Why:

  • Faster than MinIO on every operation Ryax uses, except the 16 MiB delete and listing (within noise). See the dev1 benchmarks below.
  • Faster than Garage almost everywhere. Garage reads full objects 1.2 to 1.5x slower than MinIO, reaches about half of MinIO's throughput under load, and would force new credentials on every install.
  • Lightest footprint, and no change to the credential format.

Data locality (see below) is not delivered by versitygw itself. The runner's S3 configuration has to become per-site, so locality can later be built over several stores, or reconsidered with a move to SeaweedFS.

Context

Ryax ships its filestore as the Bitnami minio chart 17.x (locked at 17.0.21) with bitnamilegacy/minio images (Chart.yaml#L56-59, values.yaml#L145-188). Both are dead ends:

  • Bitnami: bitnamilegacy/* images no longer get updates. That is the reason for #1320 (Migrate away from Bitnami Helm charts).
  • MinIO community edition:
    • The admin console was stripped in mid-2025.
    • Binaries and images stopped being published on 2025-10-23 (Chainguard).
    • It went into maintenance mode in December 2025.
    • The repo is now archived and marked "NO LONGER MAINTAINED" (github.com/minio/minio).
    • Docker Hub minio/minio was deleted on 2026-09-11, and quay.io/minio/minio is frozen at RELEASE.2025-09-07 (vonng, bex.co).

What the replacement must be: as light as MinIO, and as simple as possible, because Ryax uses very few S3 features. The workload is file hosting and sharing. It is latency sensitive: the filestore holds action inputs and outputs between workflow steps, so every file I/O of every step does at least one round trip to it.

Future requirement: data locality, meaning pinning data to a site or node pool and scheduling computation where the data lives. It is not needed for this migration, but the choice must not block it. See Future requirement: data locality below.

How Ryax uses S3 today

Code references are permalinks to the commits pinned in ryax-engine 05ea534: core 08d4e6e2, studio 423d77d0, action-wrappers 92266c4b.

Clients

All clients use the minio-py SDK 7.2.20, the version locked in core and studio uv.lock. They connect to host:port with secure=False: plain HTTP, no TLS, path-style addressing (minio-py only switches to virtual-host style for AWS and Aliyun hosts).

Consumer What it stores or reads Code
Runner Portal (form) uploads, zipped directory downloads, streaming file downloads to users (/filestore/{id}/{name} is proxied through the runner), execution data GC, reading resource_usage/ CSVs for runtime history and the IntelliScale predictor container.py#L161-176, app.py#L478-481, portal_controller.py#L182-193, stored_file_view.py#L21-53, garbage_collect_service.py#L63-69,L124, runtime_history.py#L152
Worker k8s Reads action input files, streams them to the action over gRPC in about 4 MiB chunks, writes output files, writes per-execution usage CSVs container.py#L128-136, grpc_v1_execution_service.py#L159-196,L323-327, metrics_service.py#L197-200
Worker ssh-slurm Same input and output pattern, copied over SSH to the HPC side container.py#L95-103, slurm_execution_service.py#L167,L678-680
Studio Static workflow files and project-variable files (iodata/ prefix), workflow copy (server-side copy), import and export engine.py#L22-39, filestore_service.py, workflow_service.py#L248-254, app.py#L86-87
Runner pricing and energy catalogs Anonymous read (public_read=True, no credentials) of an external, optional catalog bucket (RYAX_PRICING_CATALOG_S3_SERVER and RYAX_ENERGY_CATALOG_S3_SERVER, empty by default). It does not target the bundled filestore. container.py#L182-209, s3_engine.py#L29-33
Action containers (action-wrappers) No S3 access at all. Files and directories travel as bytes inside the gRPC IOFileValue and IODirectoryValue messages between the worker and the action. Actions never see an endpoint, a credential or a presigned URL. execution_v3.proto#L105-115
Repository, Action Builder, front, IntelliScale No S3 access.

The shared implementation is ryax/common/storage/s3/s3_service.py in core.

S3 operation inventory

S3 operation Used? Where and how
PutObject yes Known-size put: core s3_service.py#L102, studio filestore_service.py#L39. Streamed put with length=-1: core #L83-88 with 5 MiB parts, studio #L55-60 with 8 MiB parts. A stream below the part size becomes a single PUT.
Multipart (Create, UploadPart, Complete, Abort) yes, implicitly minio-py switches the streamed put to multipart once a stream exceeds one part (5 or 8 MiB), and calls AbortMultipartUpload on failure (minio-py api.py put_object). Parts are uploaded sequentially.
GetObject (whole object, streamed) yes core #L152, studio #L98. No range reads.
HeadObject (stat) yes Only before a delete, to map NoSuchKey to "not found": core #L121, studio #L75. minio-py also calls it inside CopyObject.
DeleteObject yes core #L127, studio #L80. Single key, no DeleteObjects batch.
ListObjectsV2 (prefix, recursive, no delimiter) yes core #L173, used only on resource_usage/ for runtime history and the predictor.
CopyObject (server-side) yes, studio only filestore_service.py#L114, on workflow copy. UploadPartCopy would only be used for sources over 5 GiB, but studio uploads are capped at 1 GiB (values.yaml#L92).
HeadBucket and CreateBucket yes At startup, every service creates its bucket if it is missing: core s3_engine.py#L49-50, studio engine.py#L35-36.
GetBucketLocation yes, implicitly minio-py issues it once per bucket, signed with region us-east-1, then signs every later request with the region it returns. CreateBucket defaults to us-east-1 (api.py#L481-513, #L616-667). The server must accept us-east-1 as its region.
Presigned URLs no Downloads are proxied by the runner (file_storage_controller.py#L38-60).
Bucket policies, ACLs, versioning, lifecycle, notifications, object lock, SSE, tagging no Retention is done by the runner GC, not by lifecycle rules.
MinIO admin API, mc, console, object browser no The console is disabled (minio.console.enabled: false), and the chart's mc provisioning job is not used.

Object sizes and access pattern

  • Keys follow executions-data/<uuid>/<filename> (core), iodata/<uuid>/<filename> (studio) and resource_usage/<site>/<action>/<execution>.csv (worker metrics).
  • Directories are zipped into one object. Keys never collide as both a file and a "directory".
  • Each workflow step does one GET per file input and one PUT per file output. Scalar IOs go through RabbitMQ, not S3.
  • Objects are typically KB to tens of MB; studio uploads are capped at 1 GiB. A typical file is below the 5 MiB part size, so most writes are single PUTs.
  • The latency that matters is small-object PUT and GET time to first byte.

Buckets and credentials

  • One bucket, ryax-filestore (filestoreName), shared by the runner, studio and the workers. The services create it themselves at startup; the chart does not provision it (minio defaultBuckets is empty).
  • Credentials:
    • The Secret ryax-minio-secret is rendered by common-resources (filestore-secret.yaml, values.yaml#L79-86) when global.secrets.create is true, and is supplied by the operator otherwise (install_ryax_argocd.md#L50).
    • Keys: filestore = ryax-minio.<ns>:9000, filestore-access = root-user = ryax, and filestore-secret = root-password = 12 random alphanumeric characters.
    • The services authenticate as the MinIO root user. There are no per-service keys.
  • Consumers of the Secret: the runner, studio and the worker charts (runner deployments.yaml#L96-117, studio deployments.yaml#L88-104, worker-k8s deployments.yaml#L147-163).
  • Remote workers reach the main site's filestore through Skupper as ryax-minio-ext:9000. The docs hard-code service/ryax-minio and port 9000 (worker-install.md#L392-400), and k8s-ryax-config.py rewrites ryax-minio to ryax-minio-ext (k8s-ryax-config.py#L27,L76-79).

Deployment details that matter for a migration

  • Single replica (mode: standalone) on one ReadWriteOnce 20Gi PVC (100Gi in env/dev.yaml), with updateStrategy: Recreate. The pod runs as UID 1200.
  • The PVC ryax-minio is a Helm-managed resource of the minio subchart with no helm.sh/resource-policy: keep (templates/pvc.yaml in minio-17.0.21). Removing or disabling the dependency deletes the PVC and the data, unless the PV reclaim policy is Retain.
  • minio.metrics.enabled: true, but no ServiceMonitor is enabled.
  • The data is in MinIO's xl.meta format under /bitnami/minio/data, not plain files. Moving it to anything other than RustFS needs an S3-level copy.
  • Airgap: the image list is produced by helm template (generate-image-list.sh), so a new filestore image is picked up automatically. Any migration-job image must be rendered too.
  • Tests and dev: core/docker-compose.yml and studio/docker-compose.yml run reg.mini.dev/minio:latest.

Constraints on the choice: no presigned URLs, no bucket policies and no MinIO-only API are used, so the requirement is plain S3 over HTTP. The remaining hard requirements are:

  1. Signature V4 with region us-east-1, plus GetBucketLocation, because of minio-py.
  2. CreateBucket from the client.
  3. Multipart upload.
  4. CopyObject.
  5. Ideally the current credentials (ryax and a 12-character secret) are accepted verbatim, so that GitOps installs (global.secrets.create=false) need no secret change.

Candidate comparison

Data was checked on 2026-10-05. Benchmarks marked † are third-party, single-run and setup-specific. None of them tests all candidates on the same hardware.

Criterion versitygw (POSIX backend) Garage SeaweedFS RustFS
License Apache-2.0 AGPL-3.0 Apache-2.0 (plus a paid "Enterprise" key) Apache-2.0 (CLA to RustFS, Inc.)
Latest release and cadence v1.8.0, 2026-09-04; about monthly (releases) v2.4.1, 2026-09-08; a minor every 2 to 5 months (releases) 4.48, 2026-09-28; about weekly (releases) 1.0.1, 2026-10-03; GA was 1.0.0 on 2026-09-16 (GA post)
Governance Versity Software; about 3k stars Deuxfleurs association with 4 maintainers (GOVERNANCE.md); NLnet/NGI funded; about 154 contributors One dominant maintainer (chrislusf, about 10.5k of the commits); about 400 contributors Company-driven (RustFS, Inc.); about 185 contributors
Components 1 static Go binary, 1 container, no metadata database 1 static Rust binary, 1 pod with 2 PVCs (meta and data), metadata in LMDB or SQLite 1 static Go binary (weed), but master, volume, filer and s3 inside it; weed mini or weed server -s3 runs them in one process 1 binary with an embedded console
Image size (compressed) about 31 MB, amd64 and arm64, Docker Hub and ghcr about 27 MB FROM scratch, amd64, arm64, arm and 386 (dxflrs/garage) about 194 MB (Alpine plus Rust helpers), multi-arch, Docker Hub and ghcr about 111 MB (musl) or 160 MB, amd64 and arm64, Docker Hub and ghcr
RAM No published minimum; stateless gateway 1 GB minimum per the docs (site); idle RSS about 35 MiB † No published minimum; the index costs about 16 to 20 B per object in RAM; idle RSS about 151 MiB †; chart all-in-one requests 1Gi No published minimum; idle RSS about 199 MiB †
Small-object latency PUT of 1 to 4 KiB objects 3 to 4× MinIO, GET slower than MinIO († scaleoutsean 2024). No independent comparison with the other three. 4 KiB PUT/GET 1240/176 ops/s, 256 KiB PUT p50 9.9 ms († next-template#116); write p50 3.0 ms vs MinIO 13.2 ms († gist) 4 KiB PUT/GET 2406/2708 ops/s, 256 KiB PUT p50 6.4 ms †; the best 4 KiB GET of four systems in rustfs#1500 †; designed for small files 4 KiB PUT/GET 584/2639 ops/s †; 4 KiB GET about 3× slower than MinIO (rustfs#2154); the "2.3× faster" claim has no published data
Ryax ops (Put, Get, Head, Delete, ListV2, CopyObject, multipart, Head and CreateBucket) All supported (S3 API reference) All supported (compatibility) All supported (wiki); the listing delimiter must be /, which Ryax does not use All supported (matrix)
Presigned URLs, path-style (not needed today) Yes, yes Yes, yes Yes, yes Yes, yes
Region and minio-py fit Default region us-east-1 (VGW_REGION) ✅ Default s3_region = "garage", and any other region in the SigV4 scope is rejected. Must set s3_region = "us-east-1" ⚠️ The signing key is derived from the client's scope region, so any region works ✅ ✅ (MINIO_REGION is honoured)
Accepts current creds (ryax plus a 12-character secret) Probably: no format or length check found in the v1.8 source; root creds come from ROOT_ACCESS_KEY_ID and ROOT_SECRET_ACCESS_KEY ✅ No. Since v2.3 the key ID needs at least 8 characters and the secret at least 16 (key_table.rs); before v2.3 it had to be GK+hex ❌ Yes: static s3.json identities or AWS_ACCESS_KEY_ID env, no format check ✅ Yes: the access key needs 3 or more characters and the secret 8 or more; MINIO_ROOT_USER and MINIO_ROOT_PASSWORD are mapped ✅
Declarative bootstrap Root creds from env or Secret. The bucket is created by the Ryax services (works as today). garage server --single-node with GARAGE_DEFAULT_ACCESS_KEY, GARAGE_DEFAULT_SECRET_KEY and GARAGE_DEFAULT_BUCKET since v2.3 (quick start); a different secret on restart is an error s3.existingConfigSecret (JSON) or s3.credentials.admin.existingSecret; a createBuckets hook Job Env or secret.existingSecret; no default-bucket option
Official Helm chart, operator Chart at oci://ghcr.io/versity/versitygw/charts/versitygw (0.4.5); existingSecret needs the keys rootAccessKeyId and rootSecretAccessKey; image tag defaults to latest. No operator, and none needed. Chart only in-tree (script/helm/garage), not published to a repo; its default image is amd64 only. Community operator only (garage-operator). Chart repo seaweedfs.github.io/seaweedfs/helm with allInOne mode (default data type emptyDir!); operator v1.0.40 Chart at charts.rustfs.com (distributed mode by default); operator v0.1.0, "not production-ready"
Single node vs HA Single gateway on a ReadWriteOnce PV, same as today. Scale-out needs ReadWriteMany or a shared filesystem plus the standalone IAM. No replication of its own. rf=1 works but is discouraged (one metadata copy); native rf=3 across zones 000 single copy; native replication, raft masters Single-node single-disk works, flagged "data lost if disk fails"; independent review reports weak degraded reads in distributed mode (vonng)
Data on disk Plain files (<bucket>/<key>), metadata in xattrs, or a --sidecar directory for NFS (wiki); PUT is atomic (O_TMPFILE plus linkat) Content-addressed blocks plus an LMDB or SQLite database; LMDB can corrupt on unclean shutdown (config) Volume files plus a filer leveldb; deletes are reclaimed by vacuum MinIO-like xl.meta
Migration from MinIO S3 copy (rclone or mc mirror) S3 copy S3 copy, or remote.mount plus remote.cache In place: reads MinIO single-drive xl.meta (marked Preview, one-way, no MinIO SSE objects) (doc)
Airgap One small image, no outbound calls One small image (the in-tree chart also pulls busybox) One large image Update check to version.rustfs.com on by default (set RUSTFS_CHECK_UPDATE=false)
Monitoring StatsD and DogStatsD only, no Prometheus endpoint Prometheus /metrics on the admin port Prometheus Prometheus and OpenTelemetry
Notable risks No independent small-object GET benchmark; needs a filesystem with user xattrs (or --sidecar) Region and credential incompatibility; rf=1 discouraged; known slowdown around 10M objects (known issues) Bus factor of one; more knobs (volume size limit, vacuum, many buckets exhaust volume slots) 35 security advisories in 10 months, 4 critical (auth bypass with a hard-coded token, root backdoor via ImportIam, UploadPartCopy authorization bypass, and others) (advisories); 1.0 is three weeks old

Others considered and ruled out

  • Ceph RGW (Rook): far too heavy for a single-node filestore. It needs mons with at least 5 GB RAM and 2 cores each, OSDs with at least 4 GiB each, plus a mgr and the RGW, and raw devices (Ceph minimum hardware, Rook prerequisites). That is more than 10 GB RAM and four daemon types.
  • Zenko CloudServer: alive (9.4.4, September 2026), but driven by Scality's products. It is Node.js, its Docker Hub image is stale since 2023 (current builds are on ghcr), and it is poorly documented for standalone use (repo).
  • Staying on MinIO through a fork (no migration needed):
    • pgsty/silo is AGPL, security fixes only, and has one maintainer.
    • The Chainguard EmeritOSS fork is also security fixes only, and its free image is cgr.dev/chainguard/minio:latest only (version tags are paid).
    • Either is a stop-gap, not a destination.
  • s3proxy (filesystem backend aimed at testing, Java, about 164 MB image) and rclone serve s3 ("Experimental", metadata in memory) (rclone docs): fine for CI, not for production.
  • Apache Ozone, OpenStack Swift: multi-service clusters, too heavy.

Future requirement: data locality

Added 2026-10-05 at the Product Owner's request. Not needed for the 26.10.0 migration, but the choice made now has to leave room for it.

At some point Ryax wants data locality, meaning two related abilities:

  1. Pin data. Force some data (a bucket, a project's files, or a workflow's IO) to stay on a given site or node pool. This serves sovereignty, avoids WAN transfer, and keeps a dataset next to the hardware that uses it.
  2. Move computation to the data. When an action's inputs already live somewhere, the scheduler prefers a node pool there, instead of shipping the files to wherever the action lands.

Today there is a single filestore in the main site, and remote workers reach it over Skupper (see Deployment details above). Every byte of IO for a remote action crosses the inter-site link.

What each candidate offers

Pin data to a place Find where data lives (for scheduling) What Ryax would have to build
versitygw No notion of placement: one gateway over one filesystem. No. The location is "the PV of that gateway". All of it. Run one gateway per site on local storage. Ryax records which endpoint holds each object, routes reads and writes to it, moves data between sites itself, and has no single namespace across sites. The gateway is small enough for this to be cheap to run, but the locality logic lives entirely in the runner's storage layer and scheduler.
SeaweedFS Native, per path. Volume servers carry -dataCenter and -rack tags, and fs.configure -locationPrefix=<path> -dataCenter=<dc> (also -rack, -dataNode, -replication, -collection) assigns the writes under a path prefix, and so a bucket or key prefix, to that place (fs.configure source, path-specific configuration). The replication setting decides whether, and how many, copies go to other data centers or racks. Yes. A file maps to volume ids, and the master's volume lookup returns the volume servers holding them, so a scheduler can resolve "where is this input" to a data center or rack. Map Ryax sites and node pools to SeaweedFS data centers and racks, set a path rule per pinned scope, and have the scheduler query locations. Operationally: one cluster spans the sites (masters in raft and a filer metadata store, which need a stable link), or one cluster per site with filer.sync between them. That is the HA and multi-site complexity we said we did not want for a single node today.
Garage No per-bucket pinning. Zones are first-class, but placement is by hash partition, and replicas are deliberately spread across zones for diversity (layout docs). That is the opposite of locality. Only by recomputing the partition owner from the layout. Not exposed for this use. One cluster per site, same as versitygw, plus the credential and region issues above.
RustFS No: erasure sets across the nodes of a pool, no per-bucket placement. No. One deployment per site, same as versitygw.

Impact on the recommendation

  • The latency gate does not change. The dev1 benchmark below still holds: tuned versitygw is the fastest and lightest single-node store.
  • Locality points to SeaweedFS. It is the only candidate where locality is a storage feature rather than something Ryax builds. With versitygw (or Garage, or RustFS), locality means several stores, one per site, plus Ryax-side routing and data movement. That is doable, and the gateway is tiny, but it is real runner and scheduler work.
  • Either way, Ryax needs multi-endpoint awareness. Even with SeaweedFS, a remote action should talk to a gateway in its own site, not to the main site's, or the IO still crosses the link. So the runner's storage layer has to learn "which S3 endpoint for this site" in both cases. This is the part to design now so that it does not lock us in.
  • Switching later costs a second data migration. The migration plan below (an S3 copy at upgrade) applies again. It is designed, but each one costs downtime and admin attention.

Decision needed (PO): when is locality expected?

  • Within about a year, or as part of the multi-site roadmap: choose SeaweedFS now, as a single node first (weed mini or server -s3, one data center), so that turning on locality later is a configuration change, not a migration. Accept its higher memory use and more knobs.
  • Later or uncertain: keep versitygw for 26.10.0, the simplest choice and the best latency, and make the runner's S3 configuration per site so that locality can be built over several stores, or reconsidered with a later move to SeaweedFS.

Recommendation

Use versitygw with its POSIX backend, deployed by a small Ryax-owned filestore subchart. Accept it only after a latency check on dev1 (see Open questions).

Why:

  1. It is the simplest thing that covers Ryax's needs.
    • One stateless Go binary in a container of about 31 MB.
    • No cluster layout, no metadata database, no volume and vacuum tuning, no replication factor to explain.
    • Objects are plain files on the PV. Backups (Velero file-level or rsync), debugging, and recovery after a crash need no special tooling, and a corrupt metadata database cannot take the store down.
    • The on-disk layout is the same idea as Ryax's existing posix storage backend.
  2. It covers exactly what Ryax uses (Put, Get, Head, Delete, ListV2, CopyObject, multipart, Head and CreateBucket, GetBucketLocation), and its default region is us-east-1, so minio-py works unchanged.
  3. No credential or secret changes. The root access key and secret come from env, and no format or length check was found. The new Deployment can read root-user and root-password from the existing ryax-minio-secret, so GitOps installs with global.secrets.create=false keep working with no new Secret keys.
  4. Apache-2.0, monthly releases, company-maintained, with an official chart and multi-arch images on Docker Hub and ghcr. It is the lightest footprint of the four, together with Garage.
  5. Latency: writes go straight to the local filesystem (an atomic O_TMPFILE plus linkat), and reads come straight from the page cache. Its PUT path beat MinIO by 3 to 4× on tiny objects in the one benchmark available. Its GET path was slower than MinIO in that same test, so this has to be measured (see Open questions).

Trade-off we accept: there is no built-in replication or HA. That is no worse than today's single MinIO on a single ReadWriteOnce PV.

Runner-up: SeaweedFS (weed mini, or weed server -s3 in one pod).

  • It has the best small-object numbers in the third-party benchmarks, Apache-2.0, very active development, reuses the current credentials, ignores the region, and has native replication for a future HA story.
  • Not first because it carries more moving parts and knobs for a single-node store (master, volume, filer, vacuum, volume size limits), a 194 MB image, and a bus factor of one.
  • Pick it if the latency check shows versitygw is slower than MinIO on GET, or if HA becomes a requirement.

Garage is very light and well governed, but on our usage it needs s3_region = "us-east-1". It also rejects the current credentials (ryax is shorter than its 8-character key minimum and the secret shorter than its 16-character minimum), so every install, GitOps included, would need new secret values. On top of that, rf=1 is discouraged, LMDB can corrupt on unclean shutdown, and the chart is not published to a repo.

RustFS is the only candidate that migrates in place from MinIO's data directory, and it accepts MinIO env and credentials. But its security record (35 advisories, 4 critical, in 10 months), the default phone-home update check, slower small GETs, and a GA that is three weeks old rule it out for now. Revisit it in 2027.

Migration plan for existing installs

Chart changes (ryax-engine)

  1. Add a subcharts/filestore chart, like datastore and registry. It holds:

    • a Deployment with 1 replica, strategy: Recreate, runAsUser: 1200, fsGroup: 1200, and a read-only root filesystem;
    • a PVC ryax-filestore (default 20Gi, existingClaim supported);
    • a Service on port 9000.

    The container runs a pinned image (versity/versitygw:v1.8.0 or the ghcr equivalent; never latest) with:

    • VGW_BACKEND=posix and data at /data;
    • VGW_PORT=:9000;
    • VGW_REGION=us-east-1;
    • ROOT_ACCESS_KEY_ID from ryax-minio-secret/root-user and ROOT_SECRET_ACCESS_KEY from root-password;
    • optionally --sidecar for NFS-backed volumes, replacing the current volumePermissions note.

    Owning ~5 templates avoids the upstream chart's fixed Secret key names (rootAccessKeyId/rootSecretAccessKey) and its latest default, and fits #1320's goal of dropping third-party charts for simple components.

  2. Keep the Secret name ryax-minio-secret and all its keys. Renaming it would break every GitOps install and every remote worker. Maybe rename later behind an alias value.

  3. Change the filestore endpoint: filestoreService: ryax-filestore in common-resources. For one or two releases, also render a compatibility Service ryax-minio:9000 that selects the new pod. This keeps working:

    • GitOps secrets that hard-code ryax-minio.<ns>:9000;
    • existing Skupper connectors (--workload service/ryax-minio);
    • k8s-ryax-config.py.

    Update worker-install.md and k8s-ryax-config.py to the new name.

  4. Remove the minio dependency in two steps, to avoid deleting the old PVC:

    • Release N: keep the minio subchart running, as the copy source, and set minio.persistence.annotations: {helm.sh/resource-policy: keep}. Add the new filestore and its copy initContainer. Point all services at the new filestore.
    • Release N+1: drop the minio dependency. Helm leaves the annotated ryax-minio PVC in place for rollback; the release notes tell the admin to delete it by hand once satisfied.

  5. Update the docs:

    • architecture.md (Filestore section);
    • install_ryax_argocd.md: the secret table is unchanged, the placement keys move from minio.* to filestore.*;
    • create-backups.md;
    • the chart READMEs.

    Replace reg.mini.dev/minio in the core and studio docker-compose.yml with versitygw, so tests run against the real server.

Data copy at upgrade

The copy has to read through MinIO's S3 API, because MinIO's xl.meta layout is not plain files. A pre-upgrade hook cannot be used: on the first upgrade the new filestore does not exist yet.

  • Proposed approach: an initContainer on the new filestore pod runs rclone copy from ryax-minio:9000, bucket ryax-filestore, using the root credentials in ryax-minio-secret.
    • The new store only becomes Ready, and the services can only reach it, after the copy has finished. Until then the runner and studio fail their startup create_bucket and restart.
    • Two ways to write the data:
      • Variant A: copy straight into the PV as plain files (/data/ryax-filestore/<key>). versitygw serves files that have no ETag xattr with a default ETag (posix.go HeadObject). Content-Type is lost, but Ryax keeps the mimetype in its database.
      • Variant B: S3 to S3, through a temporary versitygw on localhost inside the initContainer.
    • Both variants must be validated on dev1.
  • Idempotent: once done, the initContainer writes a marker (for example /data/.ryax-migrated-from-minio) and skips on every later restart.
  • It is a no-op on fresh installs, where there is no ryax-minio Service.
  • Downtime: the upgrade must quiesce writers, or rclone must make a second, final pass once the runner, studio and workers are switched. The simplest option is a documented maintenance window: the copy runs while the old services are still up but no workflow is running, then the services restart on the new endpoint. The copy time is roughly volume ÷ throughput. Size it on dev1.
  • Remote workers: remote sites need no data move. They only need the Skupper connector to keep resolving ryax-minio, which the compatibility Service provides.

Credentials and secret compatibility

  • No Secret changes are needed: same name, same keys, same ryax user and password.
  • Installs with global.secrets.create=false need nothing, as long as their filestore key keeps pointing at ryax-minio.<ns>:9000 (the compatibility Service) or is updated to ryax-filestore.<ns>:9000.

Rollback

  • The old PVC is only read during the copy and is kept by the keep annotation, so helm rollback to the MinIO release brings the old store back.
  • Objects written to the new store after the upgrade are lost on rollback unless they are reverse-synced with rclone sync filestore → minio. Document this as a script.
  • Once N+1 is deployed and the old PVC deleted, rollback means restoring from a backup.

Airgap

  • New images: versitygw (about 31 MB) and the copy-job image. Use rclone/rclone pinned, or reuse a Ryax core image that already ships minio-py, so no new image is needed.
  • Both images are picked up by generate-image-list.sh, because the initContainer is rendered by helm template. Add a helm-unittest assertion to keep it that way.
  • Drop bitnamilegacy/minio from the list once release N+1 ships.

Open questions

  1. Latency check, the gate for the decision. Run warp and a Ryax-shaped test on dev1, on the same PV class, against MinIO (current), versitygw and SeaweedFS:

    • object sizes 4 KiB, 64 KiB, 1 MiB and 16 MiB;
    • PUT then GET of the same object, single client and 8 parallel clients;
    • report p50 and p99.

    Pass criterion: no worse than the current MinIO. Also measure an end-to-end workflow with file IO between two actions.

  2. Filesystem support. Do the storage classes our customers use support user xattrs (ext4 and xfs do; NFS only from v4.2 with a recent kernel)? Otherwise, do we default to --sidecar? Does Velero fs-backup (kopia) preserve xattrs? If not, use sidecar mode or a CSI snapshot.

  3. Monitoring. versitygw exports only StatsD. Do we need filestore metrics (minio metrics.enabled is on but never scraped)? If yes, add a statsd_exporter sidecar.

  4. Downtime budget. Is a maintenance window acceptable at upgrade, or do we need a two-pass sync with a short freeze?

  5. Least privilege. Should services keep using the root key, or should we move to per-service IAM accounts later (versitygw has a flat-file IAM and, since v1.8, a standalone IAM service)? This is out of scope for the migration itself.

  6. Rename the Secret (ryax-minio-secret → ryax-filestore-secret) now behind a value, or leave it for a later breaking release?

  7. Multi-site. architecture.md describes a "global" public storage between sites, but the code only has a local store reached over Skupper. Confirm that no customer exposes the filestore publicly, since versitygw has an ingress but we would not enable it by default.

  8. Skip S3 entirely? Core already has a posix storage backend (RYAX_LOCAL_STORAGE_TYPE=posix), but studio and remote workers need network access, so an S3 endpoint stays required. Confirm.

  9. Data locality timeline. When is locality expected? That decides between SeaweedFS now and versitygw now with per-site stores later. See Future requirement: data locality.

Latency benchmark on dev1 (2026-10-05)

This answers Open question 1, except the end-to-end workflow measurement, which was not run.

Verdict: versitygw passes the gate, but only with VGW_KEEP_ALIVE=true.

  • With keep-alive, versitygw is faster than MinIO on every operation Ryax uses except two:

    • deleting a 16 MiB object, which is off the critical path (runner GC);
    • listing, where it is +5 % and within the run-to-run noise.

    Small PUTs are about 3× faster and small GETs about 1.3× faster. The warp mixed load does +53 % more operations per second. versitygw uses about 1/15 of MinIO's memory and half its CPU per request.

  • With the default settings it fails the gate. versitygw defaults to --keep-alive=false and answers every request with Connection: close, so each minio-py call pays a new TCP connection plus a cluster DNS lookup. As a result:

    • HEAD, GET and DELETE are about 2× slower than MinIO (+1.8 ms each);
    • warp is 25 % slower.

    The filestore subchart must set VGW_KEEP_ALIVE=true. It should also set VGW_QUIET=true, which drops the per-request stdout log line. Add both to the env list in step 1 of the migration plan.

  • SeaweedFS (weed mini) is faster than MinIO on PUT (about 1.9× on small objects), level with it on GET, and better at listing. Its weak points:

    • 16 MiB PUT tail spikes (p99 1.1 s);
    • 16 MiB GET time to first byte twice MinIO's;
    • the most memory: 0.5 to 1 GiB, where MinIO uses 0.2 to 0.6 GiB.

    It is slower than tuned versitygw on nearly every operation, so latency gives no reason to prefer it.

Setup

  • Cluster: dev1 (Scaleway Kapsule, Kubernetes v1.36.4) with two PLAY2-MICRO nodes (4 vCPU, 7.4 GiB) in fr-par-2. The bench ran in a throw-away namespace s3-bench, deleted afterwards. Nothing in ryaxns was touched.

  • Placement:

    • all servers on scw-dev1-default-2ceee72329154e3c8b7a89c1cbf9a, the node that hosts ryax-minio;
    • the client on scw-dev1-default-0801e8ba7cd84f449fcdfcfd1b769, the node that hosts ryax-runner.

    Every request crosses the node network, as runner to ryax-minio does.

  • Storage: each server has its own 20Gi PVC on sbs-default, the class of ryax-minio. That is Scaleway Block Storage formatted ext4 (relatime).

  • Servers:

    • MinIO: docker.io/bitnamilegacy/minio:2025.7.23-debian-12-r3, the exact tag running in ryaxns (MinIO DEVELOPMENT.2025-07-23). It runs with the chart's env: standalone, MINIO_BROWSER=off, root user and password from files, data in /bitnami/minio/data.
    • versitygw: docker.io/versity/versitygw:v1.8.0 with VGW_BACKEND=posix, VGW_BACKEND_ARG=/data, VGW_PORT=:9000, VGW_REGION=us-east-1, and ROOT_ACCESS_KEY_ID / ROOT_SECRET_ACCESS_KEY from a Secret. Metadata is kept in xattrs, with no --sidecar. Two configurations were measured:
      • default;
      • keep-alive, which adds VGW_KEEP_ALIVE=true and VGW_QUIET=true.
    • SeaweedFS: docker.io/chrislusf/seaweedfs:4.48 with weed mini -dir=/data -s3.port=9000 -master.telemetry=false -webdav=false -admin.ui=false. Credentials come from AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, which the log shows as Added admin identity from AWS environment variables: name=admin-ryax.

  • Resources: the same for all servers.

    • Limits: 2 CPU and 2Gi. Requests: 500m CPU and 768Mi, except 256Mi for the keep-alive versitygw, which was added mid-run and would not fit on the node otherwise. Requests only affect scheduling.
    • All run with runAsUser: 1200 and fsGroup: 1200, like the chart.
    • CPU throttling was negligible: 105 of 15 502 CFS periods for MinIO, 4 for SeaweedFS, 0 for versitygw.

  • Credentials: user ryax and a 12-character random alphanumeric secret on every server, as in ryax-minio-secret.

  • Client: a pod running python:3.12-slim with minio-py 7.2.20, the version locked in core. It is built as Minio("<svc>.s3-bench:9000", access_key, secret_key, secure=False), exactly like S3Engine.connect():

    • no region is passed, so minio-py runs GetBucketLocation and signs with the region it returns;
    • one client is shared by all threads, like S3Storage.

Method

  • Startup: bucket_exists + make_bucket, as S3Engine.create_bucket() does, timed 20 times with a fresh client each time.

  • Per object size (1 KiB, 4 KiB, 64 KiB, 1 MiB, 16 MiB): each iteration writes a fresh key executions-data/<uuid>/out.bin and times, in order:

    1. put_object with a known length, like write_file;
    2. stat_object;
    3. get_object, timed at the return of headers (time to first byte) and at the full body read;
    4. remove_object.

    The iteration counts are 300 for sizes up to 64 KiB, 200 for 1 MiB and 30 for 16 MiB. The first 20, 10 and 3 respectively were discarded as warm-up. 4 KiB was added because Open question 1 asks for it.

  • Streamed put: put_object(..., length=-1, part_size=5 MiB), like S3Storage.write, at 64 KiB, 1 MiB (single PUT) and 16 MiB (a 4-part multipart).

  • copy_object: at 64 KiB and 1 MiB, 100 timed copies after 10 warm-up copies. minio-py runs it as a HEAD followed by a PUT-copy.

  • list_objects: prefix="resource_usage/site-1/", recursive=True over 1000 keys, with another 1000 keys outside the prefix. It was timed 30 times after 3 warm-ups.

  • Concurrency 8: 8 threads share the client, and each does 60 PUTs then 60 GETs of 64 KiB, after 5 warm-ups.

  • Order: 3 rounds, interleaved MinIO → versitygw (default) → versitygw (keep-alive) → SeaweedFS, three times over. An earlier smoke round was discarded.

    • The main table pools the 3 rounds.
    • The collapsed table gives the min–max of the per-round p50 values, to show the noise.

  • warp: minio/warp:v1.3.1 mixed at 64 KiB, 8 concurrent clients, 60 s and 1000 objects, with 2 interleaved rounds per server.

  • Resources: kubectl top every 10 s, plus the cgroup memory.stat file.

Compatibility: every check passed on all servers

Check MinIO versitygw SeaweedFS
ryax with a 12-character secret accepted; a wrong secret rejected with SignatureDoesNotMatch ✅ ✅ ✅
GetBucketLocation returns us-east-1, and requests signed with it are accepted ✅ ✅ ✅
make_bucket without a region, then bucket_exists ✅ ✅ ✅
Missing key on stat or get returns NoSuchKey, which Ryax maps to FileNotFoundError ✅ ✅ ✅
Single-PUT ETag equals the MD5 ✅ ✅ ✅
Streamed 16 MiB multipart with 5 MiB parts (core, ETag -4) and 8 MiB parts (studio, ETag -2), byte-exact round trip ✅ ✅ ✅
Multipart aborted on a failing stream, with no pending upload left ✅ ✅ ✅
16 MiB copy_object, byte-exact ✅ ✅ ✅
Prefix listing returns exactly the 1000 keys ✅ ✅ ✅

versitygw xattrs: the default xattr mode works on sbs-default ext4, so no sidecar is needed. The ETag, Content-Type and user metadata survive a pod restart. Objects are plain files, for example /data/ryax-filestore-r1/x/xattr.bin.

Results: minio-py latency in ms (p50 / p95 / p99 / mean, 3 rounds pooled)

Operation Size n/server MinIO p50 / p95 / p99 / mean versitygw (default) p50 / p95 / p99 / mean versitygw (keep-alive) p50 / p95 / p99 / mean SeaweedFS p50 / p95 / p99 / mean p50 vs MinIO: vgw / vgw-ka / SWFS
bucket_exists+make_bucket (new client) - 60 8.67 / 11.92 / 22.69 / 9.47 7.43 / 8.97 / 9.80 / 7.55 4.63 / 6.46 / 6.90 / 4.88 6.12 / 7.45 / 8.08 / 6.22 0.86× / 0.53× / 0.71×
put (known size) 1KiB 900 4.47 / 6.29 / 10.96 / 4.79 3.88 / 4.94 / 5.51 / 3.97 1.56 / 2.15 / 2.44 / 1.63 2.38 / 2.99 / 3.54 / 2.43 0.87× / 0.35× / 0.53×
stat (HEAD) 1KiB 900 1.60 / 2.21 / 2.45 / 1.67 3.60 / 4.53 / 4.96 / 3.67 1.29 / 1.81 / 2.12 / 1.34 1.48 / 1.99 / 2.39 / 1.54 2.25× / 0.80× / 0.92×
get, time to first byte 1KiB 900 1.61 / 2.14 / 2.67 / 1.67 3.39 / 4.31 / 5.06 / 3.46 1.23 / 1.72 / 2.13 / 1.29 1.63 / 2.18 / 2.61 / 1.69 2.10× / 0.76× / 1.01×
get, full read 1KiB 900 1.65 / 2.21 / 2.74 / 1.71 3.51 / 4.48 / 5.17 / 3.59 1.27 / 1.76 / 2.23 / 1.33 1.67 / 2.23 / 2.65 / 1.73 2.13× / 0.77× / 1.01×
delete 1KiB 900 1.90 / 2.49 / 2.85 / 1.96 3.51 / 4.48 / 5.08 / 3.59 1.21 / 1.67 / 1.98 / 1.26 1.45 / 1.95 / 2.23 / 1.50 1.85× / 0.64× / 0.77×
put (known size) 4KiB 900 4.46 / 6.00 / 8.55 / 4.71 3.89 / 4.92 / 5.40 / 3.99 1.64 / 2.15 / 2.51 / 1.68 2.42 / 3.23 / 3.91 / 2.64 0.87× / 0.37× / 0.54×
stat (HEAD) 4KiB 900 1.59 / 2.26 / 2.62 / 1.67 3.60 / 4.48 / 5.47 / 3.68 1.28 / 1.78 / 2.14 / 1.33 1.49 / 2.06 / 2.47 / 1.55 2.26× / 0.80× / 0.93×
get, time to first byte 4KiB 900 1.60 / 2.16 / 2.58 / 1.65 3.41 / 4.28 / 4.95 / 3.48 1.25 / 1.65 / 2.08 / 1.29 1.69 / 2.29 / 2.70 / 1.76 2.13× / 0.78× / 1.06×
get, full read 4KiB 900 1.66 / 2.23 / 2.69 / 1.72 3.53 / 4.49 / 5.11 / 3.61 1.29 / 1.74 / 2.14 / 1.34 1.77 / 2.41 / 2.74 / 1.84 2.13× / 0.78× / 1.07×
delete 4KiB 900 1.90 / 2.51 / 2.83 / 1.96 3.51 / 4.44 / 4.90 / 3.58 1.22 / 1.69 / 1.89 / 1.26 1.44 / 1.91 / 2.54 / 1.52 1.85× / 0.64× / 0.76×
put (known size) 64KiB 900 5.57 / 7.13 / 9.40 / 5.77 5.27 / 6.40 / 6.97 / 5.32 2.53 / 3.16 / 3.62 / 2.54 3.60 / 4.39 / 4.98 / 3.64 0.95× / 0.45× / 0.65×
stat (HEAD) 64KiB 900 1.68 / 2.32 / 2.69 / 1.74 3.68 / 4.71 / 5.58 / 3.75 1.29 / 1.82 / 2.15 / 1.35 1.52 / 2.17 / 2.39 / 1.59 2.19× / 0.77× / 0.91×
get, time to first byte 64KiB 900 1.68 / 2.23 / 2.56 / 1.73 3.41 / 4.52 / 5.32 / 3.52 1.17 / 1.69 / 2.00 / 1.23 1.86 / 2.48 / 2.82 / 1.91 2.03× / 0.70× / 1.11×
get, full read 64KiB 900 2.25 / 2.79 / 2.98 / 2.26 4.73 / 5.85 / 6.61 / 4.80 1.85 / 2.29 / 2.54 / 1.86 2.47 / 3.08 / 3.49 / 2.52 2.11× / 0.82× / 1.10×
delete 64KiB 900 2.00 / 2.63 / 2.91 / 2.06 3.60 / 4.64 / 5.36 / 3.69 1.24 / 1.74 / 2.06 / 1.30 1.49 / 2.08 / 2.38 / 1.55 1.80× / 0.62× / 0.74×
put (known size) 1MiB 600 22.00 / 27.33 / 31.57 / 22.48 18.69 / 24.00 / 26.18 / 19.09 10.17 / 12.70 / 13.93 / 10.29 16.58 / 19.39 / 21.80 / 17.83 0.85× / 0.46× / 0.75×
stat (HEAD) 1MiB 600 1.95 / 2.50 / 2.96 / 1.98 3.92 / 4.74 / 5.23 / 3.94 1.47 / 1.92 / 2.30 / 1.51 1.91 / 2.59 / 3.49 / 1.98 2.01× / 0.75× / 0.98×
get, time to first byte 1MiB 600 2.41 / 3.32 / 4.49 / 2.47 3.39 / 4.35 / 5.06 / 3.46 1.19 / 1.83 / 2.58 / 1.28 2.52 / 3.39 / 4.25 / 2.62 1.41× / 0.49× / 1.05×
get, full read 1MiB 600 9.85 / 11.80 / 12.62 / 10.00 16.61 / 21.84 / 24.87 / 17.01 8.46 / 10.23 / 11.44 / 8.62 9.19 / 11.69 / 13.28 / 9.50 1.69× / 0.86× / 0.93×
delete 1MiB 600 2.32 / 2.99 / 3.32 / 2.35 4.26 / 5.23 / 5.82 / 4.31 1.82 / 2.39 / 2.86 / 1.87 1.92 / 2.64 / 3.05 / 1.96 1.84× / 0.79× / 0.83×
put (known size) 16MiB 90 188.33 / 260.25 / 417.73 / 200.77 185.65 / 203.08 / 206.83 / 186.10 138.33 / 160.81 / 176.95 / 141.16 158.95 / 323.38 / 1137.04 / 198.17 0.99× / 0.73× / 0.84×
stat (HEAD) 16MiB 90 1.93 / 2.83 / 3.21 / 2.03 4.11 / 4.75 / 5.11 / 4.15 1.81 / 2.40 / 2.58 / 1.86 1.72 / 2.31 / 2.55 / 1.78 2.13× / 0.94× / 0.89×
get, time to first byte 16MiB 90 2.23 / 3.26 / 3.98 / 2.36 3.46 / 4.34 / 4.66 / 3.53 1.34 / 2.53 / 3.39 / 1.52 4.30 / 5.61 / 6.46 / 4.44 1.55× / 0.60× / 1.93×
get, full read 16MiB 90 127.06 / 144.36 / 152.21 / 128.87 139.87 / 172.27 / 190.53 / 142.35 114.96 / 130.45 / 133.24 / 115.59 114.42 / 130.47 / 136.64 / 115.09 1.10× / 0.90× / 0.90×
delete 16MiB 90 2.90 / 3.33 / 3.42 / 2.91 9.47 / 11.14 / 13.33 / 9.66 7.38 / 9.73 / 11.29 / 7.58 2.41 / 3.06 / 13.72 / 3.50 3.27× / 2.55× / 0.83×
put streamed (length=-1, 5 MiB parts) 64KiB 900 5.51 / 8.04 / 10.52 / 6.03 5.15 / 6.28 / 6.78 / 5.22 2.50 / 3.11 / 3.61 / 2.50 3.80 / 4.75 / 5.46 / 3.97 0.93× / 0.45× / 0.69×
put streamed (length=-1, 5 MiB parts) 1MiB 600 22.26 / 26.95 / 33.09 / 22.81 18.97 / 24.56 / 26.97 / 19.37 9.93 / 12.09 / 13.28 / 10.03 16.35 / 18.57 / 19.83 / 17.08 0.85× / 0.45× / 0.73×
put streamed (length=-1, 5 MiB parts) 16MiB 90 188.20 / 209.18 / 342.78 / 193.00 191.50 / 213.30 / 218.37 / 191.66 141.83 / 152.27 / 164.08 / 141.82 159.07 / 336.81 / 1181.84 / 200.04 1.02× / 0.75× / 0.85×
copy_object (HEAD + copy) 64KiB 300 6.76 / 9.10 / 13.88 / 7.10 7.78 / 9.36 / 10.11 / 7.87 3.13 / 4.13 / 4.51 / 3.23 5.04 / 5.98 / 6.68 / 5.14 1.15× / 0.46× / 0.75×
copy_object (HEAD + copy) 1MiB 300 16.75 / 21.45 / 26.16 / 17.17 10.85 / 13.56 / 15.54 / 11.13 5.68 / 7.14 / 7.86 / 5.84 8.03 / 9.96 / 11.12 / 8.10 0.65× / 0.34× / 0.48×
list_objects prefix, 1000 keys - 90 104.86 / 128.21 / 142.91 / 108.05 117.50 / 135.64 / 146.92 / 118.93 109.85 / 134.28 / 144.52 / 112.86 89.94 / 105.65 / 124.42 / 90.88 1.12× / 1.05× / 0.86×
put, 8 concurrent 64KiB 1440 14.84 / 25.75 / 38.92 / 17.05 12.37 / 17.58 / 21.14 / 12.64 7.48 / 12.65 / 15.74 / 7.86 8.66 / 14.67 / 17.70 / 9.15 0.83× / 0.50× / 0.58×
get full, 8 concurrent 64KiB 1440 6.75 / 12.36 / 15.52 / 7.23 9.64 / 14.11 / 16.60 / 10.02 6.08 / 12.02 / 15.74 / 6.63 6.78 / 12.50 / 16.53 / 7.32 1.43× / 0.90× / 1.01×

Concurrency 8, 64 KiB: aggregate throughput in ops/s, one value per round

MinIO versitygw (default) versitygw (keep-alive) SeaweedFS
PUT 459, 496, 411 609, 628, 594 947, 930, 1026 841, 860, 821
GET 1092, 1055, 1010 816, 714, 768 1194, 1089, 1196 1070, 1134, 951

warp mixed, 64 KiB, 8 concurrent clients, 60 s. Each cell gives round 1 / round 2. The client is minio-go with keep-alive.

Op (64 KiB, 8 clients, 60 s) MinIO versitygw (default) versitygw (keep-alive) SeaweedFS
GET 800 / 799 obj/s; p50 2.8 / 2.7 ms; p99 8.4 / 7.1 ms 575 / 626 obj/s; p50 6.5 / 7.8 ms; p99 14.8 / 11.6 ms 1223 / 1224 obj/s; p50 3.1 / 3.1 ms; p99 8.0 / 7.5 ms 903 / 882 obj/s; p50 4.1 / 4.2 ms; p99 9.7 / 9.5 ms
PUT 267 / 266 obj/s; p50 11.2 / 12.0 ms; p99 49.5 / 47.7 ms 192 / 209 obj/s; p50 6.7 / 8.8 ms; p99 14.4 / 12.8 ms 408 / 408 obj/s; p50 3.3 / 3.3 ms; p99 8.8 / 7.7 ms 301 / 294 obj/s; p50 5.9 / 6.0 ms; p99 13.7 / 12.9 ms
STAT 534 / 533 obj/s; p50 1.8 / 1.8 ms; p99 5.6 / 5.9 ms 383 / 417 obj/s; p50 4.3 / 5.6 ms; p99 12.4 / 9.3 ms 816 / 816 obj/s; p50 2.2 / 2.2 ms; p99 6.3 / 6.8 ms 601 / 588 obj/s; p50 2.4 / 2.4 ms; p99 7.0 / 6.7 ms
DELETE 178 / 178 obj/s; p50 3.4 / 3.7 ms; p99 18.9 / 21.5 ms 128 / 140 obj/s; p50 4.4 / 4.2 ms; p99 12.0 / 8.4 ms 272 / 272 obj/s; p50 2.2 / 2.2 ms; p99 7.0 / 7.2 ms 200 / 196 obj/s; p50 2.5 / 2.5 ms; p99 7.4 / 6.9 ms
Total 1779 / 1776 obj/s 1278 / 1391 obj/s 2718 / 2720 obj/s 2005 / 1960 obj/s
Run-to-run spread of the p50 (min–max over the 3 rounds, ms)
Operation Size MinIO versitygw (default) versitygw (keep-alive) SeaweedFS
bucket_exists+make_bucket (new client) - 8.59–9.79 7.13–7.87 4.37–5.29 5.99–6.34
put (known size) 1KiB 4.35–4.62 3.81–4.00 1.52–1.60 2.37–2.40
stat (HEAD) 1KiB 1.52–1.65 3.53–3.63 1.27–1.30 1.47–1.49
get, time to first byte 1KiB 1.53–1.68 3.28–3.52 1.21–1.25 1.63–1.64
get, full read 1KiB 1.56–1.72 3.41–3.64 1.26–1.29 1.66–1.68
delete 1KiB 1.85–1.97 3.42–3.64 1.20–1.22 1.44–1.47
put (known size) 4KiB 4.28–4.65 3.84–3.96 1.58–1.72 2.30–2.67
stat (HEAD) 4KiB 1.53–1.65 3.51–3.68 1.24–1.35 1.43–1.59
get, time to first byte 4KiB 1.55–1.62 3.29–3.48 1.21–1.30 1.62–1.85
get, full read 4KiB 1.60–1.69 3.42–3.60 1.26–1.34 1.70–1.92
delete 4KiB 1.86–1.96 3.44–3.55 1.17–1.27 1.40–1.55
put (known size) 64KiB 5.40–5.72 5.14–5.36 2.50–2.54 3.37–3.74
stat (HEAD) 64KiB 1.62–1.75 3.50–3.77 1.28–1.32 1.45–1.60
get, time to first byte 64KiB 1.65–1.73 3.27–3.54 1.16–1.18 1.75–1.92
get, full read 64KiB 2.19–2.31 4.56–4.84 1.82–1.87 2.30–2.60
delete 64KiB 1.93–2.07 3.41–3.76 1.24–1.24 1.41–1.54
put (known size) 1MiB 21.56–22.95 18.53–18.94 9.54–10.59 16.47–16.76
stat (HEAD) 1MiB 1.91–2.03 3.82–3.97 1.42–1.51 1.87–1.95
get, time to first byte 1MiB 2.33–2.46 3.36–3.42 1.18–1.22 2.48–2.58
get, full read 1MiB 9.71–10.02 16.35–17.04 8.22–8.69 9.04–9.42
delete 1MiB 2.24–2.41 4.20–4.33 1.79–1.91 1.87–1.98
put (known size) 16MiB 184.65–200.75 180.02–191.88 136.23–141.89 155.32–160.92
stat (HEAD) 16MiB 1.83–2.09 4.06–4.17 1.73–1.97 1.64–1.75
get, time to first byte 16MiB 2.18–2.32 3.35–3.49 1.31–1.44 3.95–4.45
get, full read 16MiB 123.44–133.92 138.33–140.99 109.24–119.76 113.69–115.16
delete 16MiB 2.81–2.99 9.37–9.70 7.09–7.67 2.35–2.47
put streamed (length=-1, 5 MiB parts) 64KiB 5.45–5.54 5.04–5.24 2.26–2.57 3.56–3.98
put streamed (length=-1, 5 MiB parts) 1MiB 21.93–22.88 18.58–19.18 9.86–10.06 15.93–16.77
put streamed (length=-1, 5 MiB parts) 16MiB 179.54–193.98 189.82–194.00 141.42–142.39 155.74–161.62
copy_object (HEAD + copy) 64KiB 6.54–7.01 7.71–7.85 3.01–3.29 4.77–5.29
copy_object (HEAD + copy) 1MiB 16.67–16.87 10.40–11.27 5.50–6.01 7.71–8.25
list_objects prefix, 1000 keys - 100.24–111.15 114.58–121.02 108.09–114.89 86.69–92.98
put, 8 concurrent 64KiB 14.52–15.12 12.18–12.59 7.38–7.64 8.45–8.76
get full, 8 concurrent 64KiB 6.58–7.23 9.05–10.28 5.89–6.35 6.27–7.46

Resources

MinIO versitygw (default) versitygw (keep-alive) SeaweedFS
Idle working set (kubectl top) 180–400 Mi 12–21 Mi 21 Mi 625–1019 Mi
Idle anonymous memory, from cgroup memory.stat (the rest is page cache) 121–230 MiB 10–13 MiB 13 MiB 486–519 MiB
CPU under warp load, median / max 1606 / 1670 m 705 / 901 m 1228 / 1309 m 1652 / 1705 m
Working set under warp load, median / max 523 / 612 Mi 34 / 48 Mi 35 / 67 Mi 615 / 918 Mi
Server CPU per request under warp (CPU ÷ obj/s) ≈0.90 ms ≈0.53 ms ≈0.45 ms ≈0.83 ms

Caveats

  • Shared cluster: dev1 is in use, with live Ryax pods on both nodes. Interleaving the rounds keeps that noise visible, and the per-round p50 spread is small (see the collapsed table). Everything ran on one day.
  • Small, fresh stores: the stores were nearly empty. Behaviour at millions of objects was not measured, neither SeaweedFS's in-memory index nor versitygw's directory sizes.
  • Warm reads only: each GET follows its PUT, so the server reads from page cache. Cold-read latency was not measured.
  • Client cost is included: minio-py hashes every PUT body with SHA-256 over plain HTTP. That client CPU is part of the PUT numbers, as it is in Ryax, and is the same for every server.
  • 16 MiB is bandwidth-bound: 16 MiB PUT and GET are dominated by network and client throughput, about 100 to 140 MB/s.
  • warp latencies are measured at different loads: warp is closed-loop, so each server is measured at its own saturation point. Under warp, the keep-alive versitygw's GET and STAT p50 are slightly higher than MinIO's (3.1 against 2.8 ms, and 2.2 against 1.8 ms), while it serves 1.53× MinIO's throughput. Its PUT and DELETE p50 are lower: 3.3 against 11.2 ms, and 2.2 against 3.4 ms. Ryax's request rate is far below saturation, which is what the single-client minio-py tables measure.
  • Resource figures are approximate: kubectl top has a resolution of about 15 s, and its working set includes page cache.
  • Not the live MinIO: the MinIO under test is a fresh pod with the chart's env, not the ryaxns instance. The chart sets no CPU limit; the bench sets 2 CPU for all servers.
  • versitygw large deletes: deleting 16 MiB is slower (p50 7.4 ms, against 2.9 ms for MinIO), while 1 MiB is faster (1.8 against 2.3 ms). This was not investigated; freeing the extents on ext4 is a probable cause. Ryax deletes only in the runner GC.
  • SeaweedFS 16 MiB PUT stalls: p95 is about 330 ms and p99 about 1.1 s, against a p50 of 159 ms. Not investigated; weed mini auto-sized its volumes to 256 MiB. Also:
    • weed mini starts extra components even with -admin.ui=false: an Iceberg catalog, a Lance namespace, a plugin worker and an admin server, on extra ports (9333, 9340, 8888, 8181, 9101).
    • The image enables Go FIPS mode by default.
    • Telemetry is on unless -master.telemetry=false is passed.
  • versitygw lists the volume root as buckets: with versitygw on the volume root, lost+found and .vgwlocks sit next to the buckets, so lost+found would show up in ListBuckets. Ryax never calls ListBuckets, but the subchart should still point versitygw at a subdirectory of the PV, for example /data/s3.

Conclusion

  1. versitygw (POSIX, xattrs) is no slower than MinIO for Ryax's pattern, provided VGW_KEEP_ALIVE=true is set. p50 against MinIO:

    • 1 KiB: PUT 1.56 against 4.47 ms, GET 1.27 against 1.65 ms, HEAD 1.29 against 1.60 ms, DELETE 1.21 against 1.90 ms;
    • 1 MiB: PUT 10.2 against 22.0 ms, GET 8.5 against 9.9 ms;
    • 16 MiB: PUT 138 against 188 ms, GET 115 against 127 ms;
    • copy_object: 2 to 3× faster;
    • list over 1000 keys: 110 against 105 ms, within the noise.

    The p99 values are lower too, or within 2 % for list and the 8-concurrent GET, except for the 16 MiB delete. The earlier third-party result of slower GETs does not reproduce here once keep-alive is on.

  2. With the default --keep-alive=false, versitygw fails the gate: about +1.8 ms on every HEAD, GET and DELETE, and −25 % under warp. Pin VGW_KEEP_ALIVE=true, plus VGW_QUIET=true, in the subchart, and add a helm-unittest assertion for it.

  3. SeaweedFS beats MinIO on PUT and listing and matches it on GET, but it is slower than tuned versitygw almost everywhere, has 16 MiB PUT tail spikes, and uses about 40× versitygw's memory. The benchmark gives no reason to switch the recommendation to it.

  4. No compatibility blocker on any candidate: the ryax user with a 12-character secret, region us-east-1 with GetBucketLocation, NoSuchKey mapping, multipart with 5 and 8 MiB parts, abort and CopyObject all work.

  5. Open question 2, partly answered: sbs-default ext4 supports user xattrs, so no sidecar is needed on Scaleway block storage. NFS and Velero remain open.

Garage benchmark on dev1 (2026-10-05)

This is a second run of the same suite, made the same day: Garage against the MinIO baseline and the tuned versitygw (VGW_KEEP_ALIVE=true, VGW_QUIET=true). The MinIO baseline reproduces the first run closely, for example 1 KiB PUT p50 4.57 against 4.47 ms and 64 KiB GET 2.25 against 2.25 ms.

Verdict: Garage does not pass the gate, and versitygw stays the recommendation.

  • Faster than MinIO for single-client writes and metadata: PUT, HEAD, DELETE, copy and time to first byte.
  • Slower than MinIO where Ryax reads files back. That is a full GET of 4 KiB to 1 MiB objects, the inputs of every workflow step: 1.2 to 1.5× MinIO, +0.5 to +2.1 ms at p50.
  • Slower than MinIO under concurrency:
    • with SQLite, an 8-client PUT is 1.6× MinIO;
    • warp mixed reaches only about half of MinIO's throughput, because Garage is CPU-bound at its 2-CPU limit, about 2 ms of server CPU per request.
  • Tuned versitygw beats Garage on every operation except three: 1 MiB copy_object, 16 MiB delete and listing.
  • Garage also needs new credentials and a region setting (see Compatibility).

Setup (differences from the first run)

  • Placement: the same nodes and storage class. The servers ran on scw-dev1-default-2ceee72329154e3c8b7a89c1cbf9a, where ryax-minio runs, and the client on scw-dev1-default-0801e8ba7cd84f449fcdfcfd1b769, where ryax-runner runs. Every PVC was 20Gi on sbs-default. The run used a fresh throw-away namespace s3-bench, deleted afterwards.

  • Resources: limits 2 CPU and 2Gi, requests 500m CPU and 512Mi for every server. 512Mi was needed to fit four servers on the node; requests only affect scheduling.

  • MinIO: docker.io/bitnamilegacy/minio:2025.7.23-debian-12-r3, with the same env as in the first run.

  • versitygw: docker.io/versity/versitygw:v1.8.0, POSIX backend with xattrs, VGW_KEEP_ALIVE=true and VGW_QUIET=true.

  • Garage: docker.io/dxflrs/garage:v2.4.1, the latest v2.4.x, run as /garage server --single-node --default-access-key. It ran as UID 1200 with fsGroup 1200. The config file:

    metadata_dir = "/var/lib/garage/meta"   # PVC garage-meta
data_dir = "/var/lib/garage/data"       # PVC garage-data
db_engine = "sqlite"                    # "lmdb" for the LMDB variant
replication_factor = 1
rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
[s3_api]
s3_region = "us-east-1"
api_bind_addr = "[::]:9000"
    • Meta and data are on two separate PVCs for each Garage server: garage-meta and garage-data, plus garage-lmdb-meta and garage-lmdb-data.
    • GARAGE_RPC_SECRET (32 random bytes, hex) and GARAGE_DEFAULT_ACCESS_KEY / GARAGE_DEFAULT_SECRET_KEY come from a Secret.
    • --default-access-key gives the key allow_create_bucket, so the Ryax-style make_bucket at startup works. GARAGE_DEFAULT_BUCKET was not used, because the Ryax services create their bucket themselves.
    • Garage defaults were kept: metadata_fsync = false (SQLite runs with PRAGMA synchronous = OFF, in WAL mode), data_fsync = false, zstd compression_level = 1 and block_size 1 MiB.

  • Variants:

    • Garage (SQLite) is the main configuration;
    • Garage (LMDB) ran in all 3 rounds, not just one, because there was time;
    • Garage SQLite without compression got 2 extra rounds and 1 warp round after the main runs, on the same PVCs.

  • Client and method: unchanged from the first run: minio-py 7.2.20 built as S3Engine.connect(), the same operations, sizes, iterations and warm-up. There were 3 interleaved rounds (MinIO → versitygw → Garage SQLite → Garage LMDB, three times over) after a discarded smoke round, then warp mixed with 2 interleaved rounds.

Compatibility of Garage

Check Result
Current credentials (ryax with a 12-character secret) through GARAGE_DEFAULT_ACCESS_KEY / GARAGE_DEFAULT_SECRET_KEY ❌ The server refuses to start (CrashLoopBackOff): Error: Invalid default access key: Key identifiers should be at least 8 characters long
8-character ID ryaxryax with the same 12-character secret ❌ Error: Invalid default access key: Secret keys should be at least 16 characters long
The same two keys through garage key import --yes ❌ ImportKey returned InvalidRequest (400): Bad request: Invalid key format: Key identifiers should be at least 8 characters long, then the same error for Secret keys should be at least 16 characters long
Compliant credentials: key ID ryax-filestore (14 characters) and a 32-character alphanumeric secret ✅
GetBucketLocation returns us-east-1, and requests signed with it are accepted. This needs s3_region = "us-east-1"; the default "garage" was not tried. ✅
make_bucket without a region, then bucket_exists, remove_bucket ✅
Missing key on stat or get returns NoSuchKey ✅
A wrong secret is rejected ✅ with AccessDenied where MinIO and versitygw say SignatureDoesNotMatch. Ryax does not depend on that code.
Single-PUT ETag equals the MD5 ✅
Streamed 16 MiB multipart with 5 MiB parts (ETag -4) and 8 MiB parts (ETag -2), byte-exact round trip ✅
Multipart aborted on a failing stream, with no pending upload left ✅
16 MiB copy_object, byte-exact; the prefix listing returns exactly 1000 keys ✅
Connection keep-alive ✅ by default

Client-side changes: no code change was needed in minio-py or in the way S3Engine builds its client. Only the credential values differ. For Ryax, that means new filestore-access / filestore-secret (root-user / root-password) values in ryax-minio-secret on every install, GitOps ones included, plus a new rpc_secret to manage. Two more operational points:

  • No shell in the image: it is built FROM scratch, so kubectl exec can only run /garage.
  • Default keys are checked on every start: if the key exists with a different secret, the server refuses to start.

Results: minio-py latency in ms (p50 / p95 / p99 / mean, 3 rounds pooled)

Operation Size n/server MinIO p50 / p95 / p99 / mean versitygw (keep-alive) p50 / p95 / p99 / mean Garage (SQLite) p50 / p95 / p99 / mean Garage (LMDB) p50 / p95 / p99 / mean p50 vs MinIO: vgw-ka / Garage SQLite / Garage LMDB
bucket_exists+make_bucket (new client) - 60 8.82 / 11.41 / 209.77 / 17.16 4.89 / 5.78 / 5.80 / 4.80 7.61 / 9.07 / 9.61 / 7.69 6.35 / 7.87 / 8.35 / 6.47 0.55× / 0.86× / 0.72×
put (known size) 1KiB 900 4.57 / 6.17 / 9.34 / 4.80 1.52 / 2.07 / 2.67 / 1.58 1.78 / 2.52 / 3.29 / 1.87 1.53 / 2.13 / 2.86 / 1.61 0.33× / 0.39× / 0.33×
stat (HEAD) 1KiB 900 1.61 / 2.19 / 2.53 / 1.68 1.24 / 1.73 / 2.14 / 1.29 1.37 / 1.93 / 2.28 / 1.42 1.27 / 1.78 / 2.18 / 1.32 0.77× / 0.85× / 0.79×
get, time to first byte 1KiB 900 1.64 / 2.20 / 2.67 / 1.71 1.20 / 1.62 / 1.93 / 1.24 1.25 / 1.73 / 2.23 / 1.31 1.17 / 1.55 / 2.10 / 1.22 0.73× / 0.76× / 0.71×
get, full read 1KiB 900 1.68 / 2.26 / 2.72 / 1.75 1.23 / 1.68 / 1.98 / 1.28 1.29 / 1.80 / 2.29 / 1.34 1.21 / 1.60 / 2.17 / 1.25 0.73× / 0.76× / 0.72×
delete 1KiB 900 1.96 / 2.58 / 2.95 / 2.01 1.18 / 1.63 / 1.95 / 1.23 1.53 / 2.19 / 2.77 / 1.60 1.33 / 1.79 / 2.25 / 1.38 0.60× / 0.78× / 0.68×
put (known size) 4KiB 900 4.60 / 6.25 / 9.33 / 4.82 1.55 / 2.06 / 2.52 / 1.60 4.05 / 5.58 / 6.49 / 4.16 2.70 / 3.75 / 5.14 / 2.81 0.34× / 0.88× / 0.59×
stat (HEAD) 4KiB 900 1.65 / 2.33 / 2.65 / 1.73 1.25 / 1.74 / 2.07 / 1.30 1.50 / 2.11 / 2.56 / 1.57 1.33 / 1.95 / 2.35 / 1.39 0.76× / 0.91× / 0.80×
get, time to first byte 4KiB 900 1.65 / 2.23 / 2.58 / 1.71 1.21 / 1.67 / 2.01 / 1.25 1.32 / 1.76 / 2.29 / 1.37 1.21 / 1.70 / 2.02 / 1.27 0.73× / 0.80× / 0.73×
get, full read 4KiB 900 1.71 / 2.30 / 2.66 / 1.77 1.27 / 1.75 / 2.14 / 1.32 2.21 / 2.92 / 3.81 / 2.31 2.16 / 3.08 / 4.26 / 2.28 0.74× / 1.29× / 1.27×
delete 4KiB 900 1.96 / 2.65 / 3.19 / 2.03 1.18 / 1.58 / 1.96 / 1.23 1.59 / 2.20 / 2.69 / 1.66 1.39 / 2.09 / 2.88 / 1.49 0.60× / 0.81× / 0.71×
put (known size) 64KiB 900 5.48 / 7.44 / 10.12 / 5.75 2.34 / 3.00 / 3.45 / 2.38 4.93 / 6.55 / 7.52 / 5.07 3.77 / 4.66 / 5.35 / 3.83 0.43× / 0.90× / 0.69×
stat (HEAD) 64KiB 900 1.66 / 2.41 / 3.13 / 1.74 1.24 / 1.70 / 1.97 / 1.29 1.58 / 2.15 / 2.63 / 1.64 1.36 / 1.93 / 2.21 / 1.42 0.75× / 0.95× / 0.82×
get, time to first byte 64KiB 900 1.65 / 2.41 / 2.85 / 1.74 1.14 / 1.52 / 1.80 / 1.18 1.35 / 1.79 / 2.18 / 1.39 1.22 / 1.68 / 2.00 / 1.27 0.69× / 0.82× / 0.74×
get, full read 64KiB 900 2.25 / 2.93 / 3.33 / 2.30 1.84 / 2.24 / 2.61 / 1.85 3.32 / 4.16 / 4.98 / 3.39 3.21 / 4.12 / 5.19 / 3.29 0.82× / 1.47× / 1.43×
delete 64KiB 900 1.98 / 2.81 / 3.64 / 2.08 1.21 / 1.72 / 2.17 / 1.27 1.65 / 2.37 / 2.82 / 1.74 1.44 / 2.12 / 2.59 / 1.51 0.61× / 0.83× / 0.73×
put (known size) 1MiB 600 22.06 / 26.26 / 28.43 / 22.33 9.96 / 12.15 / 13.26 / 10.03 16.15 / 19.37 / 21.10 / 16.38 14.98 / 17.57 / 18.85 / 15.14 0.45× / 0.73× / 0.68×
stat (HEAD) 1MiB 600 1.92 / 2.54 / 2.80 / 1.97 1.42 / 1.98 / 2.39 / 1.47 1.75 / 2.31 / 2.69 / 1.78 1.58 / 2.09 / 2.36 / 1.62 0.74× / 0.91× / 0.82×
get, time to first byte 1MiB 600 2.41 / 3.19 / 4.12 / 2.45 1.18 / 1.78 / 2.66 / 1.27 1.33 / 1.82 / 2.10 / 1.39 1.23 / 1.62 / 1.90 / 1.27 0.49× / 0.55× / 0.51×
get, full read 1MiB 600 9.81 / 12.07 / 13.29 / 10.00 8.13 / 10.29 / 11.40 / 8.37 11.87 / 14.28 / 15.73 / 12.07 11.82 / 14.44 / 15.97 / 12.09 0.83× / 1.21× / 1.20×
delete 1MiB 600 2.30 / 3.03 / 3.61 / 2.36 1.77 / 2.46 / 2.81 / 1.81 1.98 / 2.76 / 3.24 / 2.05 1.75 / 2.28 / 2.88 / 2.12 0.77× / 0.86× / 0.76×
put (known size) 16MiB 90 186.98 / 227.42 / 303.79 / 193.59 140.32 / 160.49 / 182.16 / 141.69 147.65 / 166.70 / 182.99 / 147.32 141.20 / 159.96 / 180.13 / 141.84 0.75× / 0.79× / 0.76×
stat (HEAD) 16MiB 90 1.86 / 2.31 / 2.53 / 1.89 1.69 / 2.28 / 2.71 / 1.77 1.83 / 2.98 / 7.06 / 2.09 1.75 / 2.46 / 2.64 / 1.83 0.91× / 0.98× / 0.94×
get, time to first byte 16MiB 90 2.19 / 5.33 / 7.09 / 2.52 1.32 / 1.99 / 2.95 / 1.39 1.53 / 2.25 / 4.57 / 1.81 1.47 / 2.18 / 3.55 / 1.58 0.60× / 0.70× / 0.67×
get, full read 16MiB 90 127.11 / 141.17 / 150.06 / 128.13 113.59 / 129.55 / 134.03 / 113.68 116.07 / 133.30 / 146.16 / 116.68 114.17 / 127.99 / 136.26 / 115.34 0.89× / 0.91× / 0.90×
delete 16MiB 90 2.78 / 3.29 / 3.78 / 2.79 7.41 / 8.96 / 9.60 / 7.44 2.34 / 3.00 / 3.34 / 2.44 2.08 / 2.49 / 3.08 / 2.13 2.67× / 0.84× / 0.75×
put streamed (length=-1, 5 MiB parts) 64KiB 900 5.40 / 7.72 / 10.42 / 5.79 2.60 / 3.26 / 3.67 / 2.65 5.11 / 6.99 / 8.75 / 5.31 3.67 / 4.55 / 5.46 / 3.74 0.48× / 0.95× / 0.68×
put streamed (length=-1, 5 MiB parts) 1MiB 600 22.64 / 26.75 / 30.19 / 23.87 9.75 / 11.90 / 12.88 / 9.82 16.18 / 18.99 / 20.96 / 16.31 15.34 / 18.28 / 19.30 / 15.52 0.43× / 0.71× / 0.68×
put streamed (length=-1, 5 MiB parts) 16MiB 90 186.59 / 205.38 / 246.41 / 188.63 142.60 / 160.47 / 179.86 / 145.17 153.62 / 181.29 / 198.27 / 157.17 144.93 / 162.58 / 164.71 / 145.69 0.76× / 0.82× / 0.78×
copy_object (HEAD + copy) 64KiB 300 7.08 / 10.21 / 13.29 / 7.97 3.06 / 3.78 / 4.11 / 3.12 4.71 / 5.98 / 7.36 / 4.77 3.39 / 4.40 / 4.66 / 3.48 0.43× / 0.67× / 0.48×
copy_object (HEAD + copy) 1MiB 300 16.97 / 21.32 / 28.07 / 17.51 6.06 / 7.51 / 8.42 / 6.17 4.76 / 6.12 / 7.03 / 4.85 3.40 / 4.25 / 4.78 / 3.44 0.36× / 0.28× / 0.20×
list_objects prefix, 1000 keys - 90 103.67 / 123.91 / 138.74 / 105.86 107.73 / 130.62 / 137.01 / 110.31 102.75 / 123.75 / 133.89 / 105.09 99.37 / 113.78 / 136.42 / 101.26 1.04× / 0.99× / 0.96×
put, 8 concurrent 64KiB 1440 12.00 / 22.81 / 90.34 / 14.93 7.90 / 13.86 / 17.66 / 8.39 19.17 / 31.43 / 37.57 / 19.92 11.76 / 20.90 / 25.64 / 12.55 0.66× / 1.60× / 0.98×
get full, 8 concurrent 64KiB 1440 6.81 / 13.44 / 17.71 / 7.45 7.43 / 14.64 / 19.29 / 8.03 8.22 / 12.69 / 16.09 / 8.54 8.63 / 13.32 / 15.55 / 8.99 1.09× / 1.21× / 1.27×

Concurrency 8, 64 KiB: aggregate throughput in ops/s, one value per round

MinIO versitygw (keep-alive) Garage (SQLite) Garage (LMDB)
PUT 692, 638, 368 901, 978, 875 395, 393, 397 565, 645, 661
GET 1103, 936, 1050 960, 935, 937 847, 915, 949 865, 895, 827

warp mixed, 64 KiB, 8 concurrent clients, 60 s. Each cell gives round 1 / round 2.

Op (64 KiB, 8 clients, 60 s) MinIO versitygw (keep-alive) Garage (SQLite) Garage (LMDB)
GET 845 / 834 obj/s; p50 2.7 / 3.3 ms; p99 8.0 / 7.7 ms 1242 / 1237 obj/s; p50 3.0 / 3.0 ms; p99 94.7 / 9.4 ms 393 / 388 obj/s; p50 12.6 / 12.5 ms; p99 24.6 / 25.3 ms 431 / 461 obj/s; p50 11.4 / 10.9 ms; p99 21.2 / 20.4 ms
PUT 282 / 278 obj/s; p50 11.0 / 12.6 ms; p99 49.6 / 45.5 ms 414 / 412 obj/s; p50 3.2 / 3.3 ms; p99 10.3 / 10.3 ms 131 / 129 obj/s; p50 10.2 / 10.7 ms; p99 21.0 / 22.8 ms 144 / 154 obj/s; p50 9.1 / 8.6 ms; p99 17.0 / 17.6 ms
STAT 564 / 556 obj/s; p50 1.7 / 2.5 ms; p99 6.1 / 5.6 ms 828 / 824 obj/s; p50 2.0 / 2.2 ms; p99 5.6 / 5.6 ms 262 / 258 obj/s; p50 3.7 / 3.5 ms; p99 11.1 / 11.0 ms 287 / 307 obj/s; p50 3.3 / 3.1 ms; p99 9.3 / 8.7 ms
DELETE 188 / 185 obj/s; p50 3.6 / 5.0 ms; p99 16.0 / 15.9 ms 276 / 275 obj/s; p50 2.0 / 2.2 ms; p99 8.9 / 5.4 ms 87 / 86 obj/s; p50 4.4 / 4.3 ms; p99 12.3 / 12.5 ms 96 / 102 obj/s; p50 3.7 / 3.4 ms; p99 9.1 / 9.4 ms
Total 1879 / 1852 obj/s 2759 / 2748 obj/s 874 / 861 obj/s 958 / 1025 obj/s

Does zstd explain Garage's CPU cost? Mostly not. p50 in ms; the no-compression column comes from 2 rounds and 1 warp run.

Operation Size MinIO Garage SQLite, zstd level 1 (default) Garage SQLite, compression_level = "none"
put 1KiB 4.57 1.78 2.14
put 4KiB 4.60 4.05 5.01
get, full read 4KiB 1.71 2.21 2.16
put 64KiB 5.48 4.93 5.20
get, full read 64KiB 2.25 3.32 2.98
put 1MiB 22.06 16.15 13.62
get, full read 1MiB 9.81 11.87 12.11
put 16MiB 186.98 147.65 131.96
get, full read 16MiB 127.11 116.07 118.83
put, 8 concurrent 64KiB 12.00 19.17 19.45
get full, 8 concurrent 64KiB 6.81 8.22 7.46
warp mixed total (obj/s) 64KiB 1879 / 1852 874 / 861 991
Run-to-run spread of the p50 (min–max over the 3 rounds, ms)
Operation Size MinIO versitygw (keep-alive) Garage (SQLite) Garage (LMDB)
bucket_exists+make_bucket (new client) - 8.72–9.06 4.39–5.07 7.26–7.93 6.13–6.74
put (known size) 1KiB 4.51–4.69 1.47–1.56 1.68–1.84 1.44–1.60
stat (HEAD) 1KiB 1.57–1.68 1.19–1.28 1.31–1.41 1.24–1.31
get, time to first byte 1KiB 1.62–1.67 1.15–1.24 1.23–1.27 1.15–1.21
get, full read 1KiB 1.65–1.71 1.18–1.28 1.25–1.30 1.18–1.25
delete 1KiB 1.92–1.99 1.14–1.21 1.46–1.56 1.27–1.37
put (known size) 4KiB 4.41–4.79 1.54–1.58 3.88–4.12 2.58–2.84
stat (HEAD) 4KiB 1.58–1.78 1.23–1.29 1.44–1.56 1.26–1.42
get, time to first byte 4KiB 1.59–1.75 1.20–1.21 1.28–1.36 1.16–1.30
get, full read 4KiB 1.64–1.81 1.26–1.29 2.13–2.30 2.04–2.31
delete 4KiB 1.89–2.13 1.17–1.20 1.49–1.67 1.35–1.49
put (known size) 64KiB 5.34–5.78 2.18–2.45 4.78–5.13 3.64–3.89
stat (HEAD) 64KiB 1.55–1.75 1.22–1.27 1.53–1.65 1.32–1.45
get, time to first byte 64KiB 1.56–1.82 1.13–1.16 1.30–1.41 1.18–1.28
get, full read 64KiB 2.19–2.29 1.81–1.89 3.29–3.37 3.17–3.26
delete 64KiB 1.88–2.10 1.20–1.25 1.61–1.72 1.39–1.53
put (known size) 1MiB 21.55–22.62 9.54–10.34 15.63–16.59 14.79–15.22
stat (HEAD) 1MiB 1.87–2.00 1.29–1.52 1.71–1.78 1.55–1.62
get, time to first byte 1MiB 2.28–2.46 1.12–1.25 1.29–1.42 1.21–1.26
get, full read 1MiB 9.77–9.92 7.85–8.40 11.69–12.25 11.58–11.96
delete 1MiB 2.25–2.41 1.62–1.92 1.95–2.02 1.65–1.78
put (known size) 16MiB 186.10–191.68 136.02–141.49 144.11–153.70 134.44–147.06
stat (HEAD) 16MiB 1.81–1.94 1.64–1.74 1.65–1.96 1.65–1.99
get, time to first byte 16MiB 2.04–2.50 1.24–1.37 1.41–1.67 1.40–1.51
get, full read 16MiB 124.28–129.33 108.36–115.06 115.39–117.97 113.87–114.59
delete 16MiB 2.73–2.80 7.26–7.48 2.19–2.48 2.00–2.13
put streamed (length=-1, 5 MiB parts) 64KiB 5.21–5.63 2.49–2.67 4.95–5.26 3.49–3.75
put streamed (length=-1, 5 MiB parts) 1MiB 22.11–23.23 9.04–10.16 15.77–16.41 15.24–15.45
put streamed (length=-1, 5 MiB parts) 16MiB 180.48–188.64 141.36–144.49 148.71–171.76 144.12–148.04
copy_object (HEAD + copy) 64KiB 6.73–8.11 2.91–3.14 4.40–5.01 3.27–3.52
copy_object (HEAD + copy) 1MiB 16.52–17.32 5.91–6.16 4.21–5.04 3.26–3.62
list_objects prefix, 1000 keys - 103.36–104.38 105.43–111.85 99.33–111.17 97.52–100.57
put, 8 concurrent 64KiB 10.51–14.82 7.28–8.22 19.05–19.37 10.79–12.81
get full, 8 concurrent 64KiB 6.51–7.33 7.20–7.89 7.84–8.85 8.19–8.98

Resources

MinIO versitygw (keep-alive) Garage (SQLite) Garage (LMDB)
Idle working set (kubectl top) 218–399 Mi 26–28 Mi 6 Mi 5 Mi
Max working set during the minio-py suite 484 Mi 40 Mi 18 Mi 17 Mi
CPU under warp load, median / max 1665 / 1715 m 1268 / 1348 m 1945 / 1955 m, at the 2-CPU limit 1936 / 1958 m, at the 2-CPU limit
Working set under warp load, median / max 411 / 762 Mi 61 / 87 Mi 10 / 11 Mi 7 / 8 Mi
Server CPU per request under warp (CPU ÷ obj/s) ≈0.89 ms ≈0.46 ms ≈2.2 ms ≈1.95 ms

Garage's working set looks very low because LMDB maps and the block files are counted as page cache. The memory of the scratch image's cgroup could not be read from inside.

Caveats

  • Same limits as the first run: dev1 is a shared cluster; the stores were small and fresh; reads were warm; the kubectl top resolution is about 15 s.
  • Random payloads defeat compression: the benchmark payloads are random bytes, so zstd cannot shrink them. Real Ryax files (CSV, JSON, zipped directories) would compress partly, which saves disk but not CPU.
  • Garage's defaults are not crash-safe: metadata_fsync = false and data_fsync = false. Turning fsync on would make Garage's writes slower than measured here. MinIO and versitygw were also run with their defaults.
  • Garage was capped at 2 CPU, like the other servers. Under warp it sat at that limit, so with more CPU it would scale further, but it would still cost about 4 to 5× versitygw's CPU per request.
  • The 1 KiB / 4 KiB step in Garage's PUT is Garage storing very small objects inline in its metadata database and writing a separate block file above that threshold. The threshold is INLINE_THRESHOLD = 3072 bytes, so Ryax outputs of 3 KiB or more take the slower path.

Conclusion

  1. Garage fails "no worse than MinIO" for Ryax's pattern.
    • Its single-client PUT, HEAD and DELETE are faster than MinIO's: 1 MiB PUT 16.2 against 22.1 ms, HEAD 0.85 to 0.98×.
    • But the full GET of a 4 KiB to 1 MiB object is 1.2 to 1.5× slower: 4 KiB 2.21 against 1.71 ms, 64 KiB 3.32 against 2.25 ms, 1 MiB 11.9 against 9.8 ms. That is the read every workflow step does on its inputs. Garage returns headers quickly (time to first byte 0.55 to 0.82× MinIO) and is slow to deliver the body.
    • Under concurrency, SQLite PUT is 1.6× slower at 8 clients, and warp throughput is −54 % with SQLite and −47 % with LMDB, CPU-bound at 2 cores.
  2. Disabling compression does not fix it: warp goes from 868 to 991 obj/s, GET latency is unchanged, and the server is still CPU-bound.
  3. LMDB against SQLite: LMDB writes faster (4 KiB PUT 2.70 against 4.05 ms, 8-client PUT 11.8 against 19.2 ms, warp about +14 %) and reads at the same speed. It does not change the verdict, and it brings back the corruption risk on unclean shutdown that SQLite avoids.
  4. Tuned versitygw stays ahead: it is faster than Garage on every PUT and GET size, under concurrency, and in warp (2754 against 868 obj/s). Both stay under 100 Mi of memory. It also keeps the current credentials, needs no new secret, and stores plain files.
  5. The recommendation is unchanged: versitygw with VGW_KEEP_ALIVE=true. Garage's light footprint is real (5 to 18 Mi), but its credential rules, its required s3_region override and its slower reads make it a worse fit for Ryax than versitygw.
Edited by Michael Mercier