Switch to rustfs for logging stack (Loki) (!9184) · Merge requests · Sylva-projects / sylva-core

What does this MR do and why?

Following !8703 (merged) we have rustfs-operator available.

This MR introduces rustfs-logging tenant to replace minio-logging tenant.

New units :

| **rustfs-logging** | create rustfs-operator tenant logging | Kustomization |
| **rustfs-logging-init** | create rustfs-logging tenant secrets admin and user | Kustomization |
| **rustfs-logging-migrate** | one-shot migrate job of bucket data from MinIO logging pools to RustFS logging pools. Until this job finished, pre-upgrade logs are not queryable. | Kustomization |
| **cleanup-namespaces-minio** | strip prune-disabled protection and delete the minio-monitoring and/or minio-logging namespace(s) once their corresponding unit is disabled after migration to rustfs. | Kustomization |

Other changes:

point Loki to new rustfs-logging endpoint : logging-io.rustfs-logging:9000
change dependency for loki-init toward rustfs-logging-init
change user used in loki to point toward the new logging tenant user (loki-rustfs-user)
change dependency on minio-logging to disable it if rustfs-logging is enabled and if there's no previous minio-logging tenant
add rustfs-logging ingresses : API - rustfs-logging.sylva , UI - rustfs-logging-console.sylva
add rustfs default settings for limits/requests and nr of server and volumes per server (nr x volumes : minim of 4 to ensure erasure code - cannot be changed once deployed)
add CI test for rustfs-logging - improvement required with task #4565

Tune redundancy-related parameters with :

rustfs:
    servers_count: 4
    volumes_per_server: 4
    volumes_size: 3Gi
    server_resources:
      limits:
        cpu: 1000m
        memory: 1Gi
      requests:
        cpu: 250m
        memory: 512Mi

Use https://rustfs.com/erasure-code-calculator/ for values on servers_count and volumes_per_server ( cannot be changed once set)

New secrets :

k get secrets -n rustfs-logging
NAME                        TYPE                DATA   AGE
minio-logging-user          Opaque              2      44m
rustfs-logging-admin        Opaque              2      49m
rustfs-logging-server-tls   kubernetes.io/tls   3      48m
rustfs-logging-user         Opaque              2      48m

Login in UI using secret rustfs-logging-admin . OIDC login coming soon with #4533 ( rustfs-logging-user is used only for migration - API access only)

Fresh deployment will install rustfs-logging and dependencies , and skip minio-logging and dependencies.

Upgrade will install rustfs-logging and dependencies, leaving minio-logging and dependencies present as well. rustfs-logging-migrate job checks if this is an upgrade, and if it is, will migrate data from minio logging tenant buckets to rustfs logging tenant buckets using rclone copy https://rclone.org/commands/rclone_copy/. After the migration, grafana UI/ rustfs-logging UI needs a little time to load data from rustfs tenant buckets, but all the previous data is read correctly in grafana dashboards. (any on-going data is not affected by this migration) .

If and only if rustfs-logging-migrate job succeed, sylva-unit-status will report this one-shot unit as completed, and start a new reconciliation with that unit disabled, as well as minio-logging. This will also trigger the cleanup-namespaces-minio kustomization unit that cleans the minio-logging namespace

Grafana graph example for Loki data, at 09:20 - it is the time of the migration from minio to rustfs , when the endpoint is moved from minio to rustfs :

Local CI Test (couldn't add the login test as well, see task #4565) :

Rustfs-logging tenant buckets can be accessed from rustfs-operator UI as well.

Connected MRs:

minio-monitoring tenant with migration and cleanup - !9016 (merged)
Documentation -

Test coverage

Fresh deployment in capo with rke2 - done ok.

Upgrade from main sylva-core with migration and cleanup jobs done.

Example of ing status:

k get ing -A
NAMESPACE                  NAME                      CLASS   HOSTS                            ADDRESS         PORTS     AGE
cattle-monitoring-system   grafana-sylva-ingress     nginx   grafana.sylva                    192.168.19.27   80, 443   11h
cattle-system              rancher                   nginx   rancher.sylva                    192.168.19.27   80, 443   11h
flux-system                flux-webui-weave-gitops   nginx   flux.sylva                       192.168.19.27   80, 443   11h
harbor                     harbor-ingress            nginx   harbor.sylva                     192.168.19.27   80, 443   11h
keycloak                   keycloak                  nginx   keycloak.sylva                   192.168.19.27   80, 443   11h
loki                       loki-gateway              nginx   loki.sylva                       192.168.19.27   80, 443   11h
minio-monitoring           monitoring                nginx   minio-monitoring.sylva           192.168.19.27   80, 443   11h
minio-monitoring           monitoring-console        nginx   minio-monitoring-console.sylva   192.168.19.27   80, 443   11h
oauth2-proxy               oauth2-proxy              nginx   oauth2-proxy.sylva               192.168.19.27   80, 443   11h
policy-reporter            policy-reporter-ui        nginx   policy-reporter.sylva            192.168.19.27   80, 443   11h
rustfs-logging             rustfs-logging            nginx   rustfs-logging.sylva             192.168.19.27   80, 443   55m
rustfs-logging             rustfs-logging-console    nginx   rustfs-logging-console.sylva     192.168.19.27   80, 443   55m
rustfs-operator            rustfs-operator-console   nginx   rustfs-operator.sylva            192.168.19.27   80, 443   57m
thanos                     thanos-bucketweb          nginx   thanos-bucketweb.sylva           192.168.19.27   80, 443   11h
thanos                     thanos-query              nginx   thanos-query.sylva               192.168.19.27   80, 443   11h
thanos                     thanos-query-frontend     nginx   thanos.sylva                     192.168.19.27   80, 443   11h
thanos                     thanos-receive            nginx   thanos-receive.sylva             192.168.19.27   80, 443   11h
thanos                     thanos-storegateway       nginx   thanos-storegateway.sylva        192.168.19.27   80, 443   11h
vault                      vault                     nginx   vault.sylva                      192.168.19.27   80, 443   11h

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon	Meaning	Available values
☁️	Infra Provider	`capd`, `capo`, `capm3`
🚀	Bootstrap Provider	`kubeadm` (alias `kadm`), `rke2`, `okd`, `ck8s`
🐧	Node OS	`ubuntu`, `suse`, `na`, `leapmicro`
🛠️	Deployment Options	Deployment option list and description
🎬	Pipeline Scenarios	Available scenario list and description
🟢	Enabled units	Any available units name, by default apply to management and workload cluster. Can be prefixed by `mgmt:` or `wkld:` to be applied only to a specific cluster type
🔴	Disabled units	Any available units name, by default apply to management and workload cluster. Can be prefixed by `mgmt:` or `wkld:` to be applied only to a specific cluster type
🏗️	Target platform	Can be used to select specific deployment environment Available platform list and description
⚡	Pipeline control	`autorun`, `manual` or `blocking`. Can be used to override global config and start a deployment pipeline the required way

Global config for deployment pipelines

autorun pipelines
allow failure on pipelines
record sylvactl events

Notes:

Enabling autorun will make deployment pipelines to be run automatically without human interaction
Disabling allow failure will make deployment pipelines mandatory for pipeline success.
if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.

Edited Oct 07, 2026 by Cristina Isaroiu

Admin message

Switch to rustfs for logging stack (Loki)

What does this MR do and why?

Related reference(s)

Test coverage

CI configuration

Global config for deployment pipelines

Merge request reports