[release-1.7] enable xfs storage class units for minio-logging (!7409) · Merge requests · Sylva-projects / sylva-core

This MR specifically targets Sylva 1.7 and has no counterpart on main because Minio is being replaced by RustFS in main

What does this MR do and why?

Closes #3882

Loki generates a very large number of small chunk objects. At high cardinality, the number of files written to the MinIO data directory grows rapidly. ext4 has a fixed inode table allocated at format time. Once inodes are exhausted, MinIO cannot create new objects even when disk space is still available, causing write failures and log ingestion stalls.

So solution was to replace ext4 with XFS filesystem. XFS allocates inodes dynamically as needed, limited only by available disk space rather than a fixed pre-allocated table. This makes it significantly better suited for workloads that produce large numbers of small files, which is exactly the MinIO object storage access pattern.

Changes

Kyverno Policy kyverno-policies/minio-policies/minio-storage-xfs: There is an addition of new kyverno policy which would create a new storageclass to be used by minio-logging tenant named logging. This MR is not forcing to have minio-logging storageclass but provides an option to user if it needs to switch from ext4 to xfs storageclas. The minio-logging storage class would be the XFS version of current default storageclass.
Added the following variable under _internal.minio_logging:
- base_storage_class: Specifies the existing StorageClass that the Kyverno policy should clone when creating the XFS-based StorageClass.
  - When two-replicas-storageclass is enabled for Longhorn, that StorageClass is used as the base, and Kyverno creates an XFS variant from it.
  - Otherwise, the default StorageClass is used as the base, and Kyverno creates an XFS variant by adding the XFS filesystem configuration.
- storage_class: Selects the StorageClass to be used by the MinIO logging Tenant while preserving the existing StorageClass during upgrades unless an explicit change is requested.
- pool_name: This variable is used to compute the new name of pool when there is new storageclass which is to be used else pool name remains the same. This is required for preserving old pool name in sylva version upgrades. For an upgrade of stoageclass change for minio-logging tenant it would use new name so that it dont collide with used PVC by old storageclass.
- storageclass_changed: A boolean flag set to true when the MinIO logging Tenant StorageClass has changed, and false otherwise. The minio-logging-tenant-remove unit uses this flag to determine whether the existing logging Tenant must be deleted before reconciliation. This variable was introduced to mitigate the impact of the following upstream MinIO Operator issue: https://github.com/minio/operator/issues/2480

Issue Description: https://github.com/minio/operator/issues/2480

The minio operator needs the user credential secrets even before the tenant is getting created , else the operator creates the tenant without the user credentials provided. As a result, the users field is effectively only processed during Tenant creation.
In our upgrade scenario, changing the pool name and StorageClass created a new pool within the existing Tenant, but the Operator did not reuse the credentials provided through the existing Secret. Consequently, Loki could no longer upload chunks using the expected user credentials.
To work around this issue, the existing logging Tenant is deleted only when the MinIO logging StorageClass changes. The following minio-logging reconciliation then recreates the Tenant and provisions the Loki user using the existing credential Secret.

Scenarios

Fresh Installations

New deployments will automatically use XFS-based storage classes for the MinIO logging tenant

Applies to the following storage backends:

two-replicas-xfs-storageclass and xfs-longhorn
cinder-xfs-storageclass (for OpenStack Cinder)

Upgrades from Previous Versions

Existing deployments retain their current storage class configuration
No automatic filesystem migration occurs during upgrades
Storage class settings are preserved from the previous release to ensure data continuity

This way user would have the choice when to upgrade the filesystem.

Manual Migration Path (Optional)

Users can optionally migrate existing MinIO logging deployments to XFS storage by:

Updating the storage class configuration in

logging:
  minio:
    storage_class: minio-logging

Applying the changes, which will trigger:
- Deletion of the existing storage pool
- Creation of a new pool with XFS filesystem
It also updates the storage class reference in the logging tenant

Ci-Deployment-Values

To test the upgrade from release-1.6, which uses the old storage class, to release-1.7, which uses the new minio-logging storage class, changes were needed in the ci-deployment-values repository. A MR sylva-projects/sylva-elements/ci-tooling/ci-deployment-values!515 has been opened for this test scenario.

Observation

Following is the initial inode allocation

ext4 (inode: 196608)

$ kubectl exec -n minio-logging logging-pool-0-0 -c minio -- df -h | grep -E "Filesystem|/export"                                                             
Filesystem                    Size  Used Avail Use% Mounted on
/dev/sdf                      2.9G  279M  2.6G  10% /export0
/dev/sdh                      2.9G  279M  2.6G  10% /export1
/dev/sdi                      2.9G  279M  2.6G  10% /export2
/dev/sdg                      2.9G  279M  2.6G  10% /export3

$ kubectl exec -n minio-logging logging-pool-0-0 -c minio -- df -i | grep -E "Filesystem|/export"                                                                     
Filesystem                    Inodes  IUsed   IFree IUse% Mounted on
/dev/sdf                      196608  43647  152961   23% /export0
/dev/sdh                      196608  43647  152961   23% /export1
/dev/sdi                      196608  43647  152961   23% /export2
/dev/sdg                      196608  43647  152961   23% /export3

xfs (inode: 1572864) : Here the around 300 more pods were created to test the inode creation.

$ kubectl exec -n minio-logging logging-pool-0-0 -c minio -- df -h | grep -E "Filesystem|/export"
Filesystem                    Size  Used Avail Use% Mounted on
/dev/sdh                      3.0G  496M  2.5G  17% /export0
/dev/sdg                      3.0G  496M  2.5G  17% /export1
/dev/sdf                      3.0G  496M  2.5G  17% /export2
/dev/sdi                      3.0G  496M  2.5G  17% /export3

$ kubectl exec -n minio-logging logging-pool-0-0 -c minio -- df -i | grep -E "Filesystem|/export"
Filesystem                    Inodes  IUsed   IFree IUse% Mounted on
/dev/sdh                     1572864 179172 1393692   12% /export0
/dev/sdg                     1572864 179172 1393692   12% /export1
/dev/sdf                     1572864 179172 1393692   12% /export2
/dev/sdi                     1572864 179172 1393692   12% /export3

Test coverage

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon	Meaning	Available values
☁️	Infra Provider	`capd`, `capo`, `capm3`
🚀	Bootstrap Provider	`kubeadm` (alias `kadm`), `rke2`, `okd`, `ck8s`
🐧	Node OS	`ubuntu`, `suse`, `na`, `leapmicro`
🛠️	Deployment Options	Deployment option list and description
🎬	Pipeline Scenarios	Available scenario list and description
🟢	Enabled units	Any available units name, by default apply to management and workload cluster. Can be prefixed by `mgmt:` or `wkld:` to be applied only to a specific cluster type
🔴	Disabled units	Any available units name, by default apply to management and workload cluster. Can be prefixed by `mgmt:` or `wkld:` to be applied only to a specific cluster type
🏗️	Target platform	Can be used to select specific deployment environment (i.e `real-bmh` for capm3 )

Global config for deployment pipelines

autorun pipelines
allow failure on pipelines
record sylvactl events

Notes:

Enabling autorun will make deployment pipelines to be run automatically without human interaction
Disabling allow failure will make deployment pipelines mandatory for pipeline success.
if both autorun and allow failure are disabled, deployment pipelines will need manual triggering but will be blocking the pipeline

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.

Edited Sep 24, 2026 by Thomas Morin

[release-1.7] enable xfs storage class units for minio-logging