redefine minio tenants
We had a discussion around minio configuration yesterday with @cristian.manda @alinhg @feleouet and @rletrocquer around the following changes to do around minio tenants:
- (1) when deployed on baremetal on top of longhorn PVs:
- to avoid the data replication churn during node rolling updates when using single-replica (before rebuilding the node hosting the single replica, it needs to fully be moved elsewhere) we can move to using replicated longhorn PVs
- we should then tune minio settings to adapt to the fact that PV data will not be lost
- (2) to make tuning and operations easier, we should split the single minio tenant we have into one for logging and one for mmonitoring
Both (1) and (2) can't be easily done while preserving data during an upgrade, so the proposition is to do them both at the same time.
/cc @mihai.zaharia