thanos-storegateway is using old minio-monitoring tenant

During mgmt cluster upgrade from 1.1.1 we are recreating minio tenants with new names, but the thanos-storegateway pod is failing while trying to connect to the bucket. Seen in https://gitlab.com/sylva-projects/sylva-core/-/jobs/8417218188

Logs:

ts=2024-11-19T23:16:27.852227558Z caller=grpc.go:174 level=info service=gRPC/server component=store msg="internal server is shutting down" err="bucket store initial sync: sync block: BaseFetcher: iter bucket: The request signature we calculated does not match the signature you provided. Check your key and signing method."
ts=2024-11-19T23:16:27.852279582Z caller=grpc.go:187 level=info service=gRPC/server component=store msg="gracefully stopping internal server"
Assignee Loading
Time tracking Loading