[thanos] - partition thanos ops storegateway
What
- Time partition thanos ops storegateway
- Tune some memcached settings
Why
Since duplicating our production rules to our thanos-staging
instance, we are seeing increased workload on our ops storegateway.
This will time partition the ops storegateway to help split the workload.
Another option is we create a dedicated ops storegateway for staging, but I think time partitioning our ops storegateway is a good thing to do regardless, so it make sense to start here.
We should also be able to reduce resources per partition once we have some data coming in.
Memcached: We are hitting some async-buffer-full and max-item-size limits currently:
Edited by Nick Duff