Skip to content

[thanos] - partition thanos ops storegateway

Nick Duff requested to merge nduff/thanos-ops-partitioning into master

What

  • Time partition thanos ops storegateway
  • Tune some memcached settings

Why

Since duplicating our production rules to our thanos-staging instance, we are seeing increased workload on our ops storegateway.
This will time partition the ops storegateway to help split the workload.

image

Another option is we create a dedicated ops storegateway for staging, but I think time partitioning our ops storegateway is a good thing to do regardless, so it make sense to start here.

We should also be able to reduce resources per partition once we have some data coming in.

Memcached: We are hitting some async-buffer-full and max-item-size limits currently: image

Edited by Nick Duff

Merge request reports