Roll out Gitaly pack-objects cache in gstg and gprd
Enable the pack-objects cache across all Gitaly servers in gprd, gstg.
Background
See documentation MR gitlab-org/gitlab!57371 (merged).
Preconditions
-
✅ This thanos query must return 0. This indicates whether gitlab-org/gitaly!3264 (merged) has been deployed yet; non-zero values indicate an old version of the code so we want to see 0. - This command must return return 1 on all Gitaly servers:
grep -c pack_objects_cache_enabled /opt/gitlab/embedded/cookbooks/gitaly/attributes/default.rb
. This reflects whether gitlab-org/omnibus-gitlab!5120 (merged) has been deployed.
Changes
-
✅ Enablegitaly_upload_pack_gitaly_hooks
feature flag production#4122 (closed) - Wait for precondition 2 to apply
- Put in gitlab.rb on Gitaly servers:
gitaly['pack_objects_cache_enabled'] = true
Observability
Is the cache on?
-
PackObjectsHook request rate should be non-zero. This shows whether the
gitaly_upload_pack_gitaly_hooks
feature flag is on. Last time we did this, the total rate in gprd (across all Gitaly servers) was about 100 req/s - cache enabled gauge should be 1 for all servers. This shows whether the config.toml change has been picked up by Gitaly
Things to look out for
- disk write throughput: this is expected to go up once the cache is active. In some cases (e.g. file-praefect-XX) by a lot. Experiments suggest we will not saturate this resource but it's good to keep an eye on
Status 2021-04-08
The cache is on in gprd except for the file-hdd nodes #1004 (closed). We discovered a bug that has a noticeable negative impact on server health #999 (closed). A fix for the bug is in review. gitlab-org/gitaly!3341 (merged)