Re-evaluate redis memory usage per keyspace
In gitlab-org&8419 the groupcode review and groupsource code groups have been working on reducing the amount of data they store in Redis cache. The've been doing that in the following ways:
Group | Issue | description | Rails.cache or not |
---|---|---|---|
groupcode review | gitlab-org/gitlab#367098 (closed) | replace discussion caching for merge request with etag caching | using Gitlab::Redis::Cache , not affected by #1854 (closed)
|
groupcode review | gitlab-org/gitlab#367145 (closed) | replace diff caching for merge requests with etag caching | using Rails.cache , affected by #1854 (closed)
|
groupsource code | gitlab-org/gitlab#367160 (closed) | reduce the TTL for diff highlighting, extending the TTL when the cache is accessed | using Gitlab::Redis::Cache , not affected by #1854 (closed)
|
Because we currently don't have continuous evaluation of how much memory a certain key is using in Redis (discussing this in &763), we need to manually dump keyspace information to run aggregation across keys in the dump to know the impact of the above changes.
The scripts we've used for creating the dump is in $2352412. This needs to be run from a rails console and it will dump the output in /tmp
The dump can be analysed with the following command:
cat ../redis-maxmem-analysis/data/gprd-redis-cache-keyspace-info.ndjson | jq -c -M '{key: .key, weights: [.size, .idletime, 1]}' | ./redis-keyspace-analyzer -weighted 100 100000000 'peek:requests:' 'cache:gitlab:views:' 'cache:gitlab:(<method>size|commit_count|readme_path|contribution_guide|changelog|license_blob|license_key|gitignore|gitlab_ci_yml|branch_names|tag_names|branch_count|tag_count|avatar|exists\?|root_ref|merged_branch_names|has_visible_content\?|issue_template_names_hash|merge_request_template_names_hash|user_defined_metrics_dashboard_paths|xcode_project\?|has_ambiguous_refs\?|license|):'
This uses a slightly modified version of the keyspace analyser from redis-keyspace-analyzer!4 (closed)
We need to keep in mind that we've also reduced the TTL of
Rails.cache
in
#1854 (closed), so
we should rely on overall % of the cache, not absolute
numbers.