Fix caching logic for protected branches

• Bug report: gitlab-com/gl-infra/scalability#1601 (comment 1010867339)
• Hotfix: !91410 (merged)

Problem

Protected branches cache depends on a project cache key. But the cache key updates for every new issue, merge request and push to the project. That triggers a creation of new protected branches cache without removing the old versions of it.

As a result, we generate lots of cache keys that get stale quickly.

Proposal

• Use a better strategy for caching. As an option, we can refresh the cache only when protected branches are changed.
• Clean-up stale cache keys

Feature flags

• hash_based_cache_for_protected_branches - enables the new caching implementation (but by default it's in dry-run mode)
• rely_on_protected_branches_cache - requires hash_based_cache_for_protected_branches to be enabled and controls the dry-run mode.

hash_based_cache_for_protected_branches	rely_on_protected_branches_cache	Result
false	false	Use the old cache
false	true	Use the old cache
true	false	Use the new cache (dry-run: true)
true	true	Use the new cache (dry-run: false)

added backend groupsource code typebug labels

assigned to @vyaklushin

mentioned in issue gitlab-com/gl-infra/scalability#1601

Setting label(s) Category:Source Code Management devopscreate sectiondev based on groupsource code.

added Category:Source Code Management devopscreate sectiondev labels

added workflowin dev label

mentioned in commit 3380e7f3

mentioned in merge request !92797 (closed)

mentioned in commit 34a8e943

mentioned in commit e3f0ec39

mentioned in merge request !92922 (merged)

mentioned in commit ceb2c09a

mentioned in merge request !92934 (merged)

mentioned in commit 9a7429cd

mentioned in merge request !92937 (merged)

mentioned in commit c975f738

mentioned in commit d4f5f229

added to epic &8419

mentioned in commit a8c856a2

added workflowin review label and removed workflowin dev label

mentioned in commit e4763cd6

mentioned in commit 004a65e3

mentioned in commit 8e25b790

mentioned in commit 8aebc640

mentioned in commit 20d47111

changed milestone to %15.3

mentioned in commit 30854c49

mentioned in issue #370078 (closed)

mentioned in commit 065912ad

mentioned in commit 3dbd8c4a

mentioned in commit c71a6059

mentioned in commit 0d0ddcf9

added workflowin dev label and removed workflowin review label

mentioned in commit b63873f0

mentioned in commit 79a9ddce

mentioned in merge request !95167 (merged)

mentioned in commit 0a8975eb

mentioned in commit 426e2c43

added workflowin review label and removed workflowin dev label

mentioned in commit ead0bb68

mentioned in commit 8a5c5331

mentioned in issue gitlab-org/quality/triage-reports#8728 (closed)

mentioned in commit 64b9b5f8

mentioned in commit cd1e2dfe

mentioned in commit bd30d841

mentioned in commit 633b5cc9

@vyaklushin great job on this one. Are there any graphs or stats we can add as an outcome of this change?

Did we reduce the size of our Redis memory size or number of cache keys in any way we can measure?

@sean_carroll it's not fully deployed yet. I'm currently testing it to spot and fix any problems related to stale cache.

When it's enabled we can calculate the new size of protected branches cache. It should be much less than 5 GiB as we have now.

mentioned in commit 12649fc6

mentioned in merge request !95702 (merged)

mentioned in commit 4ac13d9f

mentioned in commit 5bd75706

changed milestone to %15.4

added missed:15.3 label

mentioned in commit 40fd6505

mentioned in commit 8cbbbe55

changed the description

mentioned in commit 04913595

mentioned in merge request !96130 (merged)

mentioned in commit eda46aa3

mentioned in merge request !96237 (closed)

mentioned in issue #370608 (closed)

mentioned in commit 6a037285

added workflowverification label and removed workflowin review label

The new protected branches logic is fully enabled for GitLab.com. I'll wait for couple days to verify that it works without any problems.

Hi @reprazent! The migration to the new protected branch cache is complete.

When you have time could you please check how much space the new cache occupies? It's interesting to see the impact of this change.

@vyaklushin Sorry for not getting back to you earlier. As we've talked about before in gitlab-com/gl-infra&763, checking the overall utilization is currently a manual process by scanning the keyspace and matching patterns for certain keys. Which is why I haven't gotten to it yet.

However, the biggest your change would have made is an increase in hit ratio for the cache. I believe you added metrics for those, did you gather results on that?

If a cache is hit enough, then we're better of allocating the extra memory for it .

mentioned in issue gitlab-org/quality/triage-reports#9137 (closed)

mentioned in issue gitlab-com/create-stage/source-code-be#8 (closed)

changed milestone to %Backlog

I've checked the hit ratio for new ProtectedBranches implementation. It fluctuates around 90% of cache hits which is a good result.

Source

closed