An error occurred while fetching the assigned milestone of the selected merge_request.

Fix RedisHLL migration related to removal of custom slots

Review changes
Download
Patches
Plain diff

Merged Fix RedisHLL migration related to removal of custom slots

280527-remove-the-possibility-to-set-redis_slot-in-known_events into master

Overview 27
Commits 7
Pipelines 13
Changes 24

All threads resolved!

Merged Niko Belokolodov requested to merge 280527-remove-the-possibility-to-set-redis_slot-in-known_events into master 1 year ago

Overview 27
Commits 7
Pipelines 13
Changes 24

All threads resolved!

What does this MR do and why?

It was discovered that in !110952 (merged) there was a bug in Redis data migration script.

The data wasn't migrated because !110952 (diffs) wasn't actually loading events (called from the wrong place), events aren't stored in db/post_migration folder.

This MR re-introduces the migration with the bug being fixed.

At the moment we have data being added to Redis with new keys. Old keys are still in Redis. ServicePing aggregates data taking the new keys into account.

This MR merged old keys with new ones, so we don't loose the old data on Gitlab.com

Custom redis_slot will be removed a following MR, it's not in use in the current codebase but still required for migration.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

Add keys to Redis HLL from rails console:

> Gitlab::Redis::HLL.add(key: "users_viewing_{analytics}_group_devops_adoption-2023-14", value: 2, expiry: 6.weeks)
=> [true, true]
[46] pry(main)> Gitlab::Redis::HLL.count(keys: "users_viewing_{analytics}_group_devops_adoption-2023-14") # old key
=> 1 
[50] pry(main)> Gitlab::Redis::HLL.count(keys: "{hll_counters}_users_viewing_group_devops_adoption-2023-14") # new key
=> 0 # check that new keys is empty

re-run old broken migration bin/rails db:migrate:redo:main VERSION=20230202211434
make sure keys weren't migrated

> Gitlab::Redis::HLL.count(keys: "{hll_counters}_users_viewing_analytics_group_devops_adoption-2023-14")
=> 0

checkout the current branch git checkout 280527-remove-the-possibility-to-set-redis_slot-in-known_events
apply patch to make migration more visible:

Index: db/post_migrate/20230302811133_re_migrate_redis_slot_keys.rb
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/db/post_migrate/20230302811133_re_migrate_redis_slot_keys.rb b/db/post_migrate/20230328111013_re_migrate_redis_slot_keys.rb
rename from db/post_migrate/20230302811133_re_migrate_redis_slot_keys.rb
rename to db/post_migrate/20230328111013_re_migrate_redis_slot_keys.rb
--- a/db/post_migrate/20230302811133_re_migrate_redis_slot_keys.rb	(revision e11703a6376860cf8042aa73180bdd9e71d76afd)
+++ b/db/post_migrate/20230328111013_re_migrate_redis_slot_keys.rb	(date 1679990300856)
@@ -45,6 +45,7 @@
     Gitlab::Redis::SharedState.with do |r|
       break unless r.exists?(old_key)
 
+      pp "Renamed #{old_key} to #{new_key}"
       Gitlab::Redis::HLL.add(
         key: new_key,
         value: r.pfcount(old_key),

run the new migration bin/rails db:migrate:redo:main VERSION=20230328111013
make sure keys were migrated

> Gitlab::Redis::HLL.count(keys: "{hll_counters}_users_viewing_analytics_group_devops_adoption-2023-14")
=> 1

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.