Skip to content

Fix RedisHLL migration related to removal of custom slots

Niko Belokolodov requested to merge 280527-remove-the-possibility-to-set-redis_slot-in-known_events into master

What does this MR do and why?

It was discovered that in !110952 (merged) there was a bug in Redis data migration script.

The data wasn't migrated because !110952 (diffs) wasn't actually loading events (called from the wrong place), events aren't stored in db/post_migration folder.

This MR re-introduces the migration with the bug being fixed.

At the moment we have data being added to Redis with new keys. Old keys are still in Redis. ServicePing aggregates data taking the new keys into account.

This MR merged old keys with new ones, so we don't loose the old data on Gitlab.com

Custom redis_slot will be removed a following MR, it's not in use in the current codebase but still required for migration.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

  1. Add keys to Redis HLL from rails console:
> Gitlab::Redis::HLL.add(key: "users_viewing_{analytics}_group_devops_adoption-2023-14", value: 2, expiry: 6.weeks)
=> [true, true]
[46] pry(main)> Gitlab::Redis::HLL.count(keys: "users_viewing_{analytics}_group_devops_adoption-2023-14") # old key
=> 1 
[50] pry(main)> Gitlab::Redis::HLL.count(keys: "{hll_counters}_users_viewing_group_devops_adoption-2023-14") # new key
=> 0 # check that new keys is empty
  1. re-run old broken migration bin/rails db:migrate:redo:main VERSION=20230202211434
  2. make sure keys weren't migrated
> Gitlab::Redis::HLL.count(keys: "{hll_counters}_users_viewing_analytics_group_devops_adoption-2023-14")
=> 0
  1. checkout the current branch git checkout 280527-remove-the-possibility-to-set-redis_slot-in-known_events
  2. apply patch to make migration more visible:
Index: db/post_migrate/20230302811133_re_migrate_redis_slot_keys.rb
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/db/post_migrate/20230302811133_re_migrate_redis_slot_keys.rb b/db/post_migrate/20230328111013_re_migrate_redis_slot_keys.rb
rename from db/post_migrate/20230302811133_re_migrate_redis_slot_keys.rb
rename to db/post_migrate/20230328111013_re_migrate_redis_slot_keys.rb
--- a/db/post_migrate/20230302811133_re_migrate_redis_slot_keys.rb	(revision e11703a6376860cf8042aa73180bdd9e71d76afd)
+++ b/db/post_migrate/20230328111013_re_migrate_redis_slot_keys.rb	(date 1679990300856)
@@ -45,6 +45,7 @@
     Gitlab::Redis::SharedState.with do |r|
       break unless r.exists?(old_key)
 
+      pp "Renamed #{old_key} to #{new_key}"
       Gitlab::Redis::HLL.add(
         key: new_key,
         value: r.pfcount(old_key),
  1. run the new migration bin/rails db:migrate:redo:main VERSION=20230328111013
  2. make sure keys were migrated
> Gitlab::Redis::HLL.count(keys: "{hll_counters}_users_viewing_analytics_group_devops_adoption-2023-14")
=> 1

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #280527 (closed)

Edited by Niko Belokolodov

Merge request reports