Broadcast message stale cache on different Gitlab revisions

What does this MR do and why?

Previously, redis expiration wasn't working in some cases where canary and production were on different revisions - see #412199 (closed)

• Fixes the cache expiration issue with Broadcast messages that can occur from canary vs production Gitlab.revision differences.
• Extends the JsonCache class to support nested key structure to allow for easy key expiration logic.

The way the fix here works is to make one cache entry with many revisions. Before there was a cache entry per Gitlab.revision. So we take advantage of the json structure here and simply move the Gitlab.revision from being part of the key for the cache to an entry in the cache as seen below.

Previous structure used for broadcast messages

cache key: cache:gitlab:broadcast_message_current_json:d9cef69a710

• namespace: cache:gitlab defined here
• key: broadcast_message_current_json or variants, defined here
• git revision: d9cef69a710 changes for code changes(causing current expire issue) defined here

[{"id"=>34,
  "message"=>"MyText",
  "starts_at"=>"2023-05-29T23:52:54.006Z",
  "ends_at"=>"2023-05-31T23:52:54.006Z",
  "created_at"=>"2023-05-30T23:52:54.006Z",
  "updated_at"=>"2023-05-30T23:52:54.006Z",
  "color"=>"#E75E40",
  "font"=>"#FFFFFF",
  "message_html"=>"<p>MyText</p>",
  "cached_markdown_version"=>2097152,
  "target_path"=>nil,
  "broadcast_type"=>"banner",
  "dismissable"=>nil,
  "target_access_levels"=>[],
  "theme"=>"indigo"}]

New structure used for broadcast messages

cache key: cache:gitlab:broadcast_message_current_json

• namespace: cache:gitlab defined here
• key: broadcast_message_current_json or variants, defined here
• git revision: d9cef69a710 now moved into becoming part of the cached result...this makes the value of the keys larger, but shouldn't take more space up in redis overall as we only have one key with many revisions now instead of a key per revision.

{"d9cef69a710"=>
  [{"id"=>58,
    "message"=>"MyText",
    "starts_at"=>"2023-05-30T02:58:18.949Z",
    "ends_at"=>"2023-06-01T02:58:18.949Z",
    "created_at"=>"2023-05-31T02:58:18.950Z",
    "updated_at"=>"2023-05-31T02:58:18.950Z",
    "color"=>"#E75E40",
    "font"=>"#FFFFFF",
    "message_html"=>"<p>MyText</p>",
    "cached_markdown_version"=>2097152,
    "target_path"=>nil,
    "broadcast_type"=>"banner",
    "dismissable"=>nil,
    "target_access_levels"=>[],
    "theme"=>"indigo"}],
 "abc123"=>
  [{"id"=>58,
    "message"=>"MyText",
    "starts_at"=>"2023-05-30T02:58:18.949Z",
    "ends_at"=>"2023-06-01T02:58:18.949Z",
    "created_at"=>"2023-05-31T02:58:18.950Z",
    "updated_at"=>"2023-05-31T02:58:18.950Z",
    "color"=>"#E75E40",
    "font"=>"#FFFFFF",
    "message_html"=>"<p>MyText</p>",
    "cached_markdown_version"=>2097152,
    "target_path"=>nil,
    "broadcast_type"=>"banner",
    "dismissable"=>nil,
    "target_access_levels"=>[],
    "theme"=>"indigo"}]}

How to set up and validate locally

Follow 'Verify by UI' steps here

to verify this(or master if verifying the before) branch you can use this branch(or master if verifying the before) as a base and then create a new branch locally with a minor diff somewhere and create a new local commit from that.

example of what those commands might look like

10088  git checkout -b test-broadcast
10089  git stash pop
10090  git add -A
10091  git commit -m 'Test broadcast cache'
10092  git rev-parse HEAD
10093  git checkout -
10094  git rev-parse HEAD

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #412199 (closed)

lib/gitlab/json_cache.rb

@@ -49,7 +49,14 @@ def read(key, klass = nil)
     end
 
     def write(key, value, options = nil)
-      value = { single_strategy_key_component => value } if @nested_key
+      # We need to 'append write' when fetch calls this
+      # else the writes will key removing each revision's other's cache entries
+      if @nested_key
+        existing_value = existing_cache(key)
+        existing_value[single_strategy_key_component] = value
+        value = existing_value
+      end
+
       backend.write(cache_key(key), value.to_json, options)
     end
 
@@ -68,6 +75,12 @@ def fetch(key, options = {}, &block)
 
     private
 
+    def existing_cache(key)
+      Gitlab::Json.parse(backend.read(cache_key(key)).to_s)
+    rescue JSON::ParserError
+      {}
+    end
+
     def parse_value(raw, klass)
       value = Gitlab::Json.parse(raw.to_s)
       value = value[single_strategy_key_component] if @nested_key
@@ -91,7 +104,7 @@ def parse_entry(raw, klass)
       return klass.new(raw) unless klass.ancestors.include?(ActiveRecord::Base)
 
       # When the cached value is a persisted instance of ActiveRecord::Base in
-      # some cases a relation can return an empty collection becauses scope.none!
+      # some cases a relation can return an empty collection because scope.none!
       # is being applied on ActiveRecord::Associations::CollectionAssociation#scope
       # when the new_record? method incorrectly returns false.
       #