Broadcast message stale cache on different Gitlab revisions

What does this MR do and why?

Previously, redis expiration wasn't working in some cases where canary and production were on different revisions - see #412199 (closed)

• Fixes the cache expiration issue with Broadcast messages that can occur from canary vs production Gitlab.revision differences.
• Extends the JsonCache class to support nested key structure to allow for easy key expiration logic.

The way the fix here works is to make one cache entry with many revisions. Before there was a cache entry per Gitlab.revision. So we take advantage of the json structure here and simply move the Gitlab.revision from being part of the key for the cache to an entry in the cache as seen below.

Previous structure used for broadcast messages

cache key: cache:gitlab:broadcast_message_current_json:d9cef69a710

• namespace: cache:gitlab defined here
• key: broadcast_message_current_json or variants, defined here
• git revision: d9cef69a710 changes for code changes(causing current expire issue) defined here

[{"id"=>34,
  "message"=>"MyText",
  "starts_at"=>"2023-05-29T23:52:54.006Z",
  "ends_at"=>"2023-05-31T23:52:54.006Z",
  "created_at"=>"2023-05-30T23:52:54.006Z",
  "updated_at"=>"2023-05-30T23:52:54.006Z",
  "color"=>"#E75E40",
  "font"=>"#FFFFFF",
  "message_html"=>"<p>MyText</p>",
  "cached_markdown_version"=>2097152,
  "target_path"=>nil,
  "broadcast_type"=>"banner",
  "dismissable"=>nil,
  "target_access_levels"=>[],
  "theme"=>"indigo"}]

New structure used for broadcast messages

cache key: cache:gitlab:broadcast_message_current_json

• namespace: cache:gitlab defined here
• key: broadcast_message_current_json or variants, defined here
• git revision: d9cef69a710 now moved into becoming part of the cached result...this makes the value of the keys larger, but shouldn't take more space up in redis overall as we only have one key with many revisions now instead of a key per revision.

{"d9cef69a710"=>
  [{"id"=>58,
    "message"=>"MyText",
    "starts_at"=>"2023-05-30T02:58:18.949Z",
    "ends_at"=>"2023-06-01T02:58:18.949Z",
    "created_at"=>"2023-05-31T02:58:18.950Z",
    "updated_at"=>"2023-05-31T02:58:18.950Z",
    "color"=>"#E75E40",
    "font"=>"#FFFFFF",
    "message_html"=>"<p>MyText</p>",
    "cached_markdown_version"=>2097152,
    "target_path"=>nil,
    "broadcast_type"=>"banner",
    "dismissable"=>nil,
    "target_access_levels"=>[],
    "theme"=>"indigo"}],
 "abc123"=>
  [{"id"=>58,
    "message"=>"MyText",
    "starts_at"=>"2023-05-30T02:58:18.949Z",
    "ends_at"=>"2023-06-01T02:58:18.949Z",
    "created_at"=>"2023-05-31T02:58:18.950Z",
    "updated_at"=>"2023-05-31T02:58:18.950Z",
    "color"=>"#E75E40",
    "font"=>"#FFFFFF",
    "message_html"=>"<p>MyText</p>",
    "cached_markdown_version"=>2097152,
    "target_path"=>nil,
    "broadcast_type"=>"banner",
    "dismissable"=>nil,
    "target_access_levels"=>[],
    "theme"=>"indigo"}]}

How to set up and validate locally

Follow 'Verify by UI' steps here

to verify this(or master if verifying the before) branch you can use this branch(or master if verifying the before) as a base and then create a new branch locally with a minor diff somewhere and create a new local commit from that.

example of what those commands might look like

10088  git checkout -b test-broadcast
10089  git stash pop
10090  git add -A
10091  git commit -m 'Test broadcast cache'
10092  git rev-parse HEAD
10093  git checkout -
10094  git rev-parse HEAD

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #412199 (closed)

.gitlab/ci/qa-common/main.gitlab-ci.yml

@@ -62,9 +62,6 @@ stages:
     GITLAB_LICENSE_MODE: test
     GITLAB_QA_ADMIN_ACCESS_TOKEN: $QA_ADMIN_ACCESS_TOKEN
     GITLAB_QA_OPTS: $EXTRA_GITLAB_QA_OPTS
-    # todo: remove in 16.1 milestone when not needed for backwards compatibility anymore
-    EE_LICENSE: $QA_EE_LICENSE
-    GITHUB_ACCESS_TOKEN: $QA_GITHUB_ACCESS_TOKEN
   # Allow QA jobs to fail as they are flaky. The top level `package-and-e2e:ee`
   # pipeline is not allowed to fail, so without allowing QA to fail, we will be
   # blocking merges due to flaky tests.