Broadcast message stale cache on different Gitlab revisions
What does this MR do and why?
Previously, redis expiration wasn't working in some cases where canary and production were on different revisions - see #412199 (closed)
- Fixes the cache expiration issue with Broadcast messages that can occur from canary vs production
Gitlab.revision
differences. - Extends the
JsonCache
class to support nested key structure to allow for easy key expiration logic.
The way the fix here works is to make one cache entry with many revisions. Before there was a cache entry per Gitlab.revision
. So we take advantage of the json structure here and simply move the Gitlab.revision
from being part of the key for the cache to an entry in the cache as seen below.
Previous structure used for broadcast messages
cache key: cache:gitlab:broadcast_message_current_json:d9cef69a710
- namespace:
cache:gitlab
defined here - key:
broadcast_message_current_json
or variants, defined here - git revision:
d9cef69a710
changes for code changes(causing current expire issue) defined here
[{"id"=>34,
"message"=>"MyText",
"starts_at"=>"2023-05-29T23:52:54.006Z",
"ends_at"=>"2023-05-31T23:52:54.006Z",
"created_at"=>"2023-05-30T23:52:54.006Z",
"updated_at"=>"2023-05-30T23:52:54.006Z",
"color"=>"#E75E40",
"font"=>"#FFFFFF",
"message_html"=>"<p>MyText</p>",
"cached_markdown_version"=>2097152,
"target_path"=>nil,
"broadcast_type"=>"banner",
"dismissable"=>nil,
"target_access_levels"=>[],
"theme"=>"indigo"}]
New structure used for broadcast messages
cache key: cache:gitlab:broadcast_message_current_json
- namespace:
cache:gitlab
defined here - key:
broadcast_message_current_json
or variants, defined here - git revision:
d9cef69a710
now moved into becoming part of the cached result...this makes the value of the keys larger, but shouldn't take more space up in redis overall as we only have one key with many revisions now instead of a key per revision.
{"d9cef69a710"=>
[{"id"=>58,
"message"=>"MyText",
"starts_at"=>"2023-05-30T02:58:18.949Z",
"ends_at"=>"2023-06-01T02:58:18.949Z",
"created_at"=>"2023-05-31T02:58:18.950Z",
"updated_at"=>"2023-05-31T02:58:18.950Z",
"color"=>"#E75E40",
"font"=>"#FFFFFF",
"message_html"=>"<p>MyText</p>",
"cached_markdown_version"=>2097152,
"target_path"=>nil,
"broadcast_type"=>"banner",
"dismissable"=>nil,
"target_access_levels"=>[],
"theme"=>"indigo"}],
"abc123"=>
[{"id"=>58,
"message"=>"MyText",
"starts_at"=>"2023-05-30T02:58:18.949Z",
"ends_at"=>"2023-06-01T02:58:18.949Z",
"created_at"=>"2023-05-31T02:58:18.950Z",
"updated_at"=>"2023-05-31T02:58:18.950Z",
"color"=>"#E75E40",
"font"=>"#FFFFFF",
"message_html"=>"<p>MyText</p>",
"cached_markdown_version"=>2097152,
"target_path"=>nil,
"broadcast_type"=>"banner",
"dismissable"=>nil,
"target_access_levels"=>[],
"theme"=>"indigo"}]}
How to set up and validate locally
Follow 'Verify by UI' steps here
example of what those commands might look like
10088 git checkout -b test-broadcast
10089 git stash pop
10090 git add -A
10091 git commit -m 'Test broadcast cache'
10092 git rev-parse HEAD
10093 git checkout -
10094 git rev-parse HEAD
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #412199 (closed)
Merge request reports
Activity
changed milestone to %16.1
assigned to @dstull
added docs-only label
- A deleted user
added backend label
4 Warnings This MR changes code in ee/
, but its Changelog commit is missing theEE: true
trailer. Consider adding it to your Changelog commits.This merge request is quite big (1029 lines changed), please consider splitting it into multiple merge requests. fbdbddb0: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. Please add a merge request subtype to this merge request. 1 Message This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge. Documentation review
The following files require a review from a technical writer:
-
doc/development/caching.md
(Link to current live version)
The review does not need to block merging this merge request. See the:
-
Metadata for the
*.md
files that you've changed. The first few lines of each*.md
file identify the stage and group most closely associated with your docs change. - The Technical Writer assigned for that stage and group.
- Documentation workflows for information on when to assign a merge request for review.
Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer backend Carla Drago (
@carlad-gl
) (UTC+2, 6 hours ahead of@dstull
)Ahmed Hemdan (
@ahmed.hemdan
) (UTC+2, 6 hours ahead of@dstull
)frontend Deepika Guliani (
@deepika.guliani
) (UTC+5.5, 9.5 hours ahead of@dstull
)Phil Hughes (
@iamphill
) (UTC+1, 5 hours ahead of@dstull
)test for spec/features/*
Carla Drago (
@carlad-gl
) (UTC+2, 6 hours ahead of@dstull
)Maintainer review is optional for test for spec/features/*
To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Danger-
removed docs-only label
- Resolved by Nicolas Dular
Allure report
allure-report-publisher
generated test report!e2e-test-on-gdk:
test report for 922f278dexpand test summary
+-----------------------------------------------------------------------+ | suites summary | +------------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +------------------+--------+--------+---------+-------+-------+--------+ | Framework sanity | 0 | 0 | 1 | 0 | 1 | ➖ | | Manage | 1 | 0 | 0 | 0 | 1 | ✅ | | Create | 8 | 0 | 1 | 0 | 9 | ✅ | | Plan | 4 | 0 | 0 | 0 | 4 | ✅ | | Monitor | 4 | 0 | 0 | 0 | 4 | ✅ | | Govern | 2 | 0 | 0 | 0 | 2 | ✅ | | Data Stores | 2 | 0 | 0 | 1 | 2 | ❗ | +------------------+--------+--------+---------+-------+-------+--------+ | Total | 21 | 0 | 2 | 1 | 23 | ❗ | +------------------+--------+--------+---------+-------+-------+--------+
added 159 commits
-
50326996...697e8b0e - 158 commits from branch
master
- 8d75e268 - Fix broadcast messages expiring cache
-
50326996...697e8b0e - 158 commits from branch
- A deleted user
added frontend label
- Resolved by Kamil Trzciński
- Resolved by Nicolas Dular
- Resolved by Nicolas Dular
requested review from @jmontal
added workflowin review label and removed workflowin dev label
@jmontal
, thanks for approving this merge request.This is the first time the merge request is approved. To ensure full test coverage, a new pipeline will be started shortly.
For more info, please refer to the following links:
added pipeline:mr-approved label
mentioned in issue #412200 (closed)
removed review request for @ayufan
added 1 commit
- c661b791 - Refine as true base / inherited class concept
added 1 commit
- fd6f1e64 - Refine as true base / inherited class concept
- A deleted user
added documentation label
added 1 commit
- f142594c - Refine as true base / inherited class concept
added 1 commit
- 83d32335 - Refine as true base / inherited class concept
- Resolved by Nicolas Dular
requested review from @ayufan
added 1170 commits
-
83d32335...f7eb8732 - 1167 commits from branch
master
- 85265462 - Fix broadcast messages expiring cache
- fbdbddb0 - Refactor to separate class
- ee5bacd3 - Refine as true base / inherited class concept
Toggle commit list-
83d32335...f7eb8732 - 1167 commits from branch
added 1 commit
- 22345bbe - Refine as true base / inherited class concept
- Resolved by Doug Stull
requested review from @nicolasdular
removed review request for @ayufan
- Resolved by Nicolas Dular
@nicolasdular, did you forget to run a pipeline before you merged this work? Based on our code review process, if the latest pipeline was created more than 6 hours ago OR finished more than 2 hours ago, you should:
- Ensure the merge request is not in Draft status.
- Start a pipeline (especially important for Community contribution merge requests).
- Set the merge request to merge when pipeline succeeds.
This is a guideline, not a rule. Please consider replying to this comment for transparency.
This message was generated automatically. You're welcome to improve it.
mentioned in commit 0bd8a2f6
marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed
mentioned in issue #412199 (closed)
added workflowstaging-canary label and removed workflowin review label
added workflowcanary label and removed workflowstaging-canary label
added workflowstaging label and removed workflowcanary label
added workflowproduction label and removed workflowstaging label
added workflowpost-deploy-db-staging label and removed workflowproduction label
added workflowpost-deploy-db-production label and removed workflowpost-deploy-db-staging label
added releasedcandidate label
mentioned in merge request kubitus-project/kubitus-installer!2224 (merged)
added releasedpublished label and removed releasedcandidate label