Enable FF track_unique_test_cases_parsed on production
Production Change
Change Summary
This is related to the FF rollout process of track_unique_test_cases_parsed
: gitlab-org/gitlab#262092 (closed)
Related discussion: gitlab-org/gitlab#262092 (comment 436265214)
The feature, when enabled globally, is expected to make a lot of calls to Redis (PFADD
, EXPIRE
). This will happen on every JUnit report parsing. While we have made benchmarks, batched requests, and defined limits in place, if things go wrong, it may still be possible to make things difficult for Redis.
Change Details
- Services Impacted - Redis
- Change Technician - @iamricecake
- Change Criticality - C4
- Change Type - changeunscheduled
- Change Reviewer - TBD
- Due Date - TBD
- Time tracking - Probably just a few seconds to disable the feature flag
- Downtime Component - N/A
Detailed steps for the change
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Enable the flag globally on production /chatops run feature set track_unique_test_cases_parsed true
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Observe the Redis related dashboards and confirm that everything looks fine. See #2902 (comment 443794653)
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Disable flag on production /chatops run feature set track_unique_test_cases_parsed false
Monitoring
Key metrics to observe
- Metric: Metric Name
- Location: https://dashboards.gitlab.net/d/redis-main/redis-overview?orgId=1
- What changes to this metric should prompt a rollback: See #2902 (comment 443794653)
Summary of infrastructure changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Summary of the above
Changes checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled). -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and resultes noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue.) -
There are currently no active incidents.
Edited by Erick Bajao