[gstg] Enable the usage new redis-sessions instance
Note: the actual switch to using the instances is behind the FF.
I created this issue similar to #5631 (closed), except that it doesn't include the actual usage of the redis-sessions instance, as it will be the part of the FF rollout.
FF rollout issue: scalability#1429 (closed)
GPRD change issue: #5969 (closed)
Production Change
Change Summary
This change will configure the Redis-Sessions instance.
scalability#1309 (closed)
This change is for staging and exists to write & test the process for the equivalent production change.
Change Details
- Services Impacted - ServiceRedis
- Change Technician - @igorwwwwwwwwwwwwwwwwwwww
- Change Reviewer - @nmilojevic1 / @alipniagov
- Time tracking - 35min
- Downtime Component - None
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 1 min
-
Ensure that gitlab-org/gitlab!74202 (merged) has been merged and deployed to at least staging - Obtain review/approval on:
-
Set label changein-progress on this issue
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 30 minutes
-
Merge https://gitlab.com/gitlab-com/gl-infra/chef-repo/-/merge_requests/983 -
From a local copy of chef-repo, run ./bin/gkms-vault-edit gitlab-omnibus-secrets gstg
and add an entry forredis_sessions_instance
alongside the existing redis configs; use the same password, just adjust the identifier at the end. -
gitlab-com/gl-infra/k8s-workloads/gitlab-com!1376 (merged) - NB: depends on the chef MR being merged first to have the expected effect; do not re-arrange the order.
-
Monitor/wait for the k8s pipeline to complete the staging deploy; it is ok to continue with the rest of the change while the QA-on-staging and production jobs are running.
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 2 minutes
We plan to start using the instance via enabling the FF, so this is not part of this change issue (we'll use FF rollout issue). Here, to validate, we should concentrate on two things:
-
Make sure the main Redis instance is healthy. We don't expect any user-invoked queries into the redis-sessions until the FF is off. -
Make sure that the redis-sessions is alive and we could connect to that.
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 30 min
-
Disable the feature flag use_multi_store
, if we enabled that previously (/chatops run feature set use_multi_store false --staging
) -
Set ENV var GITLAB_USE_REDIS_SESSIONS_STORE
to false and restart (true by default) -
If necessary (the presence of the configuration is the problem), revert the k8s and chef MRs and apply.
Monitoring
Key metrics to observe
- Metric: Redis main instance health
- Location: https://dashboards.gitlab.net/d/redis-main/redis-overview?orgId=1
- What changes to this metric should prompt a rollback: the switch to using the new instance is behind the FF, so we don't expect any activity there unless we flip it. We should concentrate on the main instance (shared state) health there.
- Metric: Redis-sessions overview
- Location: https://dashboards.gitlab.net/d/redis-sessions-main/redis-sessions-overview?orgId=1
- We don't expect any user traffic there during the rollout
Summary of infrastructure changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
None
Changes checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
This Change Issue is linked to the appropriate Issue and/or Epic -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers
and this issue and await their acknowledgment.) -
There are currently no active incidents.