Upgrade k8s-workload pinned mailroom version to 0.0.19
Production Change
Change Summary
This change issue is to upgrade the current pinned mailroom version in k8s-workloads to 0.0.19. This upgrade is a prerequisite to rollout webhook delivery method for MailRoom (&644 (closed)).
Change Details
- Services Impacted - ServiceMailroom
- Change Technician - @qmnguyen0711
- Change Reviewer -
- Time tracking - changeunscheduled
- Downtime Component - N/A
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 10 mins
-
Set label changein-progress on this issue
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Merge gitlab-com/gl-infra/k8s-workloads/gitlab-com!1520 (merged) -
Verify mailroom functionalities on gstg (see below) -
Merge gitlab-com/gl-infra/k8s-workloads/gitlab-com!1527 (merged) -
Verify mailroom functionalities on gprd (see below)
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 30 mins
GSTG
-
Open this test project's setting on staging -
Sent an email from your personal/work email account to the configured "Email address to use for Support Desk". -
Ensure the email is sent -
An issue must be created in the issue list
GPRD
-
Open this test project's setting on production -
Sent an email from your personal/work email account to "Email address to use for Support Desk": contact-project+qmnguyen0711-test-mail-room-33732257-issue-@incoming.gitlab.com
-
Ensure the email is sent -
An issue must be created in the issue list
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 10 mins
-
Rollback gitlab-com/gl-infra/k8s-workloads/gitlab-com!1520 (merged) -
Rollback gitlab-com/gl-infra/k8s-workloads/gitlab-com!1527 (merged)
Monitoring
Key metrics to observe
-
Metric: EmailReceiverWorker and ServiceDeskReceiverWorker Sidekiq metrics
- Location: https://dashboards.gitlab.net/d/sidekiq-worker-detail/sidekiq-worker-detail?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-worker=EmailReceiverWorker&var-worker=ServiceDeskEmailReceiverWorker
- What changes to this metric should prompt a rollback: RPS drops to 0, or the error rate increases
-
Metric: EmailReceiverWorker and ServiceDeskReceiverWorker Sidekiqlogs
- Location: https://log.gprd.gitlab.net/goto/c35ccfe0-8d81-11ec-a649-b7cbb8e4f62e
- What changes to this metric should prompt a rollback: unexpected high rate.
Summary of infrastructure changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Summary of the above
Change Reviewer checklist
-
The scheduled day and time of execution of the change is appropriate. -
The change plan is technically accurate. -
The change plan includes estimated timing values based on previous testing. -
The change plan includes a viable rollback plan. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change.
-
The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). -
The change plan includes success measures for all steps/milestones during the execution. -
The change adequately minimizes risk within the environment/service. -
The performance implications of executing the change are well-understood and documented. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? -
The change has a primary and secondary SRE with knowledge of the details available during the change window.
Change Technician checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
This Change Issue is linked to the appropriate Issue and/or Epic -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers
and this issue and await their acknowledgment.) -
There are currently no active incidents.
Edited by John Skarbek