Change the URL for the Spamcheck service in staging
Production Change
Change Summary
This change migrates the staging gitlab instance to a new version of spamcheck. The ML model is the same, so spam verdicts should be the same as the current service. The new service is written in python to aid in more rapid iteration of ML training and model deployment.
Change Details
- Services Impacted - ServiceWeb
- Change Technician - @ahanselka
- Change Reviewer -
- Time tracking - 5 min
- Downtime Component - none
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 1
-
Set label changein-progress on this issue
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 5
-
As an administrator-level user - Visit https://staging.gitlab.com/admin/application_settings/reporting,
- Expand the
Spam and Anti-bot Protection
section, - Change the URL in
URL of the external Spam Check endpoint
totls://grpc-py.primary.spamcheck-dev.sec.gitlab.net:443
. Take note of the current URL in case a rollback is required. - Click "Save changes"
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 5
-
Wait for 5 minutes, observing the metrics -
Submit a test issue in https://staging.gitlab.com/gitlab-org/gitlab -
inspect the logs to see that a spam verdict was calculated by the spamcheck service.
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - 1
-
Change the URL in the URL of the external Spam Check endpoint
at https://staging.gitlab.com/admin/application_settings/reporting back to the original value.
Monitoring
Key metrics to observe
- Metric: Verdicts
- Location: https://console.cloud.google.com/monitoring/metrics-explorer?pageState=%7B%22xyChart%22:%7B%22dataSets%22:%5B%7B%22timeSeriesFilter%22:%7B%22filter%22:%22metric.type%3D%5C%22logging.googleapis.com%2Fuser%2Fspamcheck_python%2Fverdicts%5C%22%20resource.type%3D%5C%22k8s_container%5C%22%22,%22minAlignmentPeriod%22:%2260s%22,%22aggregations%22:%5B%7B%22perSeriesAligner%22:%22ALIGN_RATE%22,%22crossSeriesReducer%22:%22REDUCE_SUM%22,%22alignmentPeriod%22:%2260s%22,%22groupByFields%22:%5B%22metric.label.%5C%22verdict%5C%22%22%5D%7D,%7B%22crossSeriesReducer%22:%22REDUCE_NONE%22,%22alignmentPeriod%22:%2260s%22,%22groupByFields%22:%5B%5D%7D%5D%7D,%22targetAxis%22:%22Y1%22,%22plotType%22:%22LINE%22%7D%5D,%22options%22:%7B%22mode%22:%22COLOR%22%7D,%22constantLines%22:%5B%5D,%22timeshiftDuration%22:%220s%22,%22y1Axis%22:%7B%22label%22:%22y1Axis%22,%22scale%22:%22LINEAR%22%7D%7D,%22isAutoRefresh%22:true,%22timeSelection%22:%7B%22timeRange%22:%221h%22%7D%7D&project=glsec-spamcheck-dev
- Cause for rollback: If the rate of deny verdicts looks anomalous. Short but non-subtle bursts are perhaps the most plausible (although checking back against logs is indicated). Low grade (well less than 1% total) occasional deny verdicts is reasonable; sustained rates above 1% warrant investigation. Above 10% sustained is very unlikely and good grounds to rollback unless a clear justification can be found within around 5 minutes. Correlate with logs to validate.
- Metric: Latency of issue creation:
- Location: https://nonprod-log.gitlab.net/goto/1483d4547e2892ce83c6cac746a35747
- Cause for rollback: Significant increase (more than doubling) in 99th percentile. Or any other change that appears wrong/problematic and is inexplicable.
- Metric: Latency of validation requests (using the log output probably, unless metrics get added). Details to come.
Change Reviewer checklist
-
The scheduled day and time of execution of the change is appropriate. -
The change plan is technically accurate. -
The change plan includes estimated timing values based on previous testing. -
The change plan includes a viable rollback plan. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change.
-
The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). -
The change plan includes success measures for all steps/milestones during the execution. -
The change adequately minimizes risk within the environment/service. -
The performance implications of executing the change are well-understood and documented. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? -
The change has a primary and secondary SRE with knowledge of the details available during the change window.
Change Technician checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
This Change Issue is linked to the appropriate Issue and/or Epic -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers
and this issue and await their acknowledgment.) -
There are currently no active incidents. -
If the change involves doing maintenance on a database host, an appropriate silence targeting the host(s) should be added for the duration of the change.
Edited by Alex Hanselka