Set up ops staging environment
Ops environment Change
Change Summary
Create a new staging environment opt-stg
which will be a replica of Thanos query production environment. This is part of https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15341
Change Details
- Services Impacted - ServiceThanos
- Change Technician - @swainaina
- Change Reviewer - @steveazz
- Time tracking - 1 hour
- Downtime Component - None
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 10min
-
Set label changein-progress on this issue -
Create Thanos Query DNS 👉 https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/3560 -
Add IAM identity 👉 https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/3587
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 10m
-
Create ops staging environment -
Merge 👉 gitlab-com/gl-infra/k8s-workloads/tanka-deployments!343 (merged) -
Wait for pipeline on ops.GitLab.net
to pass and fully roll out -
Check ops-stg
rollout is successful.# In 1 terminal window glsh kube use-cluster ops # In another terminal window kubectl -n monitoring-stg get deploy thanos-query kubectl -n monitoring-stg get deploy thanos-query-frontend
-
Check if https://thanos-query-stg.ops.gitlab.net/ is accessible. -
Create the thanos-iap-oauth
secret
kubectl -n monitoring get secrets thanos-iap-oauth -o yaml |\ kubectl apply --namespace=monitoring-stg -f -
-
Confirm that the secret has been successfully created
kubectl get secrets -n monitoring-stg thanos-iap-oauth
-
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 30m
-
Confirm the version of Thanos Query image v0.20.1
(same as production)kubectl -n monitoring-stg get po -l app.kubernetes.io/name=thanos-query -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}'
-
Confirm the version of Thanos Query frontend image v0.20.1
kubectl -n monitoring-stg get po -l app.kubernetes.io/name=thanos-query-frontend -o jsonpath='{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}'
-
Confirm that we are receiving the metrics https://thanos-query-stg.ops.gitlab.net/ metrics similar to production https://thanos-query.ops.gitlab.net/ -
Staging environment only accessible to GitLab team members, shouldn't be public. -
Can access the same thanos-store
asthanos-query
we have inmonitoring
-
We have logging saved in Kibana. https://nonprod-log.gitlab.net/app/home -
Distributed traces are being uploaded. -
Metrics are being scraped.
Rollback
Rollback steps - steps to be taken in the event of a need to roll back this change
Estimated Time to Complete (mins) - 20m
-
Revert this MR👉 gitlab-com/gl-infra/k8s-workloads/tanka-deployments!343 (merged) -
Delete the previously created secretskubectl delete secrets -n monitoring-stg thanos-iap-oauth
-
Delete themonitoring-stg
namespacekubectl delete ns monitoring-stg
Monitoring
Key metrics to observe
- Metric: Metric Name
- Location: Dashboard URL
- What changes to this metric should prompt a rollback: Describe Changes
Summary of infrastructure changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Summary of the above
Change Reviewer checklist
-
The scheduled day and time of execution of the change is appropriate. -
The change plan is technically accurate. -
The change plan includes estimated timing values based on previous testing. -
The change plan includes a viable rollback plan. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change.
-
The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). -
The change plan includes success measures for all steps/milestones during the execution. -
The change adequately minimizes risk within the environment/service. -
The performance implications of executing the change are well-understood and documented. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? -
The change has a primary and secondary SRE with knowledge of the details available during the change window.
Change Technician checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
This Change Issue is linked to the appropriate Issue and/or Epic -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers
and this issue and await their acknowledgment.) -
There are currently no active incidents.
Edited by Silvester Wainaina