Add coordinating nodes to Global Search Elasticsearch cluster in Staging environment
Production Change
Change Summary
As part of Global Search reconfiguration, we propose to introduce dedicated coordinating nodes. Please see for more detailed description of coordinating nodes, https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14020. This CR is to request adding coordinating node(s) to Global Search Elasticsearch cluster in staging environment.
Please reach out to #g_global_search on Slack for any questions.
Change Details
- Services Impacted - Global Search Elasticsearch Cluster
- Change Technician - DRI for the execution of this change
- Change Reviewer - DRI for the review of this change
- Time tracking - Time, in minutes, needed to execute all change steps, including rollback
- Downtime Component - If there is a need for downtime, include downtime estimate here
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 0
-
Set label changein-progress on this issue
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 10
-
Log in to Elastic Cloud https://cloud.elastic.co/ using the shared credentials in SRE 1password -
Find cluster com-gitlab-staging indexing-20200406 -
Find coordinating node setting -
Click Add Capacity -
Choose 1 GB RAM | Up to 4vCPU in Size per zone -
Choose 1 zone in Availability zones -
Save it
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 0
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Remove the coordinating nodes from the staging cluster
Monitoring
- Global Search staging cluster dashboard, https://00a4ef3362214c44a044feaa539b4686.us-central1.gcp.cloud.es.io:9243/app/monitoring#/overview?_g=(cluster_uuid:ehQM65I1Rnix_1zBl6g4uA)
Key metrics to observe
- Metric: Metric Name
- Location: Dashboard URL
- What changes to this metric should prompt a rollback: Describe Changes
Summary of infrastructure changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Summary of the above
Changes checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
This Change Issue is linked to the appropriate Issue and/or Epic -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers
and this issue and await their acknowledgment.) -
There are currently no active incidents.