[GSTG] Set database_tasks: false for CI
Production Change
Change Summary
There is a new field database_tasks
that we want to set explicitly to false
for our new ci
database on our Rails hosts. This field was introduced in gitlab-org/omnibus-gitlab!5982 (merged) and gitlab-org/charts/gitlab!2450 (merged) . This setting (when applied to the ci
configuration of the database) tells Rails not to run any migrations for this database configuration. The reason that we don't want/need to run migrations for the ci
database yet is that it points to the same database until we reach the final failover "Phase 7" of our migration gitlab-org&6160 (closed) . At that point we'll need to remove this configuration and allow us to run separate migrations for ci
again.
Technically we only need to set this configuration option on the deploy node but for consistency it probably makes sense for all Rails nodes to see this same configuration.
Change Details
- Services Impacted - ServiceDeploy-Node
- Change Technician - @DylanGriffith
- Change Reviewer - DRI for the review of this change
- Time tracking - 10
- Downtime Component - 0
Detailed steps for the change
Pre-Change Steps - steps to be completed before execution of the change
Estimated Time to Complete (mins) - 10
-
Set label changein-progress on this issue -
Confirm that we're using the version of CNG in production that supports database_tasks
correctly (gitlab-org/charts/gitlab#3203 (closed)) gitlab-org/charts/gitlab!2480 (merged)
Change Steps - steps to take to execute the change
Estimated Time to Complete (mins) - 60
-
Set ci.database_tasks: false
in omnibus hosts -
Confirm in teleport staging console that the config/database.yml
looks correctputs File.read(Rails.root.join("config/database.yml"))
-
Set ci.database_tasks: false
in CNG hosts for both Sidekiq and global -
Set label changecomplete on this issue
Post-Change Steps - steps to take to verify the change
Estimated Time to Complete (mins) - 10
-
Validate that the omnibus configuration of config/database.yml
looks correct -
Validate that the CNG configuration of config/database.yml
looks correct
Rollback
Rollback steps - steps to be taken in the event of a need to rollback this change
Estimated Time to Complete (mins) - Estimated Time to Complete in Minutes
-
Revert any of the MRs that were merged
Monitoring
Key metrics to observe
- Metric: Successful deployment to staging
- Location: Dashboard URL
- What changes to this metric should prompt a rollback: The only thing we could see going wrong is if the config is wrong of the deploy node then our database migrations might fail during deployment to staging.
Summary of infrastructure changes
-
Does this change introduce new compute instances? -
Does this change re-size any existing compute instances? -
Does this change introduce any additional usage of tooling like Elastic Search, CDNs, Cloudflare, etc?
Summary of the above
Change Reviewer checklist
-
The scheduled day and time of execution of the change is appropriate. -
The change plan is technically accurate. -
The change plan includes estimated timing values based on previous testing. -
The change plan includes a viable rollback plan. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change.
-
The complexity of the plan is appropriate for the corresponding risk of the change. (i.e. the plan contains clear details). -
The change plan includes success measures for all steps/milestones during the execution. -
The change adequately minimizes risk within the environment/service. -
The performance implications of executing the change are well-understood and documented. -
The specified metrics/monitoring dashboards provide sufficient visibility for the change. - If not, is it possible (or necessary) to make changes to observability platforms for added visibility? -
The change has a primary and secondary SRE with knowledge of the details available during the change window.
Change Technician checklist
-
This issue has a criticality label (e.g. C1, C2, C3, C4) and a change-type label (e.g. changeunscheduled, changescheduled) based on the Change Management Criticalities. -
This issue has the change technician as the assignee. -
Pre-Change, Change, Post-Change, and Rollback steps and have been filled out and reviewed. -
This Change Issue is linked to the appropriate Issue and/or Epic -
Necessary approvals have been completed based on the Change Management Workflow. -
Change has been tested in staging and results noted in a comment on this issue. -
A dry-run has been conducted and results noted in a comment on this issue. -
SRE on-call has been informed prior to change being rolled out. (In #production channel, mention @sre-oncall
and this issue and await their acknowledgement.) -
Release managers have been informed (If needed! Cases include DB change) prior to change being rolled out. (In #production channel, mention @release-managers
and this issue and await their acknowledgment.) -
There are currently no active incidents.