Make the list of pending post-deploy migrations available to release-tools
As part of &585 (closed), we need to know which post-migrations are going to be executed each time the post-deploy pipeline is executed. For that, we could store the list of pending post-deploy migrations as an artifact to later make it available to release-tools
Technical details
On the deployer, after the migration job is executed, a list of pending post-deploy migrations can be executed by running the /usr/bin/gitlab-rails db:migrate:status
. Example:
https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/6868121
ok: [deploy-cny-01-sv-gprd.c.gitlab-production.internal] =>
msg:
- There are 1 pending migrations.
- - ' down 20220412044906 Finalize traversal ids background migrations'
The rails command filters the migrations in down-state returning the pending post-migrations. It's important to run this command after the regular migration execution or the result will also include those.
We need to export that list from deployer and make it available to release tools as an artifact. Later on, this list will be used to extend the message that is posted on each post-deploy pipeline run #2362 (closed) and to determine if there are pending post-migrations #2364 (closed)
Implementation details
-
Build a play to store the pending post-migrations https://ops.gitlab.net/gitlab-com/gl-infra/deploy-tooling/-/merge_requests/392 -
Expand the gprd-migration-cny job to execute this play and send this information to release-tools https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/479 -
Add a new job on release-tools that receives the information from deployer and save it as an artifact gitlab-org/release-tools!1784 (merged)
Development log
- April 29th - Required merge requests to start the initial tests were merged:
- May 4th - Initial testing the regex to detect pending-migrations is not accurate https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/7002746
-
May 5th
- A merge request fixing the regex is merged https://ops.gitlab.net/gitlab-com/gl-infra/deploy-tooling/-/merge_requests/395
- Logic for fetching post-migrations and sending information to release-tools its moved to an individual job https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/482#note_126574
-
May 6th
- First runs of the
gprd-cny-fetch-post-migrations
job are failing https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/7024435 - https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/483 is opened up to fix these failures.
- First runs of the
-
May 9th
- It was discovered the ansible play requires to run on check mode https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/483#note_126760
-
May 10th
- https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/merge_requests/483 was updated to remove all the ruby dependencies. Last test was successful https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/7057197 / https://gitlab.com/gitlab-org/release-tools/-/jobs/2438525340