Adjust the post-migration information sent by deployer to release-tools

As part of &585 (closed) and follow up of #2362 (closed)

After the regular migrations are executed on canary, a job on the deployer pipeline sends the pending post-migrations information to release-tools, this information is later processed and stored as an artifact. Example https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/7186318 and https://gitlab.com/gitlab-org/release-tools/-/jobs/2501940109

The information sent by the deployer is a bit convoluted which causes release-tools to thoroughly parse it and store it. For example, deployer can send:

["['\\n", "  20220510192117  Index expirable unknown artifacts for removal', '\\n", "  20220523171107  Drop deploy tokens token column']"]

Or it even triggers a pipeline when no pending post-migrations were found, see https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/jobs/7229978#L5038

$ cat pending_migrations.txt
[]

It'd be great if we could:

1. adjust the output via Ansible jinja filters to make it easier for release-tools to parse the information
2. Avoid triggering pipelines if no pending post-migrations were detected on the deployer

added AutoDeploy DeliveryP4 post-deploy migrationsphase 1 teamDelivery labels

added to epic &585 (closed)

mentioned in merge request gitlab-org/release-tools!1839 (merged)

added workflow-infraTriage label

marked this issue as related to #2362 (closed)

changed title from Adjust the post-migration sent by deployer to release-tools to Adjust the post-migration information sent by deployer to release-tools

mentioned in epic &585 (closed)

assigned to @mattfield

added workflow-infraIn Progress label and removed workflow-infraTriage label

added Delivery teamBuild label

Thanks for tackling this one @mattfield. Please shout if you have any questions about Deployer, or anything else

@mayra-cabrera Now that I've had a good look through the way things are set up, the different jobs and projects, etc, I've made a start on improving some of the formatting specifically coming from the Ansible play itself so that release-tools doesn't have to do so much parsing.

Before going too far down any particular route, I'm wondering how "pre-formatted" we'd like the list of post-deployment migrations (PDMs) should be before it's POSTed to release-tools. Does anywhere currently - or could potentially in the future - use the info output by the play?

@mattfield I don't think this information will be used by the play, the main purpose of it is to be sent to release-tools so it can be stored and used later.

The final output described on https://ops.gitlab.net/gitlab-com/gl-infra/deploy-tooling/-/merge_requests/398 looks great! Two comments:

1. Dropping the down status is a great idea, it simplifies the output a lot! We just need to update release-tools to make it aware of that change since it uses that keyword as separator.
2. On the deployer side, could we modify the job script to only trigger a pipeline on release-tools if there are pending PDMs? Currently, we're triggering pipelines even when no PDMs are found (example).

@mayra-cabrera Thanks for the feedback! I'm working on an MR for release-tools at the moment.

Given that the PDMs are being spun out into their own pipeline and will be run manually in the future, is it still worth adding in a guard to the job script for safety's sake?

@mattfield You mean if there are no pending post-migrations? If that's the case, yes please, that would help simplify the logic in release-tools.

@mayra-cabrera Yes, exactly. Even though we're moving to have the separate PDM pipeline run manually by release managers for the time being it'll eventually be triggering automatically, so I reckon it's still worth spending the extra bit of effort to gate the extra processing from running ahead of the change.

@mattfield The pipeline that is triggered from deployer to release-tools is not the PDM pipeline though. This is one stores the pending post migrations info as an artifact to later be used by the PDM pipeline.

With that, I do think deployer should only trigger a release-tools pipeline with pending post-migrations:

1. Saves CI resources - Triggering a pipeline without pending post migrations means a pipeline, that is not doing anything, is triggered on every canary deployment, removing this one could lead to CI resource saving.
2. Facilitates debugging - Finding a specific job is easier when dealing with delimited concentration of pipelines. A lot of pipelines are being triggered daily on release-tools (gitlab.com) and release/tools (ops) due to the auto-deploy processes, for them we have some monitoring which makes these easier to track; but for the rest, you need to either use the pipeline UI or filter the logs using elastic search.
3. We could simplify the Artifact class scenarios to assume the class will be executed when post migrations are sent.

@mayra-cabrera Ah of course, that’s my mistake. Apologies Mayra, I’m still trying to piece together a lot of these moving parts which are quite tricky to thread together, so I’m conflating two separate workflows here (or conflating where they overlap at least)

As mentioned I do agree completely for all the reasons you have helpfully enumerated. Wondering whether we should gate at this stage was predicated on my incorrect knowledge about these pipelines. I’ll move onto that once the current tasks are tested and rounded off.

Thanks @mattfield! They're tricky indeed As discussed it may be better to postpone the gate at this stage, quoting from the slack conversation:

The problem Reuben and I discussed was that bypassing running the pipeline when there are no migrations would lead to an issue whereby post_deployment_migrations/pending.rb searching for the last execution would find a previous execution of the build job which did actually have pending migrations

With that, I think it's fine to move that part to another issue and close this one once we have tested the new changes in a PDM execution.

The new changes are working nicely, they were tested on the last PDM's pipelines execution:

• https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/1315650
• https://ops.gitlab.net/gitlab-org/release/tools/-/pipelines/1311511

With that, can we open up the follow up issue we discussed above and close this one @mattfield? Thanks!

Follow-up: #2489 (closed)

Closing this one out.

added 1 deleted label

mentioned in merge request gitlab-org/release-tools!1927 (merged)

changed milestone to %15.2

marked this issue as related to #2484 (closed)

mentioned in issue #2484 (closed)

added teamOrchestration - old label label and removed teamDelivery label

mentioned in issue #2489 (closed)

closed

marked this issue as related to #2489 (closed)

added teamDelivery-Releases label and removed teamOrchestration - old label label