Daily empty VCR cassettes jobs get cancelled by a more recent pipeline

Problem

There is a daily scheduled pipeline to run RSpec with empty VCR cassettes. This run will help to ensure that all specs that use VCR are no issues with Zuora in these specs exist and that they are re-recordable at any given time.

The VCR recording is part of three different pipeline jobs. Recently, it was noticed that two of these jobs got cancelled (this and this job) while the third completed. After some investigation, the most likely scenario for this is that a new pipeline after an MR got merged was created. No Slack message was posted as part of the daily pipeline run either (since it never made it that far).

Investigation details

In this screenshot, the pipeline on the bottom is the daily empty VCR cassettes one. The pipeline (second from the top) with the status running is the merge pipeline that seems to have cancelled the first pipeline due to being more newer and therefore making the first pipeline redundant.

Screenshot_2024-09-19_at_4.13.32_PM

Looking at the timestamps for the pipelines seem to confirm this theory:

Proposal

While the job is configured with interruptable, we still want to somehow ensure that the daily empty VCR cassettes run are finished by completing the jobs without any cancellation. Avoid parallel jobs that send real requests to Zuora at the same time are also something to look out for to avoid timeouts. These can happen if E2E tests, any other VCR recording job within an MR or the daily empty VCR cassette run run at the same time. This also came up in this discussion.

As part of the parallel job problem, removing the interruptable config isn't the correct solution. Another thing to explore for a possible solution is around an automatic retry if a job was cancelled in the scheduled pipeline. Either way, this issue is to figure out a way to make the daily empty VCR cassettes run complete and reliable at all times.

Result

Ensure daily empty VCR cassette pipeline completes at any given time without being interrupted by a newer pipeline.