Parallelize Record VCR jobs and fix daily VCR-free pipeline

Problem

We're currently recording VCR cassettes manually via one of the record_vcr_* jobs. Once one of these jobs is finished, refreshed VCR cassettes are downloadable as an artifact. It is then the responsibility of the developer to download and push these cassettes manually.

Proposal

Add a new schedule that periodically triggers pipelines running all record_vcr_* and also pushing the updated cassettes to the CustomersDot repository.

A couple of considerations:

• Before allowing CI to push to the main branch, we should test out the automation to a dedicated (protected?) branch. Push to the repository will be then made possible via a deploy key.
• It may be too much to re-record VCR cassettes every night. Maybe once a week may be enough.

This issue's scope changed over the past few months, given what needed to be accomplished from a VCR improvements point-of-view:

Parralelizing

Automating (ie. parallelizing) the three record_vcr_* jobs will allow us to meet requirements for this OKR. Today, the main record_vcr job still times out, but beyond that, there's no added value in having all the recording done in one single job, as VCR creates separate folders for each spec/**/* test. To put it another way, there won't be any overlap between each split job.

Actually, the only constraint that three split jobs present (compared to the main record_vcr) is that we're spending three time the amount in pulling the Docker image from the registry.

Daily pipeline creating notification

Via Investigate feature test failures, when no VCR ... (#8327 - closed), we're aiming to fix the VCR-free daily pipeline. This now-functioning pipeline allows us to spot any potential change that recorded VCR cassettes would not allow us to catch.

As a result, this process proposes a better way to highlight changes rather than relying on a "automate and push" process. Also, developers will be notified more easily about needs to re-record manually now since failed daily VCR-free pipeline are now reported in the s_fulfillment_engineering Slack channel.

Result

Periodically updated VCR cassettes.