Create BBM to convert old policy options format and remove fallback for policy job options

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Why are we doing this work

In !208845 (merged) we've scoped policy job options under a single policy: key. For backwards compatibility with runner and any processing of the previously created jobs, we've kept fallback for the old structure.

Some examples when this is needed:

  1. A manual job is played after a while.
  2. A job is retried after a while.
  3. On-stop actions for environments (which are deferred manual jobs in the pipeline that deployed to the environment) will run when a MR is closed/merged or branch deleted.

Proposal

We should create a BBM to convert the old format to the new one, so that we can migrate the data automatically without impacting users with a breaking change. See a related discussion where it was proposed to create BBM over ci_job_definitions.

We also need to cleanup the json schema validation for options once we complete the BBM: !208845 (comment 2828083346)

Relevant links

Verification steps

Edited by Fabio Pitino