Speed up Review Apps deploy step

Right now we have a 70-80 pending jobs waiting for a Review App job to build (https://gitlab.com/gitlab-com/www-gitlab-com/-/jobs?scope=pending). This is blocking merge requests from being merged.

We only have one runner that can pick up these jobs, and it usually takes 5 minutes/job to:

  1. Download the artifacts.zip
  2. Extract the ZIP to a directory
  3. rsync it to the right place

How can we speed this up? Some ideas:

  1. Shrink the artifacts.zip file where possible (i.e. make images, LFS objects, etc. smaller)
  2. Add more parallel runners (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8203)
  3. Deploy the Review Apps to object storage directly from the build step
  4. Send a delta of the artifacts instead of the full blown copy?
  5. Can we make the Review App an optional build step (i.e. you can merge without it passing?)

@axil @alejandro What do you think?

Edited by Stan Hu