[CNG] Investigate pipeline runtime, and possible improvements to speed

Summary

During the mitigation of a recent incident related to release tooling, we noticed that most Tag pipelines (for the Debian based images) on Dev run for approximately 50-55 minutes. This includes release tags, as well as auto-deploy tags.

We should look into these pipelines and their runtime, to investigate what we could possible do to decrease that runtime, where possible.

Current behavior

Tag pipelines run for 50+ minutes 👓

Expected behavior

Tag pipelines run for < 50 minutes 😸

Versions

v15.1.0 / 935c7f447f9b7b37ad857cbd713c23d771970c3e of CNG.

Relevant logs

See https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines?scope=tags

• all non-UBI images (not -ubi8, non -fips) are 50-55 minutes of runtime.
• all UBI images (-ubi8, -fips) are 75-95 minutes of runtime.

Cases in point:

tag	pipeline	runtime
v15.1.0-ee	https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/242241
v15.1.0	https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/242421
15.1.202206160620+802fc92ebc3	https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/241651	52m54
v15.0.3-ee	https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/241691	57m05
v15.0.3	https://dev.gitlab.org/gitlab/charts/components/images/-/pipelines/241690	52m21

Note: You may see longer time summaries on the pipeline's page, as it calculate the total runtime of all jobs, including failed / retried.