Backport CI performance tweaks to 18.08
Context
We still need to maintain 18.08 for quite some time. The CI pipeline is doing unnecessary work.
Description
Let's backport !1589 (merged) and !1553 (merged) to 18.08
Acceptance Criteria
Less unnecessary downloading in VM stage, maybe also other stages.