Assets job is sometimes missing dependencies
Problem statement
After deploying https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/1129406 completed, QA was failing, due to assets that were not preset in our bucket. Reference: production#6757 (closed)
This was a repeat of production#5925 (closed) except that we didn't roll anything back. The next package contains all expected assets and resolved the problem.
Investigation findings
We know that something may cause a silent failure during javascript dependencies fetching.
The two failures we identified were related to vendors-highlight.language.js
not being present at the time of building the assets.
Because webpack is only packaging stuff it can find, the resulting assets may be different from the CNG image and the assets container we are using to upload the assets on the bucket.
It is important to notice that having a SSOT for this will not help here, because a single pipeline can also break, in that case, we will not see the 404 but javascript errors and broken pages on the browser.
Next step
We need to identify what is installing vendors-highlight.language.js
and why it fails sometimes.
We should make sure that if a dependency fails to build the whole pipeline must fail.