Use named chunks in Webpack to ensure deterministic assets generation

Stan Hurequested to merge
sh-named-chunks-hash into master

What does this MR do and why?

In gitlab-com/gl-infra/production#20271 (closed) we found that for two different auto-deploy versions, a Webpack chunk file had identical filenames but they only differed by a single number at the end of the file.

One file had:

}},[[347,1,0,4,69,90,126]]);

The other had 127:

}},[[347,1,0,4,69,90,127]]);

That final nested array is part of Webpack’s webpackJsonp.push call in Webpack 4’s output format. It lists:

[[<moduleId>, <parentChunkId>, ... ]]

In this example, it appears the chunk ID changed from 126 to 127.

In Webpack 4, the sorting criteria is not deterministic if multiple chunks have the same “weight” (priority). It often falls back to insertion order, which depends on:

  • The order the filesystem returned files for scanning.
  • The order async imports were discovered while parsing modules.
  • The traversal order through Set (which depends on how chunks were added).

To fix this problem for Webpack we need to use named chunks. However, we need a custom function as well to ensure that unnamed async chunks are sorted in a deterministic way.

References

gitlab-com/gl-infra/production#20271 (closed)

How to set up and validate locally

  1. Check out this branch.
  2. Run:
export NODE_OPTIONS=--max-old-space-size=7680
git checkout 42182ad4f20909269ed63801b5010f618c7548cd
yarn webpack-prod
  1. Run tail public/assets/webpack/pages.user_settings.personal_access_tokens.*.chunk.js.

Instead of chunk IDs, we see:

[[347,"runtime","main","commons-pages.admin.application_settings.service_accounts-pages.admin.gitlab_duo.self_hosted-pages.a-41fb7d39","commons-pages.admin.application_settings.service_accounts-pages.admin.credentials-pages.admin.impers-a0f04463","commons-pages.admin.application_settings.service_accounts-pages.admin.impersonation_tokens-pages.gro-87e924fa","commons-pages.admin.impersonation_tokens-pages.groups.settings.access_tokens-pages.projects.settings-e3a3ee4e"]]]);

This seems to increase the total assets directory by 20 MB.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Stan Hu

Merge request reports