Skip to content

Create means to view webpack-dev-server memory metrics over time

We have had webpack-dev-server memory measurements in place as a CI job since gitlab-foss!31537 (merged), but these metrics are merely displayed in a job log and not stored anywhere.

Because of this it is a tedious chore to view these metrics in context. Here is a graph I created by opening up a series of pipelines from the master branch, finding the webpack-dev-server job, and manually copying its output into a Google spreadsheet table:

webpack-dev-server memory over time
unnamed

We need a mechanism to assemble these metrics into a time-series graph automatically, similar to the model used in https://leipert-projects.gitlab.io/is-gitlab-pretty-yet/jest

@leipert outlined some basic ideas for implementing this in a comment on the original MR, here: gitlab-foss!31537 (comment 208721797)

So in bash the logic looks like this:

curl  --fail --location --silent --show-error --output artifact.txt "$URL"
analyze >> artifact.txt

And then having the artifact.txt as an export again. (See for example here](https://gitlab.com/leipert-projects/is-gitlab-pretty-yet/blob/41f678b5fa694fbbb4361a4a0890d00c36b48df9/analysis.sh#L125-133)

I would probably download the previous report in the script step, because that makes the code a bit more sync:

webpack-dev-server:
  extends: .dedicated-no-docs-no-db-pull-cache-job
  dependencies:
    - compile-assets
    - compile-assets pull-cache
  before_script: []
  variables:
    WEBPACK_MEMORY_TEST: "true"
  script:
    - curl  --fail --location --silent --show-error --output /tmp/webpack-report.json "$ARTIFACT_URL"
    - node --version
    - node --expose-gc node_modules/.bin/webpack-dev-server --config config/webpack.config.js

And then do in your apply function:

try {
  const prevReport = require("/tmp/webpack-report.json");
  prevReport.push({
    timestamp: Math.round(new Date().getTime() / 1000),
    memory,
    commit: process.env.CI_COMMIT_SHORT_SHA
  });

  // Maybe need to switch A/B
  prevReport.sort((a, b) => a.timestamp - b.timestamp);

  fs.writeFileSync("./memory-report.json", JSON.stringify(data));
  process.exit(1);
} catch (e) {
  console.warn(`Could not append result to memory report: ${e.message}`);
  process.exit(1);
}

The mechanism we develop here can also be used in #34098 (closed)

Deliverable:

  • Create a means to collect all webpack-dev-server heap measurements across pipelines on the master branch
  • Create a way to view these metrics in a time-series graph, perhaps exported to gitlab-pages along with our webpack-bundle-analyzer output.
Edited by Mike Greiling