RFC: CI: Profile opam jobs

Context

Not for merging.

Just to store a set of tools that can be used to profile the opam jobs in the CI (or indeed, locally).

I was curious to find out why the opam jobs are so slow. This MR contains the scripts:

• scripts/profile-opam/install-wrapper.sh: this script updates the local opam switch configuration to wrap each build, remove, or install call that opam executes with scripts/profile-opam/profile-opam.sh.
• scripts/profile-opam/profile-opam.sh: this script measures the time taken for the command in argument, and writes a CSV line to $OPAM_WRAP_LOG (defaults to /tmp/opam-wrap.log) with this information.
• scripts/profile-opam/plot.py a python script that visualizes a log obtained through the scripts above.

Usage

Locally

First of all, you need to disable sandboxing in your opam configuration. Open ~/.opam/config and comment out the wrap-*-commands lines.

Then run scripts/profile-opam/install-wrapper.sh from the root of your switch. Note that this script is not idempotent and running multiple times might mess something up.

Now, run an opam command, e.g. opam install tezt. After it terminates, the file at $OPAM_WRAP_LOG should be non-empty. You can inspect it as is, the format is:

${CI_PIPELINE_ID},${CI_JOB_ID},${CI_JOB_NAME},$typ,$*,$start_date,$duration"

the first three columns will be empty locally.

To visualize the log, you must first install the python libraries pandas and matplotlib:

poetry add pandas matplotlib

Then, run:

poetry run python scripts/profile-opam/plot.py <LOG_FILE>

<LOG_FILE> defaults to /tmp/opam-wrap.log.

In the CI

The branch of this MR is patched to generated profiling logs in each opam job. The logs will be found in the artifacts of each job.

Here's the visualization of tezos-wasmer and tezos-baking-alpha (sorry for the illegible plots, I don't konw how to fix that easily. If you run the commands locally you get an interactive plot viewer where you can zoom in on stuff.):

Manually testing the MR

Checklist

• Document the interface of any function added or modified (see the coding guidelines)
• Document any change to the user interface, including configuration parameters (see node configuration)
• Provide automatic testing (see the testing guide).
• For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
• Select suitable reviewers using the Reviewers field below.
• Select as Assignee the next person who should take action on that MR