Skip to content

Add an opam lock file and a script to generate/update it

Romain requested to merge nomadic-labs/tezos:romain-lock into master

(aims to replace parts of !7026 (closed) eventually)

What

This MR adds a script: update_opam_lock.sh which, when ran, creates opam/virtual/octez-deps.opam.locked. This lock file is similar to opam/virtual/octez-deps.opam except that it contains the transitive closure of Octez dependencies, and all versions are constrained using equalities.

Why

This is a first step towards removing the tezos/opam-repository machinery.

tezos/opam-repository is, currently, an opam repository; more precisely, it is a subset of ocaml/opam-repository which only contains what Octez devs, Octez users and the Octez CI need, at specific version numbers so that everyone use the same dependencies. We believe that using a lock file to specify the version of our dependencies will be simpler than using this machinery, and that it will be more flexible.

For instance, one will be able to update the version of a single dependency by manually editing the lock file. When a dependency needs to be updated just before a release, having to update the world is scary; being able to only update this particular dependency is reassuring. This also allows to update only the dependencies we actually need in release branches, instead of all of them; release branches are supposed to not change too much, and this should also be true for dependencies.

So this MR introduces this lock file, and a way to generate it. In future MRs, we will be able to:

  • use this lock file to replace the current implementation of make build-deps so that building Octez does not rely on tezos/opam-repository except in the CI;
  • change the implementation of tezos/opam-repository so that it is not an opam repository (to make the lock file the only source of truth).

How

The script can be summarized as follows:

  • tell opam to use a specific commit from the public opam repository, namely full_opam_repository_tag from version.sh;
  • tell opam to install our dependencies;
  • tell opam to lock those dependencies into a lock file.

We ask opam to install both Octez dependencies and dev dependencies, so that it finds a solution that is compatible with both. Unfortunately, this has the side-effect of including utop (and its dependency xdg) in the lock file, because Octez depends on pyml, which optionally depends on utop. And opam lock take all dependencies, optional or not, that are currently installed. To fix this, we simply run opam remove utop in the script.

We start from a fresh opam switch so that already-installed packages have no impact, so that we can use --fake to avoid actually installing the packages, so that we don't have to use --criteria which would also mean that we would have to remove packages with avoid-version, and also just so that the script has no impact on the current switch.

Note that the resulting list of packages is not exactly what is in tezos/opam-repository. See !12029 (comment 1779084724) for a discussion about why this is the case. But the summary is that tezos/opam-repository contains everything that might be needed by devs and by the CI; some of these are not needed just to build Octez.

Manually testing the MR

scripts/update_opam_lock.sh
git diff

git diff should return an empty diff.

Checklist

  • Provide automatic testing (see the testing guide).
  • Select suitable reviewers using the Reviewers field below.
  • Select as Assignee the next person who should take action on that MR
Edited by Romain

Merge request reports