Draft: Switch to opam-monorepo

This MR introduces an opam-monorepo workflow to build Tezos!

Opam-monorepo is a tool to assemble code with its dependencies into a self-contained dune workspace. This is an alternative to installing dependencies into an opam switch. That new tool separates package management from building the resulting source code. It is an opam plugin that: (i) creates a lock file for the project dependencies; (ii) downloads and extracts the dependency sources locally; and (iii) sets up a Dune Workspace so that dune build builds everything in one go.

In particular, here are some gains for Tezos developers over the current workflow:

• Use a lock file to simplify updating tezos' dependencies. tezos/opam-repository is not needed anymore (so no need to update it and wait for the CI jobs to complete before pushing updates on tezos/tezos).
• Use dune cache for all OCaml code, that leads to finer build caching and thus faster local builds (and soon for CI builds once we've updated it to support the new workflow, more details on this bellow).
• Less failure modes. Opam's solver is not used anymore when calling make build-deps which leads to a more deterministic workflow.

The MR is a work in progress but it is already working. It includes a generated tezos.opam.locked lockfile and an updated make build-deps that you can easily use to setup an opam-monorepo based dev environment.

The following will get you started:

git fetch git@gitlab.com:NathanReb/tezos.git opam-monorepo
git checkout opam-monorepo
make monorepo-build-deps # create a local switch and a dune workspace
dune build # or `make`
dune runtest # or `make test`

Note that this will replace your current local switch if you had one.

Next steps

There are a few follow up steps before marking this as ready for merging:

1. Collect feedback to make sure we have not overlooked anything.
2. Properly document the new workflow. We need to update every bit of documentation so it up to date with opam-monorepo.
3. Adapt the CI. We need to rewrite parts of the CI scripts so that they use the lockfile and opam-monorepo for the builds.
4. Decide how to deal with dev-tools. Not all dev tools are locked at the moment. Some such as ocamlformat are locked and pulled in the duniverse, some others such as odoc are not but could likely be and finally others like merlin will still have to be installed via opam. At the moment the non locked dev-tools are installed via opam as they were before. Locking as many dev-tools as possible is probably the best approach here.

Related Issues

• Revamp tezos opam-repository
• Use opam lock instead of custom opam repo

---

The rest of the description covers in a bit more detail the opam-monorepo workflow and the changes introduced by that MR.

Lock file

The main addition of the MR is the tezos.opam.locked at the root of the project. It contains a fixed version for all of the transitive dependencies of all locally defined packages (i.e. all .opam file in the tezos tree) and allows to ensure all developpers use the same strict set of dependencies without relying on tezos/opam-repository anymore.

Updating your dev environment

To update your dev environement (whenever the lock file has changed), run:

make monorepo-build-deps

Some of those dependencies still have to be installed via opam, such as dune, the compiler or some specific dependencies such as tezos-rust-libs. This is taken care of by make monorepo-build-deps but it can be done manually by running:

opam install --deps-only --ignore-pin-depends ./tezos.opam.locked

It represent a very restricted subset of Tezos external dependencies.

The rest of the dependencies' sources must be pulled locally in the dune workspace by opam-monorepo. When you will build parts or all of the Tezos project, the required artifacts will be built by dune as if they were simply part of the repo. Note that opam-monorepo will pull them in the duniverse/ folder at the root of the project and mark them as "vendored" for dune to handle them accordingly. This is also included in make monorepo-build-deps but can be done manually by running:

opam monorepo pull

The duniverse/ is not versioned here and was added to the .gitignore to avoid noise. This means that devs have to remember to regularly run make monorepo-build-deps to ensure their duniverse/ is in sync with the lock file. Transitive external dependencies are also compiled into the lock file. Depext handling in either the opam plugin or opam 2.1 always requires running the solver which is in our case unnecessary. opam-monorepo provides a command to install all depexts of the project:

opam monorepo depexts

It is run as part of make monorepo-build-deps as well.

Upgrading the lock file

Whenever you need a new package or need to bump the version of an existing package, you can run:

make lock

That new make lock introduced in this MR that will generate a new lockfile (and so you will most probaly also need to run make monorepo-build-deps after this).

The lockfile is generated by opam-monorepo using the following command:

opam monorepo lock --recurse --lockfile tezos.opam.locked

By default opam-monorepo locks dependencies of the packages defined at the root of the project. The --recurse flag tells it to recursively look into subdirectories for packages to lock. The --lockfile option tells it where to write the lock file. This is required as opam-monorepo use the package name when there's only one or the project name as defined in dune-project otherwise. Since it's not defined in the tezos repo (for now) we have to be explicit.

When locking, opam-monorepo will use the current switch config for its solver. This means repositories and their priorities along with pins will be taken into account. The opam-monorepo solver will require that all the dependency tree (with a very few exceptions) builds with dune.

Dune ports

opam-monorepo adds some constraints on your dependencies. In particular, the dependencies need to use dune as a build system. Part of the work we have been doing to make Octez build with opam-monorepo is to port all of Tezos' dependencies to dune. The opam repository with the dune port can be found here and the process to build a dune port on the README.

Once the MR is merged, you will have to create a dune port if you want to add a dependency that doesn't use dune. This is the main constraint that comes with opam-monorepo, so let us know if any part of this workflow is unclear. The opam-monorepo team will be happy to help on this as we've gone through a fare amount of dune ports. Tarides is committed to maintain the existing forks, but we also encourage opam-monorepo users to contribute the dune ports they need to help keep it up to date.

Building

Once the dependencies are pulled-in by opam-monorepo, the build steps stay the same:

dune build # or make
dune test # or make test

Do not forget to enable the Dune cache to benefits from incremental builds when switching between branches and local clones of the tezos repository. To do so, you can run DUNE_CACHE=enabled dune build, or add (cache enable) in ~/.config/dune/config. See dune's documentation for more details.

You should be able to build as you usually do using dune. The entire tezos repo can build with this MR, i.e. dune build should successfully complete.

The vast majority of dune runtest is successful but some parts of the test suite fail because of my local setup and missing "Zcash params". I have also noticed some tests timing out.

CI

To adapt the CI, we will be reusing the current design and build a docker image that contains a cache of the build with the dependencies so that the normal CI job will only have to rebuild the source code changes.

Now that the dependencies live on the tezos/tezos repository, the docker image containing the cache will run make build-deps and dune build and cache the _build directory.

For instance, we could save this _build directory in a Docker image that will then be pushed the Docker registry, indexed by the hash of the lock file. The final stage of the CI might look like:

docker pull $CI_REGISTRY_IMAGE:$LOCK_HASH || true
docker build --cache-from $CI_REGISTRY_IMAGE:$LOCK_HASH --tag $CI_REGISTRY_IMAGE:$LOCK_HASH .
docker push $CI_REGISTRY_IMAGE:$LOCK_HASH

We'll be reporting on the progress in the next few weeks.

The MR is also still quite minimal at the moment and there is further clean up to do. In particular, removing all tezos/opam-repository related scripts.