Skip to content

data-encoding inside tezos/tezos

What

This MR makes Octez use the local data-encoding instead of the external one that is released on opam.

Why

This is part of the monorepoisation effort.
It is intended to simplify work on Octez, Etherlink, etc. which requires modifications of data-encoding. The whole code being in the same repository means a single MR can modify both the underlying library and the code using it.

How

Companion opam-repo MR: opam-repository!506

In previous MRs we added data-encoding's source code to the repo and added it to the Manifest to generate its dune and .opam files. It was put in a separate Manifest profile so that octez-deps.opam would be left untouched. The md2mld dependency (needed to build data-encoding's doc) was removed as well so that the lock file would be untouched. The data-encoding directory was marked as only containing data files in dune so that dune would ignore it. So the first step is to undo some of this:

  • put back md2mld
  • put data-encoding in the Manifest profile of Octez (removes opam/virtual/data-encoding.opam, merging it into opam/virtual/octez-deps.opam)
  • remove from data-only files

Then we move data-encoding into the octez-libs opam package. This is so that we don't have a data-encoding opam package when we release Octez, as it would conflict with the one that already exists in opam. We'll tackle the issue of releasing data-encoding separately on opam later.

Data-encoding depended on node.js and thus on npm, but this is not what we want in Octez. We don't want npm and node.js in all CI Docker images. Instead, in Octez npm is installed only by tests that need it. So we remove the explicit dependency on npm. If we release data-encoding separately, this dependency will need to be added back, but this is a problem for the future.

Once data-encoding is seen by dune, dune will try to build its documentation. So we have to add data-encoding to the list of files to copy in the CI jobs that build the doc.

Once the non-external data-encoding is used to build Octez, it must be released in the tarball of Octez. So we add data-encoding to octez-source-content.

Once the non-external data-encoding is used to build Octez, we must make sure that modifications to data-encoding will not break Octez. So we add the data-encoding directory to the list of files that trigger Octez tests.

Also, we want data-encoding tests themselves to run in the CI as well. So we add a job for that. Because those tests are not Tezt tests, the job has to be separate for now. Porting some tests to Tezt would be possible in the future, but data-encoding uses crowbar for some tests and it is unclear how to run crowbar tests in Tezt.

The CI told me to update Nix's flake.lock so I did that.

Finally, we generate all files, port the lock file to tezos/opam-repository, and use the resulting commit hash in scripts/version.sh so that the CI can run with data-encoding's dependencies.

Manually Testing this MR

The CI should cover a lot. Test releases should be done as well. In particular one should try to build Octez with the release tarballs.

Edited by Romain

Merge request reports