Manifest: lib_protocol

Context

This MR updates the dune and OPAM files of proto_*/lib_protocol by using the manifest generator. One challenge about this MR was figuring out a sane review strategy since there would be many changes, and I wanted reviewers to be able to focus on what actually changed.

In order to review this MR, please follow the instructions below, it will be much easier.

Changes

The changes to manifest/main.ml and manifest/manifest.ml in this MR should in theory produce identical/equivalent results as what currently exists on master. However, there are a few changes that were necessary/noteworthy:

1. tezos-protocol-compiler - The replace.exe used to apply the template for generating the dune files. This logic was moved over to the manifest generator. Therefore replace.exe itself became obsolete. This will be reflected by the following regular changes:
   • Many OPAM packages used to depend on tezos-protocol-compiler. This dependency is no longer needed. Furthermore this dependency was used in the build section of the opam file. Those build steps have alse been removed.
   • Many runtest stanzas in the dune files used to call out to replace.exe to generate dune.inc files. These have also been removed.
2. Makefile - The generate_dune target of the Makefile was removed because it's usage was replaced by make -C manifest.
   • The reason I did not simply change generate_dune to call make -C manifest is because it conflicts with some CI jobs. In particular, there are CI jobs that delete protocols in order to improve the build time. After deleting, they invoke make, which would then invoke make -C manifest, and since the directories were deleted, the behavior was undefined.
   • Instead, I think it would be better for developers to explicitly call make -C manfest whenever they modify the manifest. In fact, we already have a job in CI to check that make -C manifest does not produce a diff. Any CI job which runs after that job can have full confidence that the build scripts are all generated.
3. scripts/snapshot_alpha.sh - The call to replace.exe was no longer necessary. Its function is now covered by snapshot_alpha_and_link.sh which calls both snapshot_alpha.sh and link_protocol.sh. The link_protocol.sh script takes care of updating manifest/main.ml in order to generate the dune/opam files for the new snapshot.

Manually testing the MR

The diff as is shown using conventional diff tools (including the diff here on GitLab) is unfortunately not easy to read. However, with these tips, I think reviewing will be much easier. For one, please note that previously replace.exe used to generate dune.inc files, but now the manifest generator only creates dune (and opam) files. In the commit history for this branch, you will see that the penultimate commit (HEAD~1) merges the dune.inc files with old dune files. This is necessary because in the final commit we completely regenerate all of those dune files. This makes it possible to use the dune diff tool to compare the final two commits:

# Checkout _this_ branch
git checkout richard-lib-protocol

# Get a copy of the diff tool locally
git checkout richard-manifest-diff -- manifest/diff

# Run the diff tool on all the dune and opam files against HEAD~1
dune exec manifest/diff/main.exe -- HEAD~1 HEAD $(find src -name dune -o -name '*opam') | less -R

Finally, I should point out that none of my changes affect the protocol hashes. You can use the following command to verify it for yourself:

ls src | grep proto_ | while read line; do
  cat src/lib_protocol_compiler/final_protocol_versions | grep -q $(./tezos-protocol-compiler -hash-only src/$line/lib_protocol)
  echo "$line     $?"
done

The expected output is:

proto_000_Ps9mPmXa     0
proto_001_PtCJ7pwo     0
proto_002_PsYLVpVv     0
proto_003_PsddFKi3     0
proto_004_Pt24m4xi     0
proto_005_PsBABY5H     0
proto_005_PsBabyM1     0
proto_006_PsCARTHA     0
proto_007_PsDELPH1     0
proto_008_PtEdo2Zk     0
proto_008_PtEdoTez     0
proto_009_PsFLoren     0
proto_010_PtGRANAD     0
proto_011_PtHangz2     0
proto_012_Psithaca     0
proto_013_PtJakart     0
proto_alpha     1
proto_demo_counter     1
proto_demo_noops     1
proto_genesis     1

Checklist

• Document the interface of any function added or modified (see the coding guidelines)
• Document any change to the user interface, including configuration parameters (see node configuration)
• Provide automatic testing (see the testing guide).
• For new features and bug fixes, add an item in the appropriate changelog (docs/protocols/alpha.rst for the protocol and the environment, CHANGES.rst at the root of the repository for everything else).
• Select suitable reviewers using the Reviewers field below.
• Select as Assignee the next person who should take action on that MR}