Initialise the script and README file

Tanvir Roshidrequested to merge
tanvir/init/reproducibility/script into main

Description

buildstream-reprotest-script efficiently executes the buildstream-reprotest, speeding up the testing phase for developers who wish to test the reproducibility of a given Buildstream element.

The buildstream-reprotest counterpart executes the reproducibility phase less efficiently because it compiles the element’s dependencies from scratch in the second phase, while simultaneously running diffoscope against each dependency. Whilst there is nothing incorrect about this approach, it is time-consuming because of this exact approach.

buildstream-reprotest-script aims to solve this by building the element once with its dependencies, and only rebuilding the required element in the second phase and executing diffoscope against it once. This approach saves a significant amount of time when testing the reproducibility of a given Buildstream element.

We should note that both approaches provide a way to assess the reproducibility of software, while ours is a more efficient way to quickly obtain results.

Consequently, this means that our script asks the following question: If we rebuild this exact element in isolation, is it reproducible? The buildstream-reprotest script poses the question: Is the entire dependency graph reproducible at every level?

Thus, our script validates the final artefact, not the chain that produced it. Therefore, if a dependency is non-reproducible but its output artifact happens to be identical when reused, then our script will mark the element as reproducible, whereas the buildstream-reprotest marks the element as non-reproducible.

Fundamentally, this script considers target-level reproducibility testing and is supposed to supplement the original buildstream-reprotest script, rather than being a complete replacement.

Ultimately, this script should be used as a fast, preliminary check and not as a substitute for full dependency graph reproducibility testing.

Testing

This section validates that the script produces expected results when compared against known reproducibility outcomes from the freedesktop-sdk CI results.

Note that the following instance of freedesktop-sdk, i.e., 22177ec91f4e33abd0c17310e037f7317c697c5d, has been used for testing because it contains elements that have failed the reproducibility check.

The original artefacts, pipeline and associated job reside here:

The aim here is to assert the status of all five elements associated with the report and their statuses when using our script:

  1. components/podman.bst
  2. components/_private/ovmf.bst
  3. components/skopeo.bst
  4. components/git-lfs.bst
  5. components/nasm.bst

In addition to the five non-reproducible examples above, we need to assert that our script is able to detect reproducible elements from the original report. We aimed to use the tests/reproducible-test.bst element because this is the same element that freedesktop-sdk uses in its CI testing. However, the element fails to check out because it doesn't have the correct config to do so (freedesktop-sdk/freedesktop-sdk#1959 (comment 3222795026)). Instead, we pick five arbitrary elements from the original report to assert that reproducible elements remain reproducible with our script.

We aim to upload the associated diffoscope report to provide credibility for each test.

Non-reproducible examples

components/podman.bst

components_podman.bst.zip

components/_private/ovmf.bst

components_private_ovmf.bst.zip

components/skopeo.bst

components_skopeo.bst.zip

components/git-lfs.bst

components_git_lfs.bst.zip

components/nasm.bst

Executing diffoscope and storing the results inside of: /home/tanvirroshid/freedesktop-sdk/results_06_04_2026_16_24_00. The two directories used for comparison are /tmp/tmp.yjrZdnIGdS and /tmp/tmp.EEy30fx9bo.
Reproducibility test executed for: components/nasm.bst which is reproducible.

components/nasm.bst was marked as reproducible, which suggests that the non-reproducibility reported by the original script may originate from one of its dependencies rather than the element itself.

Reproducible examples

components/zip.bst

Executing diffoscope and storing the results inside of: /home/tanvirroshid/freedesktop-sdk/results_06_04_2026_16_30_19. The two directories used for comparison are /tmp/tmp.mO0F2o7i2R and /tmp/tmp.NwrXRxj9pZ.
Reproducibility test executed for: components/zip.bst which is reproducible.

components/libgcrypt.bst

Executing diffoscope and storing the results inside of: /home/tanvirroshid/freedesktop-sdk/results_06_04_2026_16_34_02. The two directories used for comparison are /tmp/tmp.t9B7I92nap and /tmp/tmp.TjdbzR5c3d.
Reproducibility test executed for: components/libgcrypt.bst which is reproducible.

components/perf.bst

Executing diffoscope and storing the results inside of: /home/tanvirroshid/freedesktop-sdk/results_06_04_2026_16_58_00. The two directories used for comparison are /tmp/tmp.sizW6hy0WG and /tmp/tmp.KE20Yahx3c.
Reproducibility test executed for: components/perf.bst which is reproducible.

components/webp-pixbuf-loader.bst

Executing diffoscope and storing the results inside of: /home/tanvirroshid/freedesktop-sdk/results_06_04_2026_17_00_39. The two directories used for comparison are /tmp/tmp.vvj6DVgvfF and /tmp/tmp.kub25QiZ5y.
Reproducibility test executed for: components/webp-pixbuf-loader.bst which is reproducible.

components/systemd-libs.bst

Executing diffoscope and storing the results inside of: /home/tanvirroshid/freedesktop-sdk/results_06_04_2026_17_04_06. The two directories used for comparison are /tmp/tmp.1KlX2rTi22 and /tmp/tmp.T2LDvoDZDR.
Reproducibility test executed for: components/systemd-libs.bst which is reproducible.

oci/toolbox-oci.bst

This is an extra element, but it pulls in 625 elements from freedesktop-sdk that we can help use to verify the correctness of our script.

Executing diffoscope and storing the results inside of: /home/tanvirroshid/freedesktop-sdk/results_06_04_2026_18_44_54. The two directories used for comparison are /tmp/tmp.mB6MWoAgjb and /tmp/tmp.TTNFkqvT3q.
Reproducibility test executed for: oci/toolbox-oci.bst which is reproducible.

Notes for oci/toolbox-oci.bst

Definition

A fresh run in this context refers to clearing the ~/.cache/buildstream directory.

Summary

On a fresh run, the oci/toolbox-oci.bst element appears to be marked as non-reproducible, as can be seen in this report: oci_toolbox_oci.bst.zip.

Our understanding is as follows: Phase 1 pulls the artefact from the remote cache, after which point, the target element is deleted and rebuilt locally using the cached dependencies. The comparison is initially based on a remotely produced artefact and a locally produced artefact, resulting in differences due to some OCI metadata causing the initial diffoscope check to fail. The report clearly indicates a SHA256 difference in one of the OCI layers.

On subsequent runs, i.e., without clearing the cache, the previously built local artifact is reused during Phase 1 instead of pulling from the remote cache. As a result, the comparison is now between two locally produced artefacts, making the output reproducible.

In summary, the initial non-reproducibility is caused by differences between remotely produced and locally produced artefacts whenever dealing with a first run. Once everything is cached locally, the builds are deterministic, and reproducibility passes consistently.

Edited by Tanvir Roshid

Merge request reports