Skip to content

Resolve "Add replication support for OCI Buildkit cache manifests"

Valery Sizov requested to merge 365858-add-replication-support-for-oci-buildkit-cache-manifests into master

What does this MR do and why?

Buildkit creates OCI incompatible images where "fat" manifest references blobs directly instead of referencing other manifests. The Geo-replication of those images was broken.

Example of Buildkit image

{
   "schemaVersion":2,
   "mediaType":"application/vnd.oci.image.index.v1+json",
   "manifests":[
      {
         "mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
         "digest":"sha256:47ab09d42056df167f4389cbfc7aa0bfaa0477c3895ed15cc34fee4241565c4e",
         "size":24803024,
         "annotations":{
            "buildkit/createdat":"2022-06-17T16:44:22.638028085Z",
            "containerd.io/uncompressed":"sha256:65feea9638f81cb1fab4ede714f970bb8453cd1a2aa23860d2bb3fdcb960068b"
         }
      },
      {
         "mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
         "digest":"sha256:7b7958bf3d0bf26b5830737a7c3d3601f29685ec8eab72543ebf3a737ce724ba",
         "size":99,
         "annotations":{
            "buildkit/createdat":"2022-06-17T16:44:28.738659213Z",
            "containerd.io/uncompressed":"sha256:ff4b415032cbffbbfb8e5c424371a1f84b12d7126bd58834fcd4dbaccbf00c33"
         }
      },
      {
         "mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
         "digest":"sha256:815feacf7fac47c668909978efb6ab3c1c309dda0bfc8166446ea49a6d887600",
         "size":299,
         "annotations":{
            "buildkit/createdat":"2022-06-17T16:44:28.755771088Z",
            "containerd.io/uncompressed":"sha256:dfe34d3ffe0427c16bb3cd8b9c49332366417aec595b67376116087c93ada006"
         }
      },
      {
         "mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
         "digest":"sha256:9981e73032c8833e387a8f96986e560edbed12c38119e0edb0439c9c2234eac9",
         "size":2716477,
         "annotations":{
            "buildkit/createdat":"2022-06-17T16:44:22.631065335Z",
            "containerd.io/uncompressed":"sha256:4f4ce317c6bbf55719e49973d32d33ba456d7cb08693a6d6fb372690eacee23b"
         }
      },
      {
         "mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
         "digest":"sha256:aa621928f77470ac279d0f61322fb0ffb9240d19a2b95cfbe3f61f092a89a529",
         "size":449,
         "annotations":{
            "buildkit/createdat":"2022-06-17T16:44:22.668059835Z",
            "containerd.io/uncompressed":"sha256:417f394ac0af61fcd518b0735d64c9a6b3f22f0a5bd7f8d570f54546d1845f43"
         }
      },
      {
         "mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
         "digest":"sha256:cafbf807fb6eae72cf9bec4f8e408d28f40cac82c707b599378564a434033bfd",
         "size":68652816,
         "annotations":{
            "buildkit/createdat":"2022-06-17T16:44:28.691009046Z",
            "containerd.io/uncompressed":"sha256:61fa4dbd08f2b4f91825923de98ba995d76ac17378e0b01cf8745e01968a73f9"
         }
      },
      {
         "mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
         "digest":"sha256:d291a579f2606d866152338f224fc9c9b19aa71b0af0749c8b3427f30e510e8b",
         "size":2438687,
         "annotations":{
            "buildkit/createdat":"2022-06-17T16:44:22.650505668Z",
            "containerd.io/uncompressed":"sha256:85f6aec46b48a48a269186654643f318cc18f2de01805ff8e9558231693d5791"
         }
      },
      {
         "mediaType":"application/vnd.oci.image.layer.v1.tar+gzip",
         "digest":"sha256:e21bc908289503f4d322fa3ce3a451c6feba8e5e8eef4b8655b3e718a3393d22",
         "size":524,
         "annotations":{
            "buildkit/createdat":"2022-06-17T16:44:29.005423255Z",
            "containerd.io/uncompressed":"sha256:7353c1566cef853eab4bcb266ec3baf8b2734e0a6303f3a9606f215b130100dc"
         }
      },
      {
         "mediaType":"application/vnd.buildkit.cacheconfig.v0",
         "digest":"sha256:1ad9bc55eaf02d732d314d01cf381b4dda5ba563e1a169c66e869b10f6d6d56e",
         "size":1753
      }
   ]
}

How to set up and validate locally

I wasn't able to check it locally because buildkit cache registry can't be insecure. Having a local registry under HTTPS doesn't make it secure either. So I ended up setting up a regular 1k Geo environment on remote instances.

buildctl build {registry-repository} \
  --output type=image,name=vsizov-primary.gogitlab.xyz:5050/root/test,push=true \
  --export-cache type=registry,ref=vsizov-primary.gogitlab.xyz:5050/root/test:buildcache \
  --import-cache type=registry,ref=vsizov-primary.gogitlab.xyz:5050/root/test:buildcache \
  --frontend=dockerfile.v0 --local context=. --local dockerfile=.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #365858 (closed)

Edited by Valery Sizov

Merge request reports