Migrate to use docker load/save for helper archives to preserve ENTRYPOINT

The following discussion from !2058 (merged) should be addressed:

  • @steveazz started a discussion: (+4 comments)

    issue: This isn't going to work for us, unfortunately, because how we are building the Docker image.

    We are exporting the image to a tar and when we import it we are going to lose any metadata associated with the image, so we are going to lose the information about the ENTRYPOINT. We can reproduce this issue with make helper-dockerarchive-host and then running docker inspect on the image the entrypoint is going to be null rather the specified one.

    This is explained in https://github.com/moby/moby/issues/8334#issuecomment-57471427

    docker export and docker import don't preserve metadata. This is by design and it's not being changed.

    Now we have to keep exporting the image because we support automatic imports which is useful for offline Runners or Runners that don't have access to the DockerHub registry.

    I see two options that we can achieve, but I'm not sure which one is best

    Apply changes when importing

    We can keep using the tar archive that we create, but then every time we import we have to specify the ENTRYPOINT like docker import --change "ENTRYPOINT [\"/usr/bin/dumb-init\", \"/entrypoint\"]". We need to make this change for make helper-dockerarchive-host and load the image inside of the Docker executor.

    The only and biggest drawback I see is that we have 3 places where we have to update the ENTRYPOINT if we want to change this, which is less then ideal. If we have to update it we have to remember to update it elsewhere, we can add a comment in this line to make the user changing the line aware of it.

    Migrate to save/load

    There is another solution, but it's a long road a lot of things can go wrong and it's a breaking change that we would have to deprecate slowly. Using docker save instead of docker export will save all the metadata of the image, because docker save exports the image and not the container image.

    We would have to do something like the following:

    1. Run both docker export and docker save on the container/image and publish both in our release for example helper-images/prebuilt-x86_64.tar.xz and helper-images/save/prebuilt-x86_64.tar.xz.
    2. Update loadPrebuiltImage or it's caller to call the docker load command instead of docker import if save/prebult-x86_64.tar.xz is presnet.
    3. Load deprecation warning if we use the load on the old tar file that is created by docker export
    4. In a Major release, we just relay on docker save/docker load logic.

    The only drawback that docker save has over docker export is the tar size since it's exporting all the Docker layers separately while docker export flattens everything so the tar file is a lot smaller (we need to measure the difference). We also end up with a bigger package overall since we have to ship both tar files at the moment, the deb/rpm packages have these bundled up, again we need to measure the difference.

    @ggeorgiev_ @tmaczukin @ajwalker @akohlbecker I wonder if we should apply an iterative approach here taking on some ~"technical debt" in favor of shipping this sooner.

    1. We implement the --changes proposal inside of this merge request so that ENTRYPOINT works as expected.
    2. In a follow-up issue we migrate to save/load

Using docker save instead of docker export would also allow us to use the same logic for Windows, which currently doesn't support docker export, and would make running Windows Docker executor tests faster.

Edited by Pedro Pombeiro