[CNG] Revisit how we compute CONTAINER_VERSION

Summary

We use CONTAINER_VERSION to essentially "map" the versions of various components are in an image, including dependencies, and that given component. The calculation of this value has a few flaws. Refactoring this could make the process much clearer.

  1. These don't take into account sum changes to underlying / base containers.
    • Example: an updated debian:bullseye-slim does not necessarily result in a rebuild of all containers built upon that container.
  2. Calculation requires re-computation in every job definition.
    • Side note: This is absolutely horrendous for manual builds by contributors.

Current

Right now, the general logic of how we compute CONTAINER_VERSION is as follows

  • TARGET_VERSION is essentially git ls-tree HEAD -- <directory of the component> and using the third column (object ref) from the output.

  • For leaf components (things that doesn't depend on any other images, gitlab-python or gitlab-graphicsmagic for example)

    • export CONTAINER_VERSION=${TARGET_VERSION}${<component version from variables.yml>}

  • For components which depend on other images

    • CONTAINER_VERSION is essentially shasum of a mix of versions of all the images this component depends on, along with TARGET_VERSION and component version from variables.yml (if any)

    • This requires us to re-compute versions of all the dependencies using get_version in the script. For example, this is what we do for gitlab-shell

      ruby_version=$(get_version gitlab-ruby)
go_dir_version=$(get_version gitlab-go)
go_version=($(echo -n "$BASE_VERSION$go_dir_version$GO_VERSION" | sha1sum))
logger_target_version=$(cat artifacts/container_versions/gitlab-logger_tag.txt)
export git_container=$(cat artifacts/container_versions/git-base_tag.txt)
gomplate_version=$(cat artifacts/container_versions/gitlab-gomplate_tag.txt)
export GITLAB_LOGGER_VERSION=($(echo -n "$go_version$logger_target_version$GITLAB_LOGGER_VERSION$(date -u +%D)" | sha1sum))
export CONTAINER_VERSION=($(echo -n "$BASE_VERSION$go_version$git_container$gomplate_version$TARGET_VERSION$GITLAB_SHELL_VERSION$(date -u +%D)" | sha1sum))

Proposal

Change computation of CONTAINER_VERSION to something like the following

export TARGET_VERSION=$(get_target_version)
export DEPS_VERSIONS=$(cat artifacts/container_versions/*.txt)    # Because of needs, only the necessary artifacts will be present here.
export CONTAINER_VERSION=$(echo -n ${TARGET_VERSION}${component_version_from_variables_yml}${DEPS_VERSIONS} | sha1sum)
echo -n ${CONTAINER_VERSION} > artifacts/container_versions/${CI_JOB_NAME}.txt

By this method, the earlier gitlab-shell example becomes

export TARGET_VERSION=$(get_target_version)
export DEPS_VERSIONS=$(cat artifacts/container_versions/*.txt)
export CONTAINER_VERSION=$(echo -n ${TARGET_VERSION}${GITLAB_SHELL_VERSION}${DEPS_VERSIONS} | sha1sum)

PS: I would also like us to rename TARGET_VERSION to something else. Something that clearly tells us that this is the git object applicable to the component.

Edited by Jason Plum