Skip to content

Track release metadata of auto-deploys as JSON files in a separate project

Yorick Peterse requested to merge track-release-data-in-json into master

This adds support for tracking release metadata when running auto-deploys. This metadata includes data such as: the version, name, SHA, and ref of every released component. Data is stored separately for Omnibus and CNG releases, and spread across different directories for different major versions. This ensures no single directory will end up having too many files for the underlying file system to handle.

Tracked data is stored at https://ops.gitlab.net/gitlab-org/release/metadata. Since metadata may include security related information (e.g. the SHA of a security commit), we can't include this in any public projects (e.g. the release-tools project on GitLab.com). Using a separate repository allows us to work around this.

Tracking is hidden behind the feature flag "release_json_tracking", allowing us to disable the feature should there be any problems with it.

We currently only support tracking of data for auto-deploys as a first step. Since the code for stable releases is quite different, we will update said code in a separate merge request.

See gitlab-com/gl-infra/delivery#690 (closed) for more information.

Example output

{
  "security": false,
  "releases": {
    "omnibus-gitlab": {
      "version": "978aff",
      "sha": "978aff",
      "ref": "foo",
      "tag": false
    },
    "gitlab": {
      "version": "467abc",
      "sha": "467abc",
      "ref": "foo",
      "tag": false
    },
    "gitaly": {
      "version": "12.9.0",
      "sha": "e3587eb85ea97eb3d69085e7144ac00f8cc6028c",
      "ref": "v12.9.0",
      "tag": true
    }
  }
}

The path for this file would be releases/cng/12/12.9.0.json. The file name is the name of the CNG/Omnibus tag. This means that given an auto-deploy version (as already passed around as an environment variable) it should be trivial to get the associated data. For this you simply need:

  1. The category ("cng" or "omnibus")
  2. The tag name

Given the tag name you then split it by ., get the first component, and use this as the directory name in which the JSON file is stored (within the category's directory).

In the JSON output, the "version" field never has a "v" prefix. Code that needs this prefix should add it manually when they retrieve the data and use it.

Testing

Testing has been performed on a separate branch (called foo) on the metadata project. This was done by running the tracking code directly instead of running the CNG/Omnibus taggers, so it's possible (but unlikely) this may break; fortunately we can just turn the feature flag off in that case.

Edited by Yorick Peterse

Merge request reports