Store auto-explain logs in the package registry

John McDonnellrequested to merge
jmd/dump-auto-explain-logs-to-package-registry into master

What does this MR do and why?

Part of Query Reporting: Identify new queries introduce... (#417789 - closed)

In this MR we begin to store auto-explain logs in the package registry for master builds

By creating this centralized artifact storage, we'll be in a position to diff query sets against master to identify new queries introduced in merge requests, supporting both query auditing and future database partitioning compatibility efforts as part of Automated Query Analysis for Merge Requests (&8440)

References

How to set up and validate locally

Take a sample of the merge-auto-explain-logs and place in your local directory at auto_explain e.g. https://gitlab.com/gitlab-org/gitlab/-/jobs/9463212353/artifacts/browse/auto_explain/

ls -alR ./auto_explain 
total 17048
drwxr-xr-x@   9 john  staff      288 Mar 20 00:53 .
drwxr-xr-x@ 134 john  staff     4288 Mar 20 01:39 ..
-rw-r--r--@   1 john  staff     6148 Mar 19 17:41 .DS_Store
-rw-r--r--@   1 john  staff  7953775 Mar 19 19:07 auto_explain.ndjson.gz
-rw-r--r--@   1 john  staff   674330 Mar 19 19:07 jsonb_column_scans.ndjson.gz

...
...
....
# Setup the script (these vars would typically be set via .gitlab-ci.yml / CI VARS)
export CI_SERVER_HOST="gdk.test:3000"
export GITLAB_WORKHORSE_FOLDER="gitlab-workhorse"
export CI_PROJECT_NAME="foobar"
export NODE_ENV="test"
export CI_PROJECT_PATH=""
export RSPEC_AUTO_EXPLAIN_LOG_PATH="auto_explain/auto_explain.ndjson.gz"


# setup according to data in your own GDK instance
export API_PACKAGES_BASE_URL="http://gdk.test:3000/api/v4/projects/21/packages/generic"
export CI_JOB_TOKEN=$GITLAB_QA_ACCESS_TOKEN
export CURL_TOKEN_HEADER='PRIVATE-TOKEN'


# source the scripts
source scripts/packages/helpers.sh
source scripts/gitlab_component_helpers.sh


# run the script
export AUTO_EXPLAIN_PACKAGE="auto-explain-logs.tar.gz"
export AUTO_EXPLAIN_PATH="${RSPEC_AUTO_EXPLAIN_LOG_PATH}"
export AUTO_EXPLAIN_PACKAGE_URL="${API_PACKAGES_BASE_URL}/auto-explain-logs/master/${AUTO_EXPLAIN_PACKAGE}"

UPLOAD_PACKAGE_FLAG=true create_and_upload_auto_explain_package

# verify that package uploaded to in our case 'http://gdk.test:3000/projects/21/-/packages

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by John McDonnell

Merge request reports