Store auto-explain logs in the package registry
What does this MR do and why?
Part of Query Reporting: Identify new queries introduce... (#417789 - closed)
In this MR we begin to store auto-explain logs in the package registry for master builds
By creating this centralized artifact storage, we'll be in a position to diff query sets against master to identify new queries introduced in merge requests, supporting both query auditing and future database partitioning compatibility efforts as part of Automated Query Analysis for Merge Requests (&8440)
References
- Automated Query Analysis for Merge Requests (&8440)
- Query Reporting: Identify new queries introduce... (#417789 - closed)
- a450e83d used as reference for this implementation
How to set up and validate locally
Take a sample of the merge-auto-explain-logs and place in your local directory at auto_explain e.g. https://gitlab.com/gitlab-org/gitlab/-/jobs/9463212353/artifacts/browse/auto_explain/
ls -alR ./auto_explain
total 17048
drwxr-xr-x@ 9 john staff 288 Mar 20 00:53 .
drwxr-xr-x@ 134 john staff 4288 Mar 20 01:39 ..
-rw-r--r--@ 1 john staff 6148 Mar 19 17:41 .DS_Store
-rw-r--r--@ 1 john staff 7953775 Mar 19 19:07 auto_explain.ndjson.gz
-rw-r--r--@ 1 john staff 674330 Mar 19 19:07 jsonb_column_scans.ndjson.gz
...
...
....# Setup the script (these vars would typically be set via .gitlab-ci.yml / CI VARS)
export CI_SERVER_HOST="gdk.test:3000"
export GITLAB_WORKHORSE_FOLDER="gitlab-workhorse"
export CI_PROJECT_NAME="foobar"
export NODE_ENV="test"
export CI_PROJECT_PATH=""
export RSPEC_AUTO_EXPLAIN_LOG_PATH="auto_explain/auto_explain.ndjson.gz"
# setup according to data in your own GDK instance
export API_PACKAGES_BASE_URL="http://gdk.test:3000/api/v4/projects/21/packages/generic"
export CI_JOB_TOKEN=$GITLAB_QA_ACCESS_TOKEN
export CURL_TOKEN_HEADER='PRIVATE-TOKEN'
# source the scripts
source scripts/packages/helpers.sh
source scripts/gitlab_component_helpers.sh
# run the script
export AUTO_EXPLAIN_PACKAGE="auto-explain-logs.tar.gz"
export AUTO_EXPLAIN_PATH="${RSPEC_AUTO_EXPLAIN_LOG_PATH}"
export AUTO_EXPLAIN_PACKAGE_URL="${API_PACKAGES_BASE_URL}/auto-explain-logs/master/${AUTO_EXPLAIN_PACKAGE}"
UPLOAD_PACKAGE_FLAG=true create_and_upload_auto_explain_package
# verify that package uploaded to in our case 'http://gdk.test:3000/projects/21/-/packagesMR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.