Skip to content

Add reports uploader script

Aleksei Lipniagov requested to merge 372242-toolbox-script into master

What does this MR do and why?

This implements the Diagnostic Reports uploader, which will deliver reports produced by Puma to the GCS.
We will run this script as a main process in the container build from the gitlab-toolbox-ee image.
It will have an access to the Shared mount, via emptyDir, to where Puma saves the reports.

Example helm chart change: gitlab-org/charts/gitlab!2791 (diffs).
You could see in this example how we will fire up the script.
A similar change will be done in https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com. gitlab-org/charts will not be changed, the MR opened against it is for testing in development.

How to set up and validate locally

  • Create your own GCP project via Sandbox Cloud Realm (guide)
  • You will set GITLAB_DIAGNOSTIC_REPORTS_PROJECT env var to the project name (example below)
  • Create a GCS bucket in this project
  • You will set GITLAB_DIAGNOSTIC_REPORTS_BUCKET env var to the bucket name (example below)
  • Create a service account, so you could download the JSON key for auth. Download the key and store it securely.
  • You will point GITLAB_GCP_KEY_PATH to the key.
  • Create or pick any empty directory that will be scanned for reports.
  • You will point GITLAB_DIAGNOSTIC_REPORTS_PATH to it.
  • For testing, you will set GITLAB_DIAGNOSTIC_REPORTS_UPLOADER_SLEEP_S env var to some small value (e.g. 3 seconds), so you would scan for uploads fast.
  • Enable gitlab_diagnostic_reports_uploader Feature Flag
  • Run the command below, replacing env var values with your own:
  • GITLAB_DIAGNOSTIC_REPORTS_UPLOADER_SLEEP_S=3 GITLAB_DIAGNOSTIC_REPORTS_BUCKET=diag_reports GITLAB_DIAGNOSTIC_REPORTS_PROJECT=alipniagov-c14e6039 GITLAB_GCP_KEY_PATH=/Users/al/dev/gitlab-keys/gcs/gcloud.json GITLAB_DIAGNOSTIC_REPORTS_PATH=/Users/al/dev/tmp-diag-reports-uploads bundle exec rails runner bin/diagnostic-reports-uploader
  • This will start the UploadAndCleanupReports checking for reports in the designated dir in a sleep loop
  • Copy any file into the directory under GITLAB_DIAGNOSTIC_REPORTS_PATH. You could try different file sizes. I tested 1M, 50M, 500M, and 1G files.
  • The file should be uploaded and removed from the dir. Check your GCS bucket.
  • Check tail -f log/application_json.log for upload requested and duration_s in upload success
  • You could put some raise StandardError, 'abc' before File.open and check tail -f log/exceptions_json.log that it is logged

If you want to test it in GKE, running from the container

helm upgrade --install gitlab . \
  --set global.hosts.domain=34.171.154.148.nip.io \
  --set global.hosts.externalIP=34.171.154.148 \
  --set certmanager-issuer.email=me@example.com \
  -f diag.yaml
  • Enable gitlab_diagnostic_reports_uploader Feature Flag (via rails console on the instance)

Then you could shell to the gitlab-webservice-default pod, into toolbox-uploader container and put the file in /var/tmp/gitlab/diagnostics directory (touch, for example). The file will be uploaded and removed from that dir. It will appear in GCS.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #372242 (closed)

Edited by Aleksei Lipniagov

Merge request reports