Add reports uploader script
What does this MR do and why?
This implements the Diagnostic Reports uploader, which will deliver reports produced by Puma to the GCS.
We will run this script as a main process in the container build from the gitlab-toolbox-ee
image.
It will have an access to the Shared mount, via emptyDir
, to where Puma saves the reports.
Example helm chart change: gitlab-org/charts/gitlab!2791 (diffs).
You could see in this example how we will fire up the script.
A similar change will be done in https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com. gitlab-org/charts
will not be changed, the MR opened against it is for testing in development.
How to set up and validate locally
- Create your own GCP project via Sandbox Cloud Realm (guide)
- You will set
GITLAB_DIAGNOSTIC_REPORTS_PROJECT
env var to the project name (example below) - Create a GCS bucket in this project
- You will set
GITLAB_DIAGNOSTIC_REPORTS_BUCKET
env var to the bucket name (example below) - Create a service account, so you could download the JSON key for auth. Download the key and store it securely.
- You will point
GITLAB_GCP_KEY_PATH
to the key. - Create or pick any empty directory that will be scanned for reports.
- You will point
GITLAB_DIAGNOSTIC_REPORTS_PATH
to it. - For testing, you will set
GITLAB_DIAGNOSTIC_REPORTS_UPLOADER_SLEEP_S
env var to some small value (e.g. 3 seconds), so you would scan for uploads fast. - Enable
gitlab_diagnostic_reports_uploader
Feature Flag - Run the command below, replacing env var values with your own:
GITLAB_DIAGNOSTIC_REPORTS_UPLOADER_SLEEP_S=3 GITLAB_DIAGNOSTIC_REPORTS_BUCKET=diag_reports GITLAB_DIAGNOSTIC_REPORTS_PROJECT=alipniagov-c14e6039 GITLAB_GCP_KEY_PATH=/Users/al/dev/gitlab-keys/gcs/gcloud.json GITLAB_DIAGNOSTIC_REPORTS_PATH=/Users/al/dev/tmp-diag-reports-uploads bundle exec rails runner bin/diagnostic-reports-uploader
- This will start the
UploadAndCleanupReports
checking for reports in the designated dir in a sleep loop - Copy any file into the directory under
GITLAB_DIAGNOSTIC_REPORTS_PATH
. You could try different file sizes. I tested 1M, 50M, 500M, and 1G files. - The file should be uploaded and removed from the dir. Check your GCS bucket.
- Check
tail -f log/application_json.log
forupload requested
andduration_s
inupload success
- You could put some
raise StandardError, 'abc'
beforeFile.open
and checktail -f log/exceptions_json.log
that it is logged
If you want to test it in GKE, running from the container
- Set up GCP project, GCS bucket, get the key
- Pull gitlab-org/charts/gitlab!2791 (closed)
- Add the GCP secret to the GKE project: https://cloud.google.com/kubernetes-engine/docs/concepts/secret#creating_secrets_from_files
- I run
kubectl create secret generic diagnostics-gcloud --from-file ~/dev/gitlab-keys/gcs/gcloud.json
- In the branch (gitlab-org/charts/gitlab!2791 (diffs)), replace
DIAGNOSTICS_PROJECT
andDIAGNOSTICS_BUCKET
with your own - Set GITLAB_DIAGNOSTIC_REPORTS_UPLOADER_SLEEP_S as some minimal value (e.g. 3)
- Run the deployment
- My example (use your own ips):
helm upgrade --install gitlab . \
--set global.hosts.domain=34.171.154.148.nip.io \
--set global.hosts.externalIP=34.171.154.148 \
--set certmanager-issuer.email=me@example.com \
-f diag.yaml
- Enable
gitlab_diagnostic_reports_uploader
Feature Flag (via rails console on the instance)
Then you could shell to the gitlab-webservice-default
pod, into toolbox-uploader
container and put the file in /var/tmp/gitlab/diagnostics
directory (touch
, for example). The file will be uploaded and removed from that dir. It will appear in GCS.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #372242 (closed)