Skip to content

Upload diagnostic reports to GCS: `curl` uploader

Aleksei Lipniagov requested to merge 372242-uploader into master

What does this MR do and why?

This MR continues !97155 (merged), replacing no-op uploader with a real one.

Other notes

Self-managed is not affected because you need to set GITLAB_DIAGNOSTIC_REPORTS_ENABLED. Currently, we are not working on enabling this for self-managed.

Local dev env is not affected unless you set GITLAB_DIAGNOSTIC_REPORTS_ENABLED to start with. Also, you need FF enabled.

⚠ Prerequisite to flip FF in prod: Obtain and configure GCS bucket for diagnostic reports (#372239 (closed)) - I asked SRE help for that. Otherwise we'll be logging errors.

⛳ FF rollout issue: #372771 (closed)

How to set up and validate locally: Uploader only

  1. I tested on a real GCS bucket: and created my own GCP project via Sandbox Cloud Realm (guide)

  2. Open tail -f log/application_json.log to check log records on success uploads or any issues

  3. Run gcloud auth print-access-token or copy your token from GCS UI.

  4. In rails c: token = "YOUR_TOKEN"; bucket = "YOUR_BUCKET_NAME"; path = "file/you/upload.json"; u = Gitlab::Memory::ReportsUploader.new(token: token, bucket: bucket); u.upload(path)

  5. The file should be uploaded to the bucket, and also you will see a log record

How to set up and validate locally: Jemalloc reports + Uploader (full integration)

You will need a real GCS bucket, as described above.

We do not produce Jemalloc reports on OSX, because to pull the actual Jemalloc report, libjemalloc must be on LD_PRELOAD (more).

Fortunately, it is already configured in GCK this way. So it is recommended to use GCK if you want to test it manually.
Alternatively, you could hack lib/gitlab/memory/jemalloc.rb, so it would produce some bogus file not related to Jemalloc stats.
For the light-weigh testing (just uploader), see the section above.

First, you need to set ENV vars to enable Jemalloc reports:

      GITLAB_DIAGNOSTIC_REPORTS_ENABLED: 'true'
      GITLAB_DIAGNOSTIC_REPORTS_SLEEP_S: '10'
      GITLAB_DIAGNOSTIC_REPORTS_SLEEP_BETWEEN_REPORTS_S: '1'
      GITLAB_DIAGNOSTIC_REPORTS_SLEEP_MAX_DELTA_S: '0'
      GITLAB_DIAGNOSTIC_REPORTS_PATH: '/home/git/gitlab/tmp/diag-reports'

Then, ENV vars for the uploader:

      GITLAB_DIAGNOSTIC_REPORTS_UPLOADER_SLEEP_S: '10'
      GITLAB_DIAGNOSTIC_REPORTS_ENABLED: 'true'
      GITLAB_DIAGNOSTIC_REPORTS_GCS_TOKEN: 'YOUR-TOCKEN'
      GITLAB_DIAGNOSTIC_REPORTS_GCS_BUCKET: 'YOUR-BUCKET-NAME'

You could put them into gitlab-compose-kit/docker-compose.override.yml

Second, you need to enable FF for Jemalloc: Feature.enable(:report_jemalloc_stats).
Then, FF for the uploader: Feature.enable(:gitlab_diagnostic_reports_uploader).

Restart the webserver after you set ENV vars FFs.

The reports will be produced and uploaded, you should also notice that the /tmp subdir under the configured GITLAB_DIAGNOSTIC_REPORTS_PATH should appear.
Check GCS bucket for the uploads.
Check tail -f log/application_json.log in case of any errors.

In my GCK, I was hitting "curl: option --fail-with-body: is unknown\ncurl: try 'curl --help' or 'curl --manual' for more information\n", although I expect us to have the latest curl in prod: https://gitlab.com/gitlab-org/build/CNG/-/blob/master/gitlab-base/Dockerfile#L14.
So I needed to remove this line temporarily for local env.
My local GCK uses 7.74, the option was introduced in 7.76. Probably rebuilding GCK will help?

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #372242 (closed)

Edited by Aleksei Lipniagov

Merge request reports