Add heap dump diagnostic report implementation
What does this MR do and why?
Refs #370077 (closed)
This is the last step in a sequence of MRs that provide us with the ability to collect object space ("heap") dumps from worker processes.
Specifically, this builds on the following MRs (not including refactors):
-
Add a new life-cycle hook on_worker_stop
that is called when a Puma or Sidekiq worker is about to shut down (!103372 (merged)) -
Wiring: Leverage memory-watchdog
to signal the worker that it should dumpObjectSpace
before shutting down (!103957 (merged)). The actual implementation was just a dummy. -
Compress reports when streaming to disk: !105115 (merged) -
This MR: Fill in the method body of the HeapDump
report to actually produce a heap dump and puts this behind anops
toggle (we want to enable/disable this selectively.)
Note that this has no changelog entry because:
- It is a SaaS-only feature currently and guarded by an extra environment switch
- Uses an ops toggle that defaults to
off
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
How to set up and validate locally
- Follow the instructions in https://gitlab.com/gitlab-org/application-performance-team/team-tools/-/blob/master/DIAGNOSTIC_REPORTS.md
- Set
Feature.enable(:report_heap_dumps)
- Wait or force a worker to violate a memory threshold; this should trigger a shutdown and trigger this report.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #370077 (closed)
Merge request reports
Activity
changed milestone to %15.7
assigned to @mkaeppler
mentioned in issue #370077 (closed)
added 1 commit
- 3f966329 - Add heap dump diagnostic report implementation
- A deleted user
added backend feature flag labels
2 Warnings ed70f315: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines. featureaddition and featureenhancement merge requests normally have a documentation change. Consider adding a documentation update or confirming the documentation plan with the Technical Writer counterpart.
For more information, see:
- The Handbook page on merge request types.
- The definition of done documentation.
1 Message CHANGELOG missing: If you want to create a changelog entry for GitLab FOSS, add the
Changelog
trailer to the commit message you want to add to the changelog.If you want to create a changelog entry for GitLab EE, also add the
EE: true
trailer to your commit message.If this merge request doesn't need a CHANGELOG entry, feel free to ignore this message.
Reviewer roulette
Changes that require review have been detected!
Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:
Category Reviewer Maintainer backend Josianne Hyson (
@jhyson
) (UTC+13, 12 hours ahead of@mkaeppler
)Alexandru Croitor (
@acroitor
) (UTC+2, 1 hour ahead of@mkaeppler
)To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.
If needed, you can retry the
danger-review
job that generated this comment.Generated by
Dangeradded featureaddition label
- Resolved by Nikola Milojevic
Allure report
allure-report-publisher
generated test report!e2e-package-and-test:
test report for ed70f315expand test summary
+---------------------------------------------------------------------------+ | suites summary | +----------------------+--------+--------+---------+-------+-------+--------+ | | passed | failed | skipped | flaky | total | result | +----------------------+--------+--------+---------+-------+-------+--------+ | Analytics | 4 | 0 | 0 | 0 | 4 | ✅ | | Plan | 120 | 0 | 0 | 0 | 120 | ✅ | | Govern | 84 | 0 | 0 | 0 | 84 | ✅ | | Create | 318 | 0 | 10 | 0 | 328 | ✅ | | Package | 0 | 0 | 6 | 0 | 6 | ➖ | | Manage | 130 | 0 | 10 | 0 | 140 | ✅ | | Verify | 104 | 0 | 6 | 2 | 110 | ❗ | | Secure | 6 | 0 | 10 | 8 | 16 | ❗ | | Fulfillment | 4 | 0 | 28 | 0 | 32 | ✅ | | Configure | 0 | 0 | 6 | 0 | 6 | ➖ | | Release | 12 | 0 | 0 | 0 | 12 | ✅ | | ModelOps | 0 | 0 | 2 | 0 | 2 | ➖ | | Version sanity check | 0 | 0 | 2 | 2 | 2 | ➖ | | Monitor | 2 | 0 | 0 | 0 | 2 | ✅ | +----------------------+--------+--------+---------+-------+-------+--------+ | Total | 784 | 0 | 80 | 12 | 864 | ❗ | +----------------------+--------+--------+---------+-------+-------+--------+
- Resolved by Nikola Milojevic
@alipniagov mind reviewing this?
requested review from @alipniagov
requested review from @nmilojevic1
removed review request for @alipniagov
- Resolved by Matthias Käppler
- Resolved by Nikola Milojevic
@nmilojevic1
, thanks for approving this merge request.This is the first time the merge request is approved. To ensure full test coverage, a new pipeline will be started shortly.
For more info, please refer to the following links:
added pipeline:mr-approved label
enabled an automatic merge when the pipeline for 413ac58a succeeds
mentioned in merge request gitlab-com/gl-infra/k8s-workloads/gitlab-com!2388 (merged)
mentioned in commit 63789a8e
mentioned in issue #385175 (closed)
mentioned in merge request gitlab-org/ruby/gems/prometheus-client-mmap!74 (merged)
added workflowstaging-canary label and removed workflowin dev label
added workflowcanary label and removed workflowstaging-canary label
added workflowstaging label and removed workflowcanary label
added workflowproduction label and removed workflowstaging label
added workflowpost-deploy-db-staging label and removed workflowproduction label
added workflowpost-deploy-db-production label and removed workflowpost-deploy-db-staging label
added releasedcandidate label
added releasedpublished label and removed releasedcandidate label
mentioned in issue #423894 (closed)
mentioned in issue gitlab-org/quality/triage-reports#20631 (closed)
mentioned in issue gitlab-org/quality/triage-reports#20992 (closed)
mentioned in issue gitlab-org/quality/triage-reports#21559 (closed)
mentioned in issue gitlab-org/quality/triage-reports#22049