Gather `derailed_benchmarks` results as part of Metrics
Problem to solve
Improve visibility of memory usage during our internal (GitLab) development cycle.
Intended users
Further details
Recently we merged the https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28698, which provides the https://github.com/schneems/derailed_benchmarks.
The derailed_benchmarks provide very comprehensive information about the boot memory usage of Rails application, like:
- the code size of gems included in project with breakdown on dependency tree (
bundle exec derailed bundle:mem
), - the boot memory allocation with breakdown, by gems allocated (
bundle exec derailed bundle:objects
), - the memory allocation over time of running application (
https://github.com/schneems/derailed_benchmarks#dynamic-app-benchmarking
) - the memory allocation by generation (
https://github.com/schneems/derailed_benchmarks#i-want-a-heap-dump
),
Proposal
We should consume as much as possible of this information, and:
- Run that as part of GitLab CE
.gitlab-ci.yml
the pipeline, as part of our testing suite, - Generate the machine-readable form that can be consumed by GitLab,
- Include the code size and memory allocation report as of information presented in Merge Request widget, similar to JUnit tests,
- Include the review of these changes as part of our review process during development,
The output of the CI job will likely be metrics report, that is generated as part of CI job. The report will contain metric-like as described https://docs.gitlab.com/ee/ci/metrics_reports.html that shows the information about all Gem allocations and all relevant information.
Example of the report:
gem_total_size 600
gem_total_memory_allocation 100
gem_code_size{name="rails", version="5.2"} 100.0
gem_memory_allocation{name="rails", version="5.2"} 5.0
What does success look like, and how can we measure that?
Enable every developer in GitLab to easily see:
- How much adding new code adds to boot memory of application?
- How much adding a new Gem bumps memory requirements of running GitLab?
- How much memory allocation during application initialization is required to run the components that we are developing?
Example use-case
- Developer adds a new gem to the project. The persona can quickly see how much adding the gem results in higher memory requirements,
- The team (and developer) can constantly measure how adding new code impacts the overall size of the application.
Future extensions
Measure code-size by the team with figuring out which Gems are required by each team. Try to split the codebase into product stages.
Measuring that allows us to make team aware of the cost of the code and all dependencies to provide a subset of the product and make a guided decision on following more microservices approach when required.