## Top Links #### Tools Overview table https://docs.google.com/spreadsheets/d/1S9_pR90Q79H5p_593Cxo9wofx34rPPky27ZrKNviJaw/edit?usp=sharing #### Developer Tooling Survey https://docs.google.com/forms/d/1xnU5rUXmavylagsknUckPw2uKrc1UiidSdOPm5DcfQQ/edit?usp=sharing Results: https://gitlab.com/gitlab-org/gitlab/-/issues/350153#results ## Context We currently provide a confusing set of tools for performance profiling, with overlapping functionality and no clear guidance on when these tools should be used. We should review which of these tools are in use, add value, are most widely applicable, and whether we can condense them into a more manageable "tool belt". Ideally, we would offer developers a single unified tool that can take both CPU and memory profiles. It should be usable in code and from the web UI and support useful visualizations like flame graphs. ## Expected outcome 1. An instruction (a doc page) for our Engineers which tool is applicable to each case, in which order to try them, how to interpret the results. Basically, we want to rework [this page](https://docs.gitlab.com/ee/development/performance.html#tooling), make it clear and consistent. 2. Deprecate unused tools, fix broken useful tools. 3. Try to reduce the number of tools if there is an intersection. ## Current State Summary [updated] The list will contain only the tools that require attention. It wouldn't include: - The tools and approaches that are indisputable useful and widely used: PerformanceBar, Kibana logs, etc - The tools that are outside of the scope (for the first iteration): e.g. Golang tools, FE-specific tools - Things related to wider processes, such as perf-related Slack channels Refer to a table on the top for a full list. :pencil: To edit the table fast, I edit the data in [google doc](https://docs.google.com/spreadsheets/d/1S9_pR90Q79H5p_593Cxo9wofx34rPPky27ZrKNviJaw/edit?usp=sharing) (`Sheet2`) and run through the [converter](https://tabletomarkdown.com/convert-spreadsheet-to-markdown/). | Tool / Approach | Short description | Status | Notes | | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | -------------------------------------------------------------------------- | | Document when to pick each of the tools, how to interpret the results | [https://gitlab.com/gitlab-org/gitlab/-/issues/349958](https://gitlab.com/gitlab-org/gitlab/-/issues/349958) | Todo - we could work on docs rework in parallel during the next few milestones | We already reworked the Performance page and added an Index linking all performance-related sections across the docs | | Sherlock | Records various queries if configured: Web, ActiveRecord, ActionView.<br>Allows to track the line where it was invoked. | :white\_check\_mark: Removed | | | Gitlab::Profiler<br><br> | enable profiling a GET or POST request to a specific URL, either as an anonymous user (the default) or as a specific user. via ruby-prof. | :white\_check\_mark: OK | We are going to keep it. We just need to describe its place in a workflow. | | Gitlab::RequestProfiler | 2 parts: The ruby-prof profiler and the memory Profiler.<br>Gitlab::RequestProfiler is currently only available for admin users, it is in the admin menu | Blocked: deprecating | We deprecate it and are going to remove in 15.0 | | gitlab-profiler<br>bin/profile-url | builds on Gitlab::Profiler to add some additional niceties, such as allowing<br>configuration with a single YAML file for multiple URLs, and uploading of the<br>profile and log output to S3. | :white\_check\_mark: Archived | Project is archived | | Stackprof | We support taking CPU and object profiles of a worker process with stackprof, which can be triggered via SIGUSR2. | :white\_check\_mark: OK | All fine, we are going to use and have it documented | | Import/Export Performance tool | Uses gitlabhq\_export.tar.gz.<br>Runs the import on the staging via API, and post the results to Slack | :white\_check\_mark: OK | There's an activity in the dedicated Slack channel | | CI jobs tracking memory usage: memory-static | load gems from the Gemfile and check the amount of RAM consumed as well as the number of objects allocated and retained.<br>Aggregated values will be included as 'metrics' into MRs while full reports will be downloadable as job artifacts.. | :white\_check\_mark: Removed | | | CI jobs tracking memory usage: memory-on-boot | very similar but preloads the app in production env, hit it with a single request, and gather total gem memory consumption data using derailed exec perf:mem from derailed\_benchmarks<br> | :white\_check\_mark: OK | We calculate the weighted result to avoid outliers | | CI jobs tracking memory usage: free -m | memory-test (it is not a separate job, wrapper around how we run specs) is to log memory stats after running each spec file and compile the report.<br>Executes free -m after every example group and collects the data across all test jobs in a single CSV for further analysis. It should help detecting memory leaks in specs and other nasty things. Also, top offenders in terms of mem-usage. | :white\_check\_mark: Removed | | | gdk measure<br>gdk measure-workflow | Generate a sitespeed.io report for given URL(s)<br>Generate a sitespeed.io report for given workflow(s) | :white\_check\_mark: OK | We could ask Denys to expand the docs if needed, he mentioned using this tool quite a lot |