Extend performance bar with overall statistics
Problem to solve
Performance bar is a very useful tool for page performance analysis. A downside is that currently it shows performance statistics for loading only the single page and these data are lost/expired after few minutes. It would be useful to build aggregated statistics about profiled requests so users could easily identify potential issues across all profiled requests.
Intended users
Personas are described at https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/
Proposal
Extend peek with storing/updating also overall stats. When storing peek request we would also update following overall stats (stored in Redis, expiration e.g. 24 hours) with:
- for each SQL/gitaly query in the peek output we bump overall counter, overall time and how many times per single request it was called (we would keep only location of the query - first backtrace line, instead of the query itself to make sure these data are anonymous and no sensitive data could leak), same for gitaly and elasticsearch calls
- add additional counters for other performance bar data in next iterations (e.g. bullet warnings or frontend stats)
Then we would expose these data in the performance bar widget - add a link "overall stats", after clicking this it would show these overall stats - so we would show these data only to users who can use performance bar.
So user could easily identify for example methods which most often call SQL query (or gitaly), showing also its aggregated time.
The benefit is that it would be an additional analytic tool which would help Gitlab developers identify specific places in the code which cause performance issues and which can't be identified in Kibana logs - especially slow SQL queries, or N+1 issues in SQL or gitaly calls. Although this wouldn't be perfect because it would gather statistics only from users having performance bar enabled, it would be still valuable to have performance data across many requests.