Investigate heavily CPU, slow endpoints

While investigating production#3433, I was looking into CPU heavy endpoints and thought that we should focus on addressing these.

https://log.gprd.gitlab.net/goto/811bbb2973af0d7da68c040cc5eb4342

https://log.gprd.gitlab.net/goto/a1b4e07d89730f1280fc8c8ad25e68bc

One of the reasons why these requests are problematic is that, in a Ruby environment with a GIL, we see spill-over of poor performance from these requests to other requests being handled by the same process at the same time, due to GIL contention.

Summary of investigation 2021-02-15

• Projects::CompareController - Big page to render. Similar pages use pagination. This could be ported to the compare view. see comment thread
• Projects::ForksController - Bad for large namespaces but only 13 instances out of 7000 were longer than 10s.
• Projects::MargeRequest::CreationsController - 0.05% were slow.
• Groups::MilestonesController - The slow ones are gitlab-org/gitlab and the output is simply very very big. Quick-win to lower the limit.

Summary of investigation 2021-03-11

Next Steps	Status	json.meta.caller_id.keyword: Descending	Count	Median json.cpu_s	99th percentile of json.cpu_s
Conversations continue on the issue	⏸ The tab shows too many issues Issue gitlab-org/gitlab#325512 (closed) Discussion #840 (comment 496729848)	Groups::MilestonesController#issues	440	13.188575029373200	27.343913269043000
Conversation continues on the issue	⏸ Big files get re-rendered frequently Issue #949 (closed) Dicussion #840 (comment 519089722)	Projects::BlobController#show	419	16.963513946533200	45.82170635223390
Conversations continue on the issue	⏸ Issue #950 (closed) Issues has too many notes. Occur with @gitlab-qa's issues. Investigation #840 (comment 519011093)	/api/:version/projects/:id/issues/:issue_iid	4708	11.968472367349800	18.17736568450930
No further action at this time	⏸ A handful of users have too many projects Investigation #840 (comment 519032786)	/api/:version/users/:user_id/projects	2202	14.455781169732400	29.926130142211900
Conversation continues on the Epic	🏃 Epic gitlab-org&5390 (closed) Discussion #840 (comment 496718145)	Projects::CompareController#show	3812	18.746006535895100	46.79319671630860
Conversation continues on the Epic	👆	Projects::CommitController#show	8589	12.981461648122500	34.53196852684030
Conversation continues on the Epic	👆	Projects::MergeRequests::CreationsController#diffs	1654	20.15095072448570	59.792599487304700
Conversation continues on the Epic	👆	Projects::MergeRequests::DiffsController#diffs_batch	1439	13.740001234974900	50.35572544097900
No further action at this time	❓ Problem occurs for 2 domains only. Will not take further action at this time. Investigating at #840 (comment 519045082)	/api/:version/internal/pages	1059	27.944854819585400	36.197709426879900
No further action at this time	⏸ Occur only with gitlab-* users	/api/:version/groups/:id	1030	12.533604576474100	19.247128295898400
No further action at this time	⏸ Occur only with gitlab-* users	/api/:version/groups/:id/projects	1013	15.273215675354000	21.938460006713900
No further action at this time	⏸ Occur only with 1 repository belong to Gitlab	/api/:version/projects/:id/repository/tags	994	10.594236798719900	14.242667503356900
Done	✅ Done #840 (comment 519052729)	/api/:version/runners	714	24.82361831665040	33.58497772216800
No further action at this time	⏸ Occurs only with 1 repository	/api/:version/groups/:id/registry/repositories	665	11.240019416809100	13.561281538009600
Work happening on related issue	🏃 gitlab-org/gitlab#322386 (closed)	Projects::PipelinesController#create	620	15.583755027680200	27.645602989196800
Work happening on related issue	🏃 https://gitlab.com/gitlab-org/gitlab/-/issues/321788#note_520871433	Projects::PipelinesController#index	528	12.965385595957400	30.861846275329600
GraphQL visibility on related issue	❓ Lack of observability	GraphqlController#execute	494	13.733983008066800	40.84306274414060