Investigate top 2 slowest API endpoints for Code Review

Context

In https://gitlab.com/gitlab-org/gitlab/-/issues/523857, we identified the top 2 slowest endpoints owned by groupcode review which and causied the most apdex budget consumption.

  1. GET /api/:version/projects/:id/merge_requests
  2. GET /api/:version/groups/:id/merge_requests

This is confirmed to still be the case as of last week: https://gitlab.com/gitlab-com/create-stage/weekly-status-check-in/-/issues/110#note_2593833499

Goal

  1. Investigate impact of the current performance (e.g. percentiles of duration for above-target queries, # of namespaces/users impacted, etc)
  2. Investigate ways to improve the performance of the endpoints

Kibana Dashboard

https://log.gprd.gitlab.net/app/r/s/tkljo

This dashboard includes the following data:

  • Total slow requests (broken down by project/group API endpoints)
  • Total slow requests with search (broken down by project/group API endpoints)
  • Unique count of slow requests by root namespace
  • Unique count of slow requests by UA
  • Top 10 root namespaces
  • Top 10 UA
  • Top 10 users
  • Request rate of top 5 root namespaces
    • This shows total request rate for the 2 endpoints including non-slow ones.
  • Duration of slow project list MRs API requests (median, P75, P95, P99)
  • Duration of slow group list MRs API requests (median, P75, P95, P99)
  • Time spent on each component
    • This shows median time spent on DB, redis, gitaly, and view.
    • One caveat with view_duration_s is that it can include time spent not on actual rendering but method calls that were called during serialization of the entity
Edited by Patrick Bajao