TTFB on Issues List is ~700ms

Summary

Despite the fact that time spent in the database (40ms) and Redis (10ms) is very low, and Gitaly and ES is 0, the gitlab-org/gitlab issues list takes between 500ms-1000ms to return the initial HTML page.

Example: https://gitlab.com/gitlab-org/gitlab/-/issues

Since the actual HTML rendered includes the navbar, sidebar holder (or supersidebar) and otherwise empty divs, the problem could be in one of those components. Any improvement here would be likely to improve every Project-scoped page load-time or even site-wide load-times.

Since the issues list is now rendered from the frontend cache, this performance issue is the last item blocking an almost native-feeling experience.

Investigation

The initial pageload includes counts for assigned issues, assigned MRs and todos. However, time spent in the database is very low.

Loading static assets, such as fonts, takes ~50ms so the request cycle itself is fine but these requests bypass the Rails stack.

Strangely, the flamegraph only extends to about 200ms, so there are ~500ms unaccounted for.

What is visible from the flamegraph is the significant contribution to rendering time of nested layouts, helpers and HAML:

image

The average Duration for this endpoint is around 300ms:

image

Performance tails off further around the 90th percentile. IOW, around 10% of requests take 500ms or more to render just the HTML.

Possible leads

  • There is a strong possibility of something contributing to slowness somewhere in the Rails/server stack that isn't showing up in the flamegraph.
  • Nested templates are known to increase rendering time, these could be flattened.
  • HAML templates are being rendered on every request. Could these be precompiled for things like the application layout?
Edited by John Hope