Environment dashboard is slow with a few projects

Problem

We recently introduced Environment dashboard, which you can access via https://gitlab.com/-/operations/environments on gitlab.com today.

In the page, FE requests operations/environments_list JSON endpoint to render the contents, and this endpoint seems quite slow.

In my case, it executes 249 database queries, 9 Gitaly calls and 52 Redis calls.

This, in fact, results in >8 seconds per fetch.

Chrome

Grafana

TODO

• gitlab-com/gitlab-profiler!49 (merged)
• Investigate bottle necks. => https://docs.gitlab.com/ee/development/profiling.html
• Fix potential N+1
• [-] Pagination? => #33895 (closed)
• Monitor the endpoint in Grafana

/cc @jagood @stanhu @darbyfrey