Loading a stage's job is slow (n+1) and there is not Visual cue about it

It is taking >= 4 seconds to load the jobs for a stage in gitlab-org/gitlab. It also runs 518 (!) SQL queries

1. Go to !76620 (merged)
2. Click the test stage of the Merge request's latest pipeline.
3. Wait for the stage pop-up to finish loading

Possible cause

I think there's a N+1 problem. I see 495 matches for the exact SQL query below:

I see:

SELECT "ci_builds_metadata"."project_id", "ci_builds_metadata"."timeout", "ci_builds_metadata"."timeout_source", "ci_builds_metadata"."config_options", "ci_builds_metadata"."config_variables", "ci_builds_metadata"."interruptible", "ci_builds_metadata"."has_exposed_artifacts", "ci_builds_metadata"."environment_auto_stop_in", "ci_builds_metadata"."expanded_environment_name", "ci_builds_metadata"."secrets", "ci_builds_metadata"."build_id", "ci_builds_metadata"."id", "ci_builds_metadata"."runtime_runner_features" FROM "ci_builds_metadata" WHERE "ci_builds_metadata"."build_id" = :id LIMIT 1

[
  "lib/peek/views/active_record.rb:55:in `block in setup_subscribers'",
  "app/models/concerns/ci/metadatable.rb:77:in `read_metadata_attribute'",
  "app/models/concerns/ci/metadatable.rb:45:in `options'",
  "app/models/concerns/ci/metadatable.rb:33:in `degenerated?'",
  "app/models/ci/build.rb:420:in `archived?'",
  "app/models/ci/build.rb:455:in `retryable?'",
  "app/serializers/job_entity.rb:56:in `retryable?'",
  "app/serializers/job_entity.rb:18:in `block in <class:JobEntity>'",
  "app/serializers/base_serializer.rb:16:in `represent'",
  "app/serializers/concerns/with_pagination.rb:21:in `represent'",
  "app/controllers/projects/pipelines_controller.rb:158:in `stage'",
  "ee/lib/gitlab/ip_address_state.rb:10:in `with'",
  "ee/app/controllers/ee/application_controller.rb:44:in `set_current_ip_address'",
  "app/controllers/application_controller.rb:490:in `set_current_admin'",
  "lib/gitlab/session.rb:11:in `with_session'",
  "app/controllers/application_controller.rb:481:in `set_session_storage'",
  "app/controllers/application_controller.rb:475:in `set_locale'",
  "app/controllers/application_controller.rb:469:in `set_current_context'",
  "ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'",
  "lib/gitlab/jira/middleware.rb:19:in `call'",
  "lib/gitlab/database/query_analyzer.rb:42:in `within'"
]

Proposal

1. MVC could be a UI treatment that shows the user it's loading and delayed which could be frontend only - Added to the design management section.
2. Solve for N+1 problem

Measures of success

• Our target is that the gitlab-org/gitab project should have pipelines starting in 3 seconds or less in p99
• Smaller projects should have pipelines starting in 400ms or less
• These will be measured by ??