Skip to content

Eliminate N+1 queries for pipeline GraphQL endpoint

Furkan Ayhan requested to merge 330707-pipeline-graphql-nplusone into master

What does this MR do?

This MR removes N+1 queries from the project pipeline GraphQL endpoint.

  • The metadata relation is the most common N+1 query.
  • The downstream_pipeline relation is for bridge jobs, not so common.

In this MR, the structure of preloading is also changed. (I have a plan to unify the preloading mechanism for commit statuses)

Related to #330707 (closed)

detailed comment: #330707 (comment 658019136)

previous work: !64700 (merged)

Screenshots or Screencasts (strongly suggested)

Screen_Shot_2021-08-23_at_16.29.31

How to setup and validate locally (strongly suggested)

  1. Run the development env with RAILS_PROFILE=true.
  2. Create a pipeline with some jobs.
  3. Open GraphiQL http://gdk.test:3000/-/graphql-explorer.
  4. Trace the logs tail -f log/development.log in gitlab.
  5. Run the query with variables;
query getPipelineDetails($projectPath: ID!, $iid: ID!) {
  project(fullPath: $projectPath) {
    pipeline(iid: $iid) {
      stages {
        nodes {
          groups {
            nodes {
              jobs {
                nodes {
                  status: detailedStatus {
                    detailsPath
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
  1. See the logs.

Metrics

This will help us to check later:

75th and 99th percentiles of duration and db count: https://log.gprd.gitlab.net/goto/b08207d25d2259903d5140b938f612e7

Last 7 days:

Screen_Shot_2021-08-23_at_19.32.54

Conformity

Availability and Testing

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Furkan Ayhan

Merge request reports