Skip to content

Implement lazy association loader

Mehmet Emin INAC requested to merge minac_lazy_association_loader into master

What does this MR do and why?

This MR introduces an abstraction to preload associated records of an Active Record model lazily on GraphQL. It makes it possible to sort, limit, and filter has_many associations without causing an N+1 query issue.

See [Draft] Use lazy association loader (!115241 - closed) for its usage.

Database review

The below queries are generated for a GraphQL query to load only the last state transition record for a list of vulnerabilities.

GraphQL query
query getVulnerabilities($project_fullpath: ID!) {
  project(fullPath: $project_fullpath) {
    vulnerabilities(first: 20) {
      nodes {
        stateTransitions(last: 1) {
          nodes {
            fromState
          }
        }
      }
    }
  }
} 
Here is an example query generated by the `TopNLoader` utility class
SELECT
    "vulnerability_state_transitions".*
FROM
    vulnerabilities
    JOIN LATERAL (
        SELECT
            "vulnerability_state_transitions".*
        FROM
            "vulnerability_state_transitions"
        WHERE (vulnerability_state_transitions.vulnerability_id = vulnerabilities.id)
    ORDER BY
        "vulnerability_state_transitions"."id" ASC
    LIMIT 2) AS vulnerability_state_transitions ON TRUE
WHERE
    "vulnerabilities"."id" IN (78324807, 78324806, 78324805, 77964096, 77896757, 77896756, 77896755, 77896754, 77625632, 77625631, 77625630, 77524835, 77524834, 77524833, 77498088, 77498087, 77498086, 77069619, 77069618, 77046383)

https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/16812/commands/57213

Old query to load all the state transitions
SELECT
    "vulnerability_state_transitions".*
FROM
    "vulnerability_state_transitions"
WHERE
    "vulnerability_state_transitions"."vulnerability_id" IN (78324807, 78324806, 78324805, 77964096, 77896757, 77896756, 77896755, 77896754, 77625632, 77625631, 77625630, 77524835, 77524834, 77524833, 77498088, 77498087, 77498086, 77069619, 77069618, 77046383)

https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/16938/commands/57551

Note that the latter one performs better but it loads all the state transitions from the database which could introduce memory bloat and serious performance issues.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Mehmet Emin INAC

Merge request reports