Batch Orbit item counts in GraphQL and add indexing lastCompletedAt

Michael Angelo Riverarequested to merge
orbit/graph-status-graphql-backend into master

What does this MR do and why?

The Orbit overview page needs item counts for each group. Today the page sends one REST graph_status request per group. This MR lets one GraphQL request get the counts of up to 100 groups with one Orbit call.

  • orbit.itemCounts batches its paths like orbit.indexing does. Both resolvers share one batch method in StatusResolvable.
  • The field limit goes from 1 to 100 groups or projects in one request.
  • orbit.indexing.lastCompletedAt is new. It replaces the last REST field that the frontend reads. It is an experiment from 19.6, it accepts ai_workflows tokens (AI_WORKFLOW_SCOPES), and its description says when it is null.
  • The orbit_item_counts rate limit goes from 10 to 30 requests per minute per user. This is the same as orbit_indexing_status.
batch_orbit_status(:orbit_item_counts) do |client, paths, request_context|
  client.get_item_counts(user: current_user, traversal_paths: paths, request_context: request_context)
    .fetch(:namespaces)
end
Request Orbit field that Rails sets Response that Rails reads
One path traversal_path domains
More paths traversal_paths namespaces

Rails never sets both fields. Orbit runs one count query per path, up to 8 at once. If the query for a path fails, Orbit leaves that path out of namespaces. A path that Orbit does not return gets null. It never gets zero counts.

Rails sends traversal_path for one path only until the Orbit release with gitlab-org/orbit/knowledge-graph!3034 (merged) is the minimum version. A follow-up switches Rails to traversal_paths for every request.

Minimum Orbit version

This MR needs the Orbit release that includes gitlab-org/orbit/knowledge-graph!3034 (merged). That release adds GetItemCounts.traversal_paths. Deploy that Orbit release before this MR.

  • An older Orbit still answers a request for one group.
  • For a request with more groups, an older Orbit returns an error, so itemCounts is null with a GraphQL error. The overview card then shows "Counts are not available right now". It does not show zeros.

Orbit proto gem

This MR bumps gitlab-orbit-proto to 0.156.0. That release has GetItemCountsRequest.traversal_paths and NamespaceItemCounts from gitlab-org/orbit/knowledge-graph!3034 (merged).

The bump uses bundle lock --update gitlab-orbit-proto --cooldown 0, because 0.156.0 is our own first-party release from gitlab-org/orbit/knowledge-graph!3034 (merged) and is younger than the 7-day Bundler cooldown.

SQL queries

itemCounts runs the same number of SQL queries for 2 and 6 groups. For projects, itemCounts adds no SQL queries to a plain query for 2 or 6 projects. The specs check both cases.

References

Related to gitlab-org/orbit/knowledge-graph#263

Screenshots or screen recordings

This MR has no UI change. The frontend MR !264744 (closed) has the screenshots.

How to set up and validate locally

These numbers are for one overview page load with 50 groups on a GDK with the Orbit change.

REST, per group GraphQL, batched
Count gRPC calls 50 1
ClickHouse count queries 50 50, one per group

The REST and GraphQL counts are equal for all 50 groups.

  1. Send this query as a user with 2 or more groups that have Orbit turned on:

    { orbit { memberGroups(enabled: true, first: 50) { nodes { orbit {
  indexing { lastCompletedAt }
  itemCounts { domains { name entities { name count } } }
} } } } }

  2. Make sure that the Orbit log has one GetItemCounts call.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Michael Angelo Rivera

Merge request reports