Breaking change exception: Stop returning total count headers on the group merge requests REST API

Breaking Change Exception Request

Executive summary

We propose to stop returning X-Total, X-Total-Pages and the rel="last" link on GET /groups/:id/merge_requests by default. Today every request runs an exact SELECT COUNT(*) (capped at 10,001 rows) to produce them. This endpoint is one of the slowest REST endpoints on GitLab.com, and the count is its largest fixed cost per page.

  • Still returned: X-Page, X-Per-Page, X-Next-Page, X-Prev-Page and the next, prev and first links. Clients that follow next-page links are unaffected.
  • A request parameter (name still to be decided) opts back in to the count and the headers. No client loses the capability. Only the default changes.
  • Implementation: gitlab-org/gitlab!257173, behind the feature flag skip_total_count_on_group_merge_requests_api (default off, per-group actor). Rollout issue: gitlab-org/gitlab#630469.

Why it matters:

  • On the gitlab-org group, the count took 1240 ms of 2400 ms of database time in one request.
  • The count ignores page and offset, so page 1 and page 50 of the same query re-run the identical count.
  • Walking all open merge requests of gitlab-org (about 5,400) costs 54 counts at per_page=100, or 270 at the default page size of 20.
  • The heaviest callers walk every page repeatedly. One caller loops through pages 1 to 100 roughly every 14 minutes.
  • Three root namespaces account for about 75% of the slow calls. Customer names are in an internal confidential issue (https://gitlab.com/gitlab-org/gitlab/-/issues/591321) and are not named here.
  • The rest of the request (mergeability checks, approvals) is being optimised separately. The count can only be skipped or cached, not made cheaper.

Does your breaking change meet one of these three criteria?

Criterion 2: negligible customer impact.

  • The REST API docs already say X-Total and X-Total-Pages are omitted when a list has more than 10,000 results ("Pagination response headers", https://docs.gitlab.com/api/rest/#pagination-response-headers). Any correct client of this endpoint already has to handle missing headers.
  • The client library that sends the most traffic to this endpoint (gitlab4j-api, Java) has fallbacks when X-Total and X-Total-Pages are missing.
  • GitLab's own frontend never calls this REST endpoint. The web group merge request list uses GraphQL and already caps its counts at 999.
  • The opt-in parameter keeps the old behaviour available.
  • TODO: Kibana numbers on how many distinct tokens and root namespaces call this endpoint per week on GitLab.com, and what share of the customer base that is.

Impact assessment

How many users and customers are impacted? What Tier/ARR of customer?

  • Affected: API clients that read X-Total, X-Total-Pages or the rel="last" link from this endpoint.
  • We cannot see from logs whether a client reads a response header.
  • A sign of a client reading X-Total-Pages is jumping straight to a high page number.
  • TODO: weekly count of distinct tokens and root namespaces calling this endpoint on GitLab.com (Kibana).
  • TODO: which of those callers jump straight to a high page number.
  • TODO: Tier and ARR of the affected customers (from the internal issue, via account teams).

Can we get the same outcome without a breaking change?

Not fully. We considered:

  • Cache the count for 15 minutes per user and filter set. Needs a Kaminari patch, a cache key and a staleness window. It only helps from page 2 onwards, so one-off callers see no gain. This was the previous version of the merge request.
  • Cap the count at 1,000, like the web UI. X-Total is still dropped above the cap, so it is also a contract change, and page 1 still pays for a count.
  • Count only on page 1. Later pages lose X-Total, so it is still a contract change, with inconsistent headers across pages.
  • Keyset pagination. Merge requests have no keyset orderings on REST yet. The pagination guidelines require offset pagination to stay supported, so the count would still run for offset callers. Moving callers to keyset is itself a breaking change.

When do you want to make the breaking change?

  • GitLab.com first, behind the feature flag, enabled group by group and then globally.
  • Self-managed and Dedicated keep the old behaviour until the deprecation window ends.
  • TODO: target milestone for GitLab.com enablement, and for the default change on self-managed (the owner will set the Milestone field).

What is the alternative for customers to do the same job the change will break?

  • Follow the next link or X-Next-Page header until it is empty. This needs no total.
  • If a total is truly needed, send the opt-in parameter to get the count and headers back (parameter name to be decided).

How difficult is it for customers to migrate to the alternative? Is there a migration plan?

  • Low effort. Clients that need totals add one request parameter.
  • Clients that walk pages by next-page links need no change.
  • Plan: deprecation notice, docs note, and direct outreach to the top callers (see Customer Communication).

Does this breaking change interact with other proposed or approved breaking changes?

None known. The audit events keyset change is unrelated in scope.

Rollout & Communication plan

Rollout: enable the flag group by group, starting with the namespaces that generate the slow calls, then enable it globally on GitLab.com. Monitor the endpoint's duration and error rate throughout.

Have we handled similar breaking changes in the past? What can we learn from the success or failure of those methods?

Are you employing any strategies to lessen the customer or support impact?

  • Feature flag per group.
  • Opt-in parameter to restore the headers.
  • Staged enablement, starting with the heaviest callers.
  • Monitoring of the endpoint's duration and error rate during rollout.
  • Self-managed and Dedicated keep the old behaviour until the deprecation window ends.

Internal Communication

  • Engineering, Product, Security, Customer Success, and Sales: expect the question "will my API client break?" Key points: next-page links still work, the opt-in parameter restores totals, and this already matches behaviour above 10,000 results.
  • Support: needs to know that the headers are gone by default on this endpoint, that the opt-in parameter restores them, and that this matches the existing behaviour above 10,000 results.
  • A Support Preparedness issue will be opened after approval.

Customer Communication

  • A deprecation entry on the deprecations page.
  • A note in the merge requests API docs, like the one in the commits API docs.
  • Direct outreach through the account teams to the top callers identified in the internal issue.
  • Customer comms to be approved by Corporate Communications and Customer Success.

Approval Process

You should ensure you have received approval and communicated your breaking change at least three months in advance of the implementation date. It is recommended that you open this request at least six months in advance to have time for the approval process. It is the requestor's responsibility to drive the approval process forward and ensure it is completed in the recommended timeframe.

  • Tag in the appropriate Product Director and Engineering Director for your product area. They may ask for further clarifications.
    • Approved by Product Director or GPM: < Add Name >
    • Approved by Engineering Director: < Add Name >
  • Once you have approval from both Product GPM or Director & Engineering Director, add the label "Breaking Change: Awaiting Support Approval" and remove the prior label.
  • Share this issue in the #support_leadership slack channel and ask for a review from the support perspective. Once you get signoff from 1 Support Senior Leader, add the label "Breaking Change: Awaiting VP Approval" and remove the prior label.
    • Approved by Support Stable Counterpart (If Applicable): < Add Name >
    • Approved by Support Senior Manager or Director: < Add Name >
  • Tag in the appropriate Product and Engineering VPs or Senior Directors for your product area. They may ask for further clarifications.
    • Approved by Product VP or Senior Director: < Add Name >
    • Approved by Engineering VP or Senior Director: < Add Name >
  • Once you have approval from both the Engineering & Product VP or Senior Director, add the "Breaking Change: VP Approved" label. Please keep this issue open until the breaking change has been implemented. This ensures all breaking changes are visible to leadership on this board.
  • Once you have VP approval, create a public deprecation issue that will serve as a source of truth for customers in regards to the change.
  • Ensure the change is added to the deprecations docs page by following this guidance.
  • Seek approval on any customer comms from the Corporate Communications and Customer Success teams