Draft: Migrate non-search Tags API pagination to ListRefs

Emma Parkrequested to merge
614152-tags-api-list-refs into master

What does this MR do and why?

What happened

Repeated full tag fetches saturated Gitaly CPU and slowed the Tags API and other Git operations in an incident. An automated client was paginating a repository with many tags. For page 2 and later, FindAllTags fetched details for every tag before Rails kept only the requested page. Rails webservice pods also ran out of memory and were killed, likely because concurrent requests built full tag lists in Ruby.

How this fixes it

The project-scoped, default-off use_new_tags_finder_api fetches a page * per_page prefix through ListRefs, slices it, then batches commit/tag-message requests for the selected page. Page 5 at 100 per page fetches 500 refs but details only for tags 401–500. Normal keyset is supported; search/version sorting retain legacy behavior.

Oversized selected messages are downloaded; Rails returns the existing “too big” placeholder. Avoiding extra size requests is an accepted tradeoff.

Why ListRefs instead of limited FindAllTags?

Limiting FindAllTags still transfers detailed data for 500 tags rather than the selected 100. ListRefs fetches the prefix, then loads details only for the selected page; its median was 61% lower in a preliminary local benchmark.

Approach Median
FindAllTags(limit: 500), select 100 298.37 ms
ListRefs(limit: 500), select 100, batch commits + tag messages 115.21 ms
Benchmark setup and limitations

Benchmark results: local GDK with 771 tags, page 5 at 100 per page, updated_desc; 3 warmups and 20 measured runs, alternating approaches. Times include metadata/materialization but exclude tagger timestamps (created_at), releases, protection, and HTTP rendering. This is supporting preliminary local evidence, not a final or production benchmark.

Gitaly dependency

Gitaly prerequisite merged October 8, adding include_tagger_date/Reference.tagger_date to preserve created_at. Committed locks remain 19.4.0; compatible Rails bindings/server availability is pending. Local runs used published 19.5.0.pre.rc3 with a matching rebuilt server as uncommitted local overrides, not a shipping dependency bump, per version policy. Once compatible versions are available, Rails must adopt them and standard CI must rerun.

Pagination follow-ups(existing problems discovered during this work)

  1. Cursor-only requests: Without this temporary correction, requesting ?page_token=v1.0.0&per_page=100 without pagination=keyset returns 100 tags after the cursor. The shared pager then generates an offset next link: ?page_token=v1.0.0&per_page=100&page=2. The cursor stays the same. Following that link returns the same 100 tags, not the next 100, because the finder prioritizes page_token and ignores page=2. The existing new branches finder has this issue; the new tags finder would inherit it. For a quick fix, this MR adds force_keyset, defaulting to false, and enables it only for new tags requests with a cursor. The next link then uses the last returned tag as page_token and includes pagination=keyset. Other callers retain the shared default. A separate issue is needed to decide whether to reject requests missing pagination or automatically use keyset pagination. The issue here
  2. Page 0: The finder normalizes to 1, but the pager reports X-Page: 0, next page 1, repeating results. The existing branches issue also affects new tags; legacy Kaminari avoided it. Track separately to preserve incident scope.

References

Screenshots or screen recordings

Not applicable: API-only change.

How to set up and validate locally

  1. Use compatible bindings/server and three fixture-equivalent tags: annotated, lightweight, nested. Prepare associated releases and protected tags. Enable Feature.enable(:use_new_tags_finder_api, project) in Rails console.
  2. Request /api/v4/projects/:id/repository/tags?page=2&per_page=2. Test explicit keyset with per_page=1 and follow links.
  3. Supply page_token without pagination; follow the link, confirming an advancing cursor and pagination=keyset.
  4. Disable the flag. Compare results, releases/protection, created_at (including null), and normal supported pagination metadata.

MR acceptance checklist

  • Local Tags API/shared pager: 181 examples, 0 failures, 9 conditional pending; pager: 27 passed after final adjustment.
  • RuboCop: 14 files; hooks passed.
  • No changelog: feature flag defaults off.
  • Adopt compatible dependencies and rerun standard CI; full CI success is not established.
  • Complete the acceptance checklist.
Edited by Emma Park

Merge request reports