FY26-Q1 Outlook and Weekly Updates

Latest Update

We are trying out the proposal in gitlab-com/gl-infra/mstaff#406 to link to a quarterly updates issue/epic for the weekly Grand Review instead of posting new comments in Infrastructure Platforms Department.

Last updated: 2025-05-01

2025-05-01 Weekly Update for Grand Review

▶️ Meeting Recording Playlist | 🎥 Recording

Help Needed / Blockers:

None.

📕 To Be Closed

groupdatabase operations - Additional observability - part 2 (gitlab-com/gl-infra/data-access/dbo&4 - closed)

🌟 Highlights

groupcode review / groupdatabase operations - Graceful offloading of data from Merge Request Diffs table - production rollout began on 2025-04-22 to stop writing messages to this database with no issues. Results - In the five days before the feature flag was enabled, the merge_request_diff_commits growth was +8.14GB. Since the feature flag has been fully enabled , the growth was 5.99GB. This shows a reduction of 35.89% in growth.
groupdurability / groupgit / groupgitaly have been collaborating to debug Gitaly transactions. Two distinct bottlenecks discovered:
- gitlab-org/gitaly#6710 (comment 2468159671) - Stalls in fsync, waiting to commit to the ext4 filesystem journal.
- gitlab-org/gitaly#6710 (comment 2468242316) - Stalls in linkat, unlinkat, and mkdirat, waiting on lock contention on inodes' reader/writer semaphore.
groupgitaly - bundle-uri is in abled in production for the gitaly repository. The goal of this is to reduce CPU pressure during repository cloning. We are actively working to create a custom gitlab-runner instance which can take advantage of the bundles for the gitaly repository in an effort to validate the performance improvements.
groupgit - We've successfully deployed a fix for the non-linear scaling problems that affected git bundle create. These improvements were backported to Git 2.49 and rolled out to Gitaly production early on Friday, April 25th (UTC).
- What we're seeing: The CPU time metrics for the git bundle command show a clear positive change – the previous spikes in latency are no longer present.
- What it means: We expect significantly faster git bundle creation times, particularly for our larger repositories or those with a high number of references. We use Git bundles in Gitaly to create backups of the repository and also for bundle-URI which is used to decrease the load in CI systems.
groupgit - The opportunistic migrator is now live on staging, we've migrated 15k repositories already. The migrations are running as expected without many hiccups. Once we have sufficient information, we can tweak the heuristics by making more informed decisions.
groupdatabase frameworks - We have finalized the migration to daily tables for web_hook_logs, which is crucial for scalability reasons.