Data Access - Durability (#1) · Epics · Durability team

Data Access - Durability

The [Durability team](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/data-access/durability/) is responsible for ensuring we meet customer expectations for data retention and durability. It has the following three main responsibilities: 1. Tooling and monitoring for backup and restores of the following deployments of the .com SaaS, Cells, Self-managed and Dedicated. 2. Sidekiq and Redis performance, data management and capacity planning. 3. Gitaly infrastructure support to support Raft and Kubernetes. To manage our work we follow [SaaS Platforms Project Management](https://handbook.gitlab.com/handbook/engineering/infrastructure/platforms/project-management/). ## Project Work ### :soon: Ready Linked epics that are ready to start | **Topic** | |-----------| | [Unified Backup CLI and Disaster Recovery Backlog](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/35) ~"group::tenant services" | | [Move KAS to its own Redis instance](https://gitlab.com/groups/gitlab-org/-/epics/9412) ~"group::tenant services" | | [Improvements to project recovery for project deletions on the .com SaaS](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/21) ~"group::tenant services" | ### :white_check_mark: Completed Work Items that have been completed <details> | **Topic** | **Started** | **Ended** | **Summary** | |-----------| ------------| ----------| ------------ | | [Performance and Data Management for Sidekiq/Redis](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/4) | 2024-05-01 | 2025-03-26 | **2025-02-04**: Since the cleanup on Redis DB Loadbalancing migration is done, this epic can be closed :tada: . This epic holds various other epics and issues related to Sidekiq and Redis in FY25Q4. **Nested Epics: 2** • https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/13+ **2025-02-04**: Redis DB Loadbalancing has been migrated to Redis Cluster ~"Service::RedisClusterDbLoadBalancing" and is now horizontally scalable. Primary CPU utilization went down from [80% to 40%](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/13#note_2297493099). • https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1414+ **2024-11-20**: The feature flags for session map and per-database data consistency are deleted and the feature is released. We are left with minor improvement works https://gitlab.com/gitlab-org/gitlab/-/issues/501762+   | | [Sidekiq workload throttling/circuit-breaking by database usage](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1410) ~"group::tenant services" | 2024-10-12 | 2025-08-11 | **2025-08-05**: We've finished all works in this epic. The feature has not been rolled out yet, and it will be tracked in a [follow-up epic](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/42). This epic can be closed during Grand Review :tada: | | [Onboard the new Durability team](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/2) | 2024-11-06 | 2024-12-10 | **2024-12-03**: **Final Status** :tada: **achievements**: - Created a brand new team, scoped out the responsibilities and setup project management - We cleaned up some of the documentation around DR - Started on-boarding a new team member - We've established some new conventions in the data-access teams for groups and project locations. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/2#note_2237241724_  | | [Research spike into backup tool operations for Dedicated on AWS](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/3) | 2024-11-07 | 2025-01-14 | **2025-01-14**: The work on this epic is complete. We have a good understanding of the current backup and restore operations of the dedicated team. The information gathered here will be used to shape our approach for implementing backups and restores for cloud instances. Thanks to all who participated. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/3#note_2295204766_  | | [Performance investigations and incident followups](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/5) | 2024-11-11 | 2025-05-05 | **2025-04-29**: Memory pressure problem is resolved; wrapping up incident follow-ups. Fruitful results so far in the Gitaly transactions bottleneck analysis. Found some unexpected contention patterns, clarifying precisely where the stalls occur in the kernel code. This is teaching us which kinds of workloads will tend to be more acutely affected and also may hint at what options we have to reduce that contention. :tada: **achievements**: - **Gitaly transactions testing**: - Discovered 2 distinct bottlenecks: - https://gitlab.com/gitlab-org/gitaly/-/issues/6710#note_2468159671 - Stalls in `fsync`, waiting to commit to the ext4 filesystem journal. - https://gitlab.com/gitlab-org/gitaly/-/issues/6710#note_2468242316 - Stalls in `linkat`, `unlinkat`, and `mkdirat`, waiting on lock contention on inodes' reader/writer semaphore. Additionally, to support this analysis, here are reference notes on kernel's [filesystem locking](https://gitlab.com/gitlab-org/gitaly/-/issues/6710#note_2468329618) code. - Continuing analysis of both bottlenecks. There are several viable directions to take this research. Will have more focus time for this once the memory pressure issue is wrapped up (which should be soon). - **Memory pressure on Gitaly:** Resolved a years-long pattern of intermittently severe memory pressure spikes on Gitaly. - The [research into anonymous memory spikes](https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/128) on the canary gitaly node uncovered the root cause, triggering conditions, and two distinct outcomes that have different symptoms depending on the intensity of the memory pressure. It turns out we have had similar incidents over the years, and the solution we implemented should prevent them. (There are of course many ways to induce memory pressure, but this one was particularly important and recurring.) - Later today I will write a summary in the issue description and close it, so see there for further details. - Two separate severe hour-long regressions made the canary node (including the `gitlab-org/gitlab` project) unusable. (See [incident](https://app.incident.io/gitlab/incidents/518).) In those cases, the memory pressure was so intense that 97% of CPU time was spent attempting to do memory reclaim -- which was ineffective because it remained outpaced by demand from arriving requests. - Mitigated the root cause by leveraging the distinctive signals uncovered by the research to proactively relieve memory pressure before it became too intense. - Then prevented future recurrence by breaking the coupling between the slow client and the waiting server-side process. - **Additional wins** from the above issue: - The same approach may be generally useful for self-managed customers too. We may update recommendations after observing the effects of rolling out to the rest of our Gitaly fleet. - The temporary mitigation we developed as an out-of-band reactive memory pressure release valve could be converted into an in-product load-shedding feature. We may explore this as a follow-up. :issue-blocked: **blockers**: - No hard blockers, just stretched for time between competing priorities: - Gitaly memory pressure: The memory pressure incident stole much of the time I had planned to spend working with the Rapid Action. Outcomes were excellent, but it delayed competing work. - Gitaly transactions bottleneck analysis: This is a time-sensitive blocker for Gitaly, as transactions are foundational for several other upcoming projects, and we need to choose a course of action for addressing the performance trade-offs. - DB replication throughput: This is on hold for now, but I expect this needs to come back into focus soon. :arrow_forward: **next**: - Wrap up the memory pressure research and incident issues. - Continue bottleneck analysis for Gitaly transactions' in-kernel stall events. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/5#note_2474578961_  | | [Migrate redis-sessions to redis cluster](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/9) | 2024-12-12 | 2025-04-04 | **2025-03-18**: We've finished the post migration cleanups. This epic can be closed during Grand Review :tada: | | [Provision a new Redis Cluster for container registry group](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/22) ~"group::tenant services" | 2025-03-10 | 2025-05-23 | **2025-05-20**: :tada: * Migration of `redis-registry-cache` to `redis-cluster-registry` is only pending which has a [separate epic](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/24), so will mark this particular epic as completed. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/22#note_2513381663_  | | [Sidekiq shard for jobs like import/export that have local storage requirements](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/23) ~"group::tenant services" | 2025-05-05 | 2025-08-01 | **2025-07-29**: This epic should be closed during Grand Review. Here are the final updates just incase anyone missed it: **Winnings:** :tada: * We don't [see/rarely see pod evictions](https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/161#note_2654476321) due to storage anymore. * The [p95 latency has decreased](https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/161) as we moved to `import-shared-storage` gke nodepool due to local SSDs performance. * By isolating this shard from other `memory-bound `workloads, we will reduce the blast radius of incidents like https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19577 in which the apdex SLO was violated due to `Projects::ImportExport::RelationExportWorker` . * Established isolation ensuring only `import-shared-storage` sidekiq processes utilize expensive local SSDs, protecting other Kubernetes workloads from unnecessary resource consumption. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/23#note_2655172996_  | | [Migrate redis away from omnibus gitlab installations](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/17) ~"group::tenant services" | 2025-05-19 | 2025-08-11 | **2025-08-05**: :tada: **achievements**: logging configurations were [created](https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/11579) to export audit logs for MemoryStore redis from GCP to our ELK stack. :arrow_forward: **next**: Create index in elasticsearch for memorystore logs and enable dual-write changes to `gstg` for tracechunks. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/17#note_2668401521_  | | [Rollout Gitaly OS upgrades to production Gitaly fleet](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/44) ~"group::tenant services" | 2025-08-01 | 2025-09-12 | **2025-09-09**:  :tada: **achievements**: Successfully completed OS upgrades for all 157 Gitaly nodes in production from Ubuntu 20.04 to 22.04. The upgrades were executed with minimal service disruption using improved tooling and coordination processes developed during the project. All packertest nodes successfully scaled down post-completion Dashboard verification: [Node OS Status](https://dashboards.gitlab.net/explore?schemaVersion=1&panes=%7B%22qgq%22%3A%7B%22datasource%22%3A%22mimir-gitlab-gprd%22%2C%22queries%22%3A%5B%7B%22refId%22%3A%22A%22%2C%22expr%22%3A%22count%28node_os_info%7Bpretty_name%3D%7E%5C%22Ubuntu.\*%5C%22%2C+type%3D%5C%22gitaly%5C%22%7D%29+by+%28pretty_name%2C+type%29%22%2C%22range%22%3Atrue%2C%22instant%22%3Atrue%2C%22datasource%22%3A%7B%22type%22%3A%22prometheus%22%2C%22uid%22%3A%22mimir-gitlab-gprd%22%7D%2C%22editorMode%22%3A%22code%22%2C%22legendFormat%22%3A%22__auto%22%7D%5D%2C%22range%22%3A%7B%22from%22%3A%22now-14d%22%2C%22to%22%3A%22now%22%7D%7D%7D&orgId=1) Cleanup MR: https://ops.gitlab.net/gitlab-com/gl-infra/config-mgmt/-/merge_requests/11975 :issue-blocked: **blockers**: None :arrow_forward: **next**: Epic complete - all exit criteria met. GitLab's Gitaly infrastructure is now running on supported Ubuntu 22.04 with current security updates and vendor support. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/44#note_2737750194_  | | [Migrate redis-tracechunks to MemoryStore](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/43) ~"group::tenant services" | 2025-08-01 | 2025-11-21 | **2025-11-18**: :tada: **achievements**: [Cleanup](https://gitlab.com/gitlab-com/gl-infra/tenant-scale/tenant-services/team/-/issues/311) has been completed to get rid of migration code/configurations and this concluded the work scoped in this epic. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/43#note_2894578453_  | | [Enable Sidekiq circuit-breaking on production](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/42) ~"group::tenant services" | 2025-08-01 | 2025-11-17 | **2025-11-11**: No exciting update since last week. We're still cleaning up a bunch of feature flags and configuration before closing this one out. :arrow_forward: **next**: * https://gitlab.com/gitlab-com/gl-infra/tenant-scale/tenant-services/team/-/issues/267+ * https://gitlab.com/gitlab-com/gl-infra/tenant-scale/tenant-services/team/-/issues/292+ _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/42#note_2879981619_  | | [Unified Backup CLI alpha release](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/41) ~"group::tenant services" | 2025-08-01 | 2025-11-05 | **2025-11-04**: Closing status This is the final update of this Epic. Remaining issues will be left as is for the time being, while next steps for the product are determined with the move to Geo. **Nested Epics: 2** • https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/15+ • https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/30+ **2025-07-22**: :tada: **achievements**: - Decouple epic closed few weeks ago :issue-blocked: **blockers**: - :arrow_forward: **next**: - Porting Portable Backups to the new codebase _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/30#note_2642457680_  | | [Backup and Restore testing environment improvements](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/40) ~"group::tenant services" | 2025-08-01 | 2025-11-17 | **2025-11-11**: 🎉 Achievements 1. Secrets migration completed – All secrets have been securely migrated from the repository into Vault, enabling seamless and secure access by CI/CD pipelines. 2. Pipeline reliability restored – Fixed and stabilized pipelines for both 1k and 3k reference architectures, ensuring consistent and reliable execution. 3. Local testing environment established – Introduced a fully functional local setup for GitLab backup and restore validation, allowing engineers to test and iterate independently of CI. This mirrors the production-like configurations used in CI for both reference architectures (Issue #286, MR !12). 4. Enhanced documentation – Added comprehensive documentation that improves user experience, project onboarding, and maintainability. The new docs provide context for the local testing setup, enabling faster iteration and clearer guidance (MR !20). 5. Exit criteria achieved – The Durability team now has full ownership of the backup and restore testing infrastructure, complete with automated testing capabilities and clear operational processes. This milestone marks the successful completion of this epic, a major step forward in making backup and restore testing more reliable, maintainable, and accessible. 🚀 _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/40#note_2879981735_  | | [Create new catchall redis shard](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/39) ~"group::tenant services" | 2025-08-01 | 2025-11-17 | **2025-11-04**: :tada: **achievements**: This epic can be closed at the grand review: * The `catchall-b-sidekiq-shard` deployment in gprd has completed successfully. * `audit_events` and `continuous_integration` feature categories have been migrated to the `catchall_b` sidekiq shard. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20805. * This should now allow us to alleviate pressure on the catchall and reduce daily load on it. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/39#note_2864625631_  | | [Move redis-registry-cache to redis-cluster-registry](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/24) ~"group::tenant services" | 2025-08-01 | 2025-10-24 | **2025-10-21**: This epic can be closed at the grand review. :tada: **achievements**: - **Infrastructure Consolidation**: Successfully migrated all container registry workloads (cache, rate limiting, and DB load balancing) to a single Redis cluster(redis-cluster-registry) - **Phased Migration**: Executed a careful rollout across environments (pre → gstg → gprd) using dual-write strategy to ensure zero data loss - **Chart Enhancement**: Added `redis.cache.cluster` support to GitLab Helm charts to enable cluster-based caching. - **Production Cutover**: Completed final switchover from sentinel to cluster mode on October 1st, 2025 - **Decommissioned** the old `redis-registry-cache` cluster across all environments. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/24#note_2833122505_  Have chalked out a rough plan for the migration as follows: 1. Preparation: - Have already setup cluster `redis-cluster-registry` keeping [sizing properties of registry-cache in mind. ](https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/115). - Add support in application code to write to both clusters simultaneously. - Implement feature flags to control read/write behavior? (both points to be confirmed with @jdrpereira ) 2. Migration Process - Enable dual-write mode (write to both clusters `redis-registry-cache` and `redis-cluster-registry`). - Gradually shift read traffic to `redis-cluster-registry` ( lets do in 10% increments? ). - Monitor performance and error rates during the transition. 3. Once 100% of reads are on `redis-cluster-registry` and data is verified, lets switch remaining writes to `redis-cluster-registry` | | [GitLab Durability - FY26Q3 Keep the Lights On](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/37) ~"group::tenant services" | 2025-10-02 | 2025-11-06 | **2025-11-04**: Very busy this week dealing with the team transition. This will likely be the last update for this epic as I go through https://gitlab.com/gitlab-com/gl-infra/tenant-scale/tenant-services/team/-/issues/293 _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/37#note_2864625720_  | | [FY26Q3 Performance investigations and incident followups](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/47) ~"group::tenant services" | | 2025-11-10 | **2025-11-04**: :tada: **achievements**: - **Prevent recurring OOM-kills:** My pieces of this work are essentially done. Reached clear root cause, and the vendor (Wiz) has patched it for their next release. Happily, this performance improvement will have a broad impact, benefiting more than just GitLab: For all workloads that have a high process creation rate, especially on larger machines, the Wiz Sensor agent will be much more CPU-efficient, less likely to drop events, and less likely to get OOM-killed. Internally, we also calibrated and tested our short-term work-around. We also found a design flaw in our chef recipe for managing config updates to Wiz Sensor, with a fix in progress. - **Gitaly benchmarking:** Currently focusing mostly on the overlayfs testing; at this point we need to improve the benchmark itself before we can do confident bottleneck assessment. Also evaluating some improvements to the baseline benchmark's workload. :issue-blocked: **blockers**: - No technical blockers, but practically there are several unrelated factors consuming time and attention (e.g. org changes, Q3 kickoff, prep for the on-call platform change, helping diagnose metrics problems, etc.). :arrow_forward: **next**: - Gitaly benchmarking as much as possible this next week. Planning at least one more pairing session to expedite it. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/47#note_2864625794_  | | [GitLab Durability - FY26Q2 Keep the Lights On](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/29) ~"group::tenant services" | | 2025-08-11 | **2025-08-05**: :tada: This is the final update for this epic as we have move work into the [Q3 epic](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/37). Based on the epic details and status updates, here's a comprehensive closing summary for management review: | | [FY26Q2 Performance investigations and incident followups](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/27) ~"group::tenant services" | | 2025-08-11 | **2025-08-05**: Highlights for this quarter's performance and scaling investigations: * Gitaly Transactions bottlenecks and mitigations: * Gitaly's transaction mode is a foundational component for many pieces of Gitaly's technical roadmap, including next-gen HA. Supporting this work has been my primary focus. * Studying performance on the gitaly canary node revealed exactly 2 critical bottlenecks at the filesystem layer: inode semaphore contention and journal flush latency spikes. Collectively they sum to over 90% of all significant off-CPU stalls. * When either of these two stall events occurs, they add delays to gitaly's snapshot creation and/or WAL apply, causing a cascading negative effect on gRPC response time and throughput. Such stalls can impede Gitaly's critical path for serialized WAL apply and for creating stable snapshots for read-only requests. All of this amounts to client-facing bursts of slowness that exceed our SLO target. * We implemented and tested a series of improvements to reduce demand on those critical bottlenecks. These include filesystem tuning, workload isolation, pruning/filtering of residual files/dirs in git repos to reduce syscall count during snapshot creation, etc. * After improving performance to the point where the gitaly canary node remained within SLO, we tentatively started evaluating performance on a gitaly node with a more typical workload. That confirmed the same low-level bottlenecks but also revealed that the performance improvements so far are not yet enough to reliably meet SLO. * Our path forward: * We need to make snapshot creation even cheaper. * For making near-constant-time snapshots, we are researching `overlayfs` as a candidate solution. (We also considered `btrfs` but discarded it due to it adding risk, compatibility barriers, and heavy migration costs for customers. Overlayfs is flexible and widely available, making it a much more viable candidate from a customer adoption perspective.) * We plan to focus on overlayfs as a robust widely-used kernel feature that meets our functional requirements. At an early stage, so far it seems like a good fit for our performance requirements. We expect this will completely avoid at least one of the two current bottlenecks and may completely shift the scaling constraints. * Concurrently, we are implementing a dry-run mode to help estimate the overhead transaction mode would incur for individual repos across the gitaly fleet. This analysis will help identify any other cases that unexpectedly accumulate bloat (garbage files/dirs that would slow down snapshot creation). It will also help determine how effective our pruning mechanism is and proactively predict which repos will perform poorly in transaction mode. This data can inform our next steps for mitigations, testing, and use-case modeling. * DB scaling constraints, key saturation risks, and performance regressions: * The DB scaling constraints that most worry me are ones where our workload pushes architectural limits, where adding machine capacity or config tuning no longer helps. * Proposed a research spike to explore feasibility of a db sharding solution. That proposal is currently on hold, due to Gitaly transactions being higher priority. * Created reusable analytical methods for incident response and root cause analysis where `BufferMapping` LWLock contention occurs. This is becoming a more common saturation event in the 6-12 months, and it hints at a hidden saturation condition driven by interactions between query concurrency, working set size, and buffer eviction rate. Tactical mitigations help resolve incidents, but we also need a long-term strategy. * Building on my past research, I empirically identified the approximate capacity ceiling for one of our critical constraining db throughput bottlenecks, which we can use in capacity forecasting and alerting. This capacity ceiling is influenced by tension between 2 dynamic factors of the workload, so rather than being a fixed point, it is a joint property of both the workload and the infrastructure. * Discovered a pair of subtle bugs affecting one of the two key metrics we use for analyzing DB stall events. The other metric is fine, but the defective one shows misleading spikes and dips that are effectively measurement error. Fixing this defect will avoid people making decisions based on incorrect data. That had been one of the blockers for adding alerting, but once we deploy the fix, both metrics will give reliable signal. (For context, these two metrics measure the same behavior from distinct observability points, so they should generally agree at a high level.) * Gitaly memory saturation abuse vector: * In an incident root cause analysis, I discovered a subtle design flaw that a bad actor was attempting to abuse to induce memory saturation on Gitaly nodes. * A key component of this abuse vector was inducing Gitaly's pack-objects-cache mechanism to not cache the gRPC call and force the backing `git-pack-objects` process to consume and hold a large amount of memory for many minutes. * To break that abuse vector, we exposed a new config option to allow pack-objects-cache to act as a disk-based (rather than memory-based) response payload buffer, preventing a standing accumulation of memory pressure. * As follow-up to this mitigation, I also checked if this change in disk IO workload had affected any critical bottlenecks. It hadn't. Both conventional metrics and ad hoc off-CPU profiling confirmed that the extra IO was well within capacity and was not causing significant stall events. * CI abuse: * A persistent adversary has been iterating on their abuse methods. * Analyzed their binary to determine the nature of abuse (fortunately simply cryptocurrency mining, not a more malicious actor). * Collaborated across teams on a series of mitigations. * Each round of abuse causes long delays for legitimate customers' CI jobs and wastes GitLab's money on compute resources, so thwarting this abuse is worth the effort. * Currently we are successfully blocking this actor's attempts. As the above highlights illustrate, my work in this epic tends to involve deep-dive analyses leading to proactive or reactive systemic improvements. I generally tend to focus mostly on the areas where my unusual skills and background can be uniquely helpful: Unblocking and augmenting high-value work, amplifying efficiency, uncovering impending systemic risks while we have time to proactively address them, etc. Currently Gitaly transactions and DB scaling constraints remain my top points of focus. Other points of focus included saturation events in redis, sidekiq, k8s, CI, etc. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/27#note_2668401606_  | | [FY26 Q2 Unified Backup Roadmap](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/25) ~"group::tenant services" | | 2025-08-11 | **2025-08-05**: **Nested Epics: 1** • https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/26+ | | [Gitaly OS kernel updates and reboot scheduling](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/8) ~"group::tenant services" | | 2025-08-11 | **2025-08-05**:  :tada: **achievements**: - The production upgrades have been planned and scheduled to take place over the next 3 weeks. These will occur during off-peak weekend hours. You can find more details in the [upgrade plan epic](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/44). - We’re currently finalising our communication strategy. Internally, we’ll share updates via Slack, and externally, we’ll publish a maintenance window notice on the GitLab status page. We expect minimal disruption during the upgrades. However, customers with data spanning multiple storage nodes may experience more interruptions, especially if nodes do not exist in the same batch (prior [discussions](https://gitlab.com/gitlab-com/gl-infra/observability/team/-/issues/1376#note_723585374) show that it is not feasible to group batches around a specific customer for batching). If we can identify these customers in advance, we plan to reach out to them directly. The final work on this is currently being moved to the Q3 epic https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/44 _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/8#note_2668401655_  **Nested Epics: 1** • https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1640+ | | [Backup and Restore for PD, Object Storage and CloudSQL.](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/19) ~"group::tenant services" | | 2025-05-20 | **2025-05-13**: The Unified Backup CLI project has made significant progress this week, with key objectives achieved and new features implemented. :tada: **achievements**: - Successfully completed the CloudSQL [operation timeout feature](https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/98)): - [The MR](gitlab-backup-cli!58) has been merged, adding configurable timeout settings for asynchronous operations - This resolves the issue of indefinite waiting during CloudSQL operations - Added contributor documentation: - [Added CONTRIBUTING.md](gitlab-backup-cli!72) file to the project - Provides guidelines for new contributors to the backup CLI project - Continued progress on https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/103+: - [The MR](gitlab-backup-cli!62) is awaiting review :issue-blocked: **blockers**: - Two issues are currently awaiting the https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/103+ - https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/105+ - https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/156+ :arrow_forward: **next**: - Complete review and merge of [manifest implementation](https://gitlab.com/gitlab-com/gl-infra/data-access/durability/gitlab-backup-cli/-/merge_requests/62) - Begin work on Cloud operations status check functionality ([issue #105](https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/105)) - Design test and validation procedures to ensure backup and restore operations work correctly across all supported environments - Start work on GCP PD disk snapshots, which is next on the objectives list _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/19#note_2499877935_  | | [GitLab Durability - FY26Q1 Keep the Lights On](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/16) ~"group::tenant services" | | 2025-05-05 | **2025-04-29**: This will be the last status update for "Keep the Lights On" for Q1 as we will be replacing this with a Q2 epic. For a high level overview of what we completed in this epic as well as Durability as a whole [see our Q1 recap presentation](https://docs.google.com/presentation/d/1BK62Ayp5sXUmMN8ZCXRXoGKFx29H78PHcLZEZS2b3xU/edit?slide=id.g23dede780b6_2_36#slide=id.g23dede780b6_2_36) :tada: **achievements**: - scaling out redis-cluster-shared-state https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/138#note_2475150483 , nice work (and results) @ahmadsherif - Created a [Q1 recap for Durability](https://docs.google.com/presentation/d/1BK62Ayp5sXUmMN8ZCXRXoGKFx29H78PHcLZEZS2b3xU/edit?slide=id.g23dede780b6_2_36#slide=id.g23dede780b6_2_36) :issue-blocked: **blockers**: - https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/133 is blocked until we make a final call on NATS :arrow_forward: **next**: - We will be closing out our Q1 epics and creating new ones _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/16#note_2474579156_  | | [Bootstrap the Unified Backup project.](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/14) ~"group::tenant services" | | 2025-03-08 | **2025-03-04**: We've fixed the remaining issues with the `semantic-release` configuration. Once https://gitlab.com/gitlab-com/gl-infra/infra-mgmt/-/merge_requests/1597+ is merged, https://gitlab.com/gitlab-com/gl-infra/data-access/durability/gitlab-backup-cli/-/issues/5+ and this Epic can be closed _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/14#note_2376952277_  | | [Durability team work and unplanned tasks](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/10) | | 2025-02-05 | **2025-02-04**: :tada: **achievements**: - Final status update! - We have finalized the [list of Q1 epics](https://gitlab.com/gitlab-com/gl-infra/data-access/durability/team/-/issues/53#epics-for-q1) _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/10#note_2329090859_  **Nested Epics: 1** • https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/18+ **2025-01-31**: some status update _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/18#note_2325535717_  | | [Gitaly infra support in Durability](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/7) | | 2025-02-18 | **2025-02-11**: With https://gitlab.com/gitlab-org/gitaly/-/merge_requests/7552 being merged, we are ready to progress with dry-running reftables on staging (https://gitlab.com/gitlab-org/gitaly/-/issues/6626). _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/7#note_2340836317_  **Nested Epics: 1** • https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/11+ | | [Restore testing automation improvements for .com SaaS Gitaly](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/6) ~"group::tenant services" | | 2025-04-29 | **2025-04-22**: We ended up dropping the `git fsck` verification as it not very straight-forward to get useful information out of it in a fast way. This drop makes verification time go from hours to about half an hour, which allows us to have more nodes verified during the day. Nothing more to do for this epic. _Copied from https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/6#note_2461028922_  | </details> ### :x: Cancelled Epics that were cancelled <details> | **Topic** | **Ended** | **Summary** | |-----------| ----------| ------------ | | [Provision a Redis Cluster for container registry group](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1312) | 2025-03-10 | | | [Protocell Backup and Disaster Recovery Validation](https://gitlab.com/groups/gitlab-com/gl-infra/data-access/durability/-/epics/48) | 2025-11-04 | | </details>

epic