FY26 Q2 GitLab Delivery Stage Highlights

Group Top Level Summary: Infrastructure Platforms - GitLab Delivery (&1451)

Roadmap - https://www.figma.com/board/8Tp929Qyh3wW715IJh3KC0/GitLab-Delivery-FY26-Roadmap?node-id=0-1&t=vqdeD2zDJnriBXXO-0

Review 2025-07-24

Recording 📹

Blockers ⚠️

Build:

https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/24+
- ❗ Breaking Change: Debian package names will need to change - Pulp requires the version string to include the OS version number. So instead of 18.1.1-ee.0, it will have to be something like 18.1.1-ee.debian12.0
- Build team is now down to 3 engineers, delaying project work.

To Be Closed 📕

Nothing this week

Highlights 🎉

Build
- https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/24+: is in progress with findings being documented in https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/1767+
Self-managed:
- Self-Managed: Implementation work for OpenBao: Chart now supports for Ingress, e2e encryption and automatic unsealing.
- Consolidate Delivery Readiness Process for new components:
  - Assessment question for groupbuild are reviewed and refined. With this we can meet the first exit criteria of this epic.
  - A comprehensive document is prepared to guide product teams on how to use readiness assessment.
Framework:
- Discovery: Cloud Native First Reference Architectures: Target environment sizes decided - Small (S), Medium (M), Large (L), Extra Large (XL)
- https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/framework/-/epics/35+: Completed Observability based analysis for middle_purple_beaver - graphs and details. Some key takeaways: 1) Over time analysis for peak RPS is crucial to identify automated additional workloads that customer has along with breakdown by activity by username to clarify which automations generate traffic 2) For Git workload analysis, with the aim to find unique user workload RPS - query needs to be further tweaked to remove duplication of negotiation requests
Release & Deploy:
- Extend the maintenance policy to account for three releases: Continuing the pilot. We now have backport MR data from two patches: https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21044#backport-bug-fixes
- Earlier branch creation for monthly releases:
  - 📖 A process has been established to address failures during the pre deployments #21097 (closed)\
    - 🧪 A testing strategy is being devised to verify the release candidate creation with a specific SHA #21343 (closed)

Previous Statuses

Review 2025-07-17

Recording 📹 https://www.youtube.com/watch?v=_AxUaulP24w

Blockers ⚠️

Build:

Pulp UI is insufficient (community project) and will need improvements in order to replace Packagecloud's. We will need Frontend capacity to deliver Pulp.
Build team is now down to 3 engineers, delaying project work.

To Be Closed 📕

Highlights 🎉

Build
- Implement Universal Build Toolchain: Identified path towards gradual roll-out of POC component implementations that may leverage QA work done to date.
Self-managed:
- Self-Managed: Implementation work for OpenBao: gitlab/openbao chart has been created, and published to charts.gitlab.io . This is now enabled for development use within the wider gitlab/gitlab chart.
- Consolidate Delivery Readiness Process for new components:
  - Build team questions have been added and checklist breakdown is completed. Ideally this will make its application easier.
  - Started readiness assessment of Search Service (Opensearch / Elasticsearch
Framework:
- Discovery: Cloud Native First Reference Architectures: HPA Scaling testing: Completed multiple Webservice HPA scaling test rounds with different configurations and endpoints, targeting CPU-intensive workloads for discovery. Requested data from Dedicated to help inform further.
Release & Deploy:
- Extend the maintenance policy to account for three releases: we now have a GLQL table to keep track of merged bug fix MRs and backport MRs for patch releases: https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21044#backport-bug-fixes
- Earlier branch creation for monthly releases: The release candidate automation was successfully tested on 18.2 #21327 (closed). Two release candidates were completely created via automation (RC42 details, RC43 details). This gets us one step closer to switching the monthly schedule without adding manual tasks for release managers.

Review 2025-07-10

Recording 📹

Blockers ⚠️

UBT: Python modules cross-compilation currently does not work, putting us at risk. Staff availability issues are a further blocker.

To Be Closed 📕

Define feature review process for Self-Managed concerns: A readiness project and checklist has been established. Zoekt has started adopting that process. More iteration is needed but this can be closed.
Navigating a path towards Cloud Native: Approach Evaluation: Evaluation document completed and will serve as input for the upcoming SM A/B blueprint.
Continuous Auto-Deployment of Helm Charts: Helm chart auto deployments are working on cells!

Highlights 🎉

Build
- Reevaluate Package Cloud replacement: Some limitations in relation to the Pulp UI and how URLs are structured to serve content have been identified. Evaluation is in progress with findings being documented in https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/1767+
Self-managed:
- Self-Managed: Implementation work for OpenBao: Work is underway for implementation of OpenBao Helm chart, consumable by the GitLab umbrealla chart https://gitlab.com/groups/gitlab-org/distribution/-/epics/119+
Framework:
- Exploring Solutions for Architecture Sizing Improvements: This epic explores solutions for architecture sizing improvements by analyzing Dedicated customer data and incidents to identify trends that will enable more accurate workload-based sizing recommendations
Release & Deploy:
- Extend the maintenance policy to account for three releases:
  - More groups (groupauthentication groupglobal search) have been onboarded onto the pilot
  - The pilot groups' maintainers were able to merge bug fix backports themselves, and those commits have been published in the latest patch release!
  - The pilot will be extended until we rollout (alongside early branch creation rollout). Discussions ongoing around the exact timeline (between 18.3 and 18.4)
  - Documentation and templates have been updated to reflect the updated maintenance policy, so that once we roll out the extension, they will point to the SSoT for the updated maintenance policy and maintained versions.
- Transform release environments into production-like self managed environments:
  - Release Environments now supports GitLab Environment Toolkit (GET) with Geo! 🎉 (caveat: only 18.1 at the moment)
  - Engineers can access using Google OAuth. Try it yourself:
    - https://gitlab.18-1-stable-security-get.env.release.gitlab.net
    - https://geo.gitlab.18-1-stable-security-get.env.release.gitlab.net
  - All secrets are stored on Vault
- Earlier branch creation for monthly releases.:
  - 📓 Monthly release documentation was updated to reflect the new process\
  - 🔍 Analysis on how to improve the monthly release transparency for GitLab engineers is in progress. Outcome so far includes: centralized documentation, revamp the release dashboard, adjust existing tooling and create training for awareness

Review 2025-07-03

Recording 📹

Blockers ⚠️

To Be Closed 📕

Highlights 🎉

Build
- Implement Universal Build Toolchain: We've started implementing the test pipelines using self-provisioned infrastructure.
  - Python implementation is in-progeress with early signs of success.
  - We are still on track with our deliverables and plan to be able to build the entire stack by end of Q2 and start rolling out UBT artifacts in Q3.
- Reevaluate Package Cloud replacement: Some limitations in relation to the Pulp UI and how URLs are structured to serve content have been identified. We'll continue documenting these limitations and potential solutions to them, so we can have an implementation plan by the end of Q2.
Self-managed:
- Define feature review process for Self-Managed concerns:
  - The checklist is finalized and merged into the newly created organization-wide readiness project.
  - The checklist is adopted for GitLab Zoekt readiness assessment to validate it.
Framework:
- Discovery: Cloud Native First Reference Architectures:
  - Finished discovery for new Sidekiq approach. The new Reference Architectures will follow Dedicated's Sidekiq setup to gain better capacity and segmentation of jobs. This aims to ensure customer's will see less Sidekiq queue build up.
  - Finished discovery for new testing approach - We'll be implementing a new testing approach that is notably more modular and lighter for faster feedback and even further confidence across the Reference Architectures and GET.\
  - Solution moved forward for for specific AWS EKS challenge with Persistent Volume storages clashes that can prevent deployment of Gitaly - Without this solved Gitaly will eventually fail to deploy on AWS EKS as the pods wouldn't be able to access their zonal discs.
Release & Deploy:
- Container Registry Post-Deployment Migrations Automation for GitLab.com: The MR to modify the registry Helm chart is under review. And we now have a way to run only Container Registry E2E tests in pre and staging.
- Extend the maintenance policy to account for three releases:
  - There are some backports from groupsource code planned for the upcoming patch release for the pilot
  - Documentation and templates have been updated to reflect the updated maintenance policy, so that once we roll out the extension, they will point to the SSoT for the updated maintenance policy and maintained versions.

Review 2025-06-26

Recording 📹 https://www.youtube.com/live/JPaR7KZsLxQ

Blockers ⚠️

Extend the maintenance policy to account for three releases:
- We're not getting much engagement and feedback for the pilot as we anticipated, so we're looking to extend the pilot to any group who's looking to backport bug fixes. We're also investigating into possible blockers for the pilot groups from backporting their bug fixes.

To Be Closed 📕

https://gitlab.com/groups/gitlab-org/distribution/-/epics/94+: The core goal to ensure UBI/FIPS pipelines are as complete and efficient as Debian pipelines has been achieved. This epic is completed and can be closed. We'll iterate on this in Q4, with evaluation of Docker and other vendor hardened images and the Debian migration
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/framework/-/epics/27+: The epic has been completed by establishing the Framework Readiness assessment content and a Proposed Delivery stage readiness process with ongoing collaboration with Self-Managed. This epic can now be closed.
Cloud Native Hybrid GET Molecule testing (gitlab-com/gl-infra/software-delivery/framework&30 - closed): New scenarios have been added for Molecule for both Cloud Native and Cloud Native Hybrid versions of GitLab. All issue have now been closed out, and all work is complete. This epic can now be closed.

Highlights 🎉

Build
- UBT: GKE-backed matrix QA testing is up and running. Engaged QA with the RFH and getting some momentum. Rust-based Ruby gems can now be cross-compiled.
- Pulp PoC: Pulp has been deployed to EKS (AWS) as they lack support for GCP blob storage (GCS). Started the evaluation process and already identified a few limitations, which are being discussed in #pulp_poc
Self-managed:
- Support Rollout of Container Registry for Self-Managed Instances:
  - The Geo and Backup/Restore support discussions are still ongoing. We are following and supporting
- Define feature review process for Self-Managed concerns:
  - A new project for or readiness assessment checklists, processes, and tools was created https://gitlab.com/gitlab-org/architecture/readiness. Further consolidation and review work are on track.
Framework:
- Cloud Native First Reference Architectures:
  - First test result runs posted for proposed Reference Architecture targets that show good results. Wrote up initial Q3 implementation plan draft for alignment.
Release & Deploy:
- Transform release environments into production-like self managed environments: The whole Geo-supported GET deployment is automated 🎉. A short demo is shown in the weekly Demo https://youtu.be/CYj5ymApSfE?t=107. A solution for OAuth integrated with GET was provided, so we can give access to all engineers to the environments in the future

Review 2025-06-19

Recording 📹

Blockers ⚠️

To Be Closed 📕

Highlights 🎉

Build
- UBT: Slower week due to resource constraints (people sick, on DRI etc.). QA-related progressed forward with GKE-backed tests. System library inventory notification mechanism spike is complete. Work on libclang (and related components) addition to toolchain is ongoing.
Self-managed:
- Support Rollout of Container Registry for Self-Managed Instances:
  - The MR to implement Create a Default Database for the Container Reg... (gitlab-org/omnibus-gitlab#8818 - closed) is now ready to review. Once this gets merged, for users using the GitLab-managed Omnibus PostgreSQL, this simplifies a bit the move from legacy registry filesystem metadata to database metadata. This is also one step further to providing HA support needed by GET and consequently dedicated.
  - The Geo and Backup/Restore support discussions are still ongoing. We are following and supporting
- Define feature review process for Self-Managed concerns:
  - A "how-to" guide is added to the checklist. It depicts the context and basic workflow on how to iteratively complete the checklist over feature maturity.
  - A significant number (100+) of comments and suggestion are provided and resolved on this MR and it has gone through a comprehensive review. This is ready to be merged in the next few days to form a baseline for Infrastructure Platform Readiness Assessment. If merged, we meet the first exit criteria of this epic.
- Navigating a path towards Cloud Native: Approach Evaluation: Evaluation based on PoC results started.
Framework:
- GET, 3.7.0 has been completed and released 🚀 with Gitaly Server Side Backups Support, Advanced Search Custom Config and Ansible Remote Environment variable support
- Discovery: Cloud Native First Reference Architectures:
  - Further refinements to proposed Reference Architecture designs and specs based on initial testing and feedback
  - Selected new suggested Machine Type targets from Cloud Providers
  - Started scoping for new docs approach with Technical Writing team.
  - Sought feedback on best Redis deployment approaches from various stakeholders
- Define Delivery Framework Operational Readiness Process: Synced with Self-Managed and Delivery teams to discuss Delivery stage process overview. General concerns are aligned - sheer size of current assessments (how to condense), Production .com deployment readiness before package distribution evaluation (how to balance between allowing teams POC versus providing full scope of bigger operational requirements for new components), various checklists in different places (how to deduplicate), assessment should be in stages of maturity.
Release & Deploy:
- Extend the maintenance policy to account for three releases:
  - 🛠️ Working alongside Guarantee the readiness of GitLab releases. (&1555 - closed) epic to improve the stable branch pipeline efficiency #20964 (closed) - This will bring great value by reducing the time for stable branches review for releases, given the work done in Update stable branches review on patches and in... (#21167 - closed)\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
  - 📝 Getting more documentation and template updates ready for the rollout of the extension
  - Next: Continue supporting the pilot backports. Currently there is one set of backports. We extended the pilot invitation to groups who have been requesting bug fix backports. We are considering ending the pilot earlier to expedite the rollout to get more feedback to the new policy. https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21044

Review 2025-06-12

Recording 📹 https://www.youtube.com/watch?v=ouMEPeYHPKs

Blockers ⚠️

To Be Closed 📕

Analyse gaps in our current development feature flag solution and process: The project transition to DevEx is complete 🚀, the work will continue in gitlab-org/quality/tooling&50, and Alessio will be available for consultancy from now on.

Highlights 🎉

Build
- UBT: Some issues were encountered working with Ruby and Rust. Subsequently we need to invest more time into components we didn't account for initially. On QA side we are close to having baseline for building distro-based containers.
- UBI images: We now produce multiarch variants for UBI and FIPS images. This makes them work natively on arm64 hosts. All the relevant components (like Zoekt, GitLab CE, etc.) now have UBI/FIPS variants.
Self-managed:
- Support Rollout of Container Registry for Self-Managed Instances: ~"group::Self Managed" and groupcontainer registry agreed on how to best split the work on the Gaps. We used group labels to clarify which teams will work on which issues. This issue distribution was also added to the Exit Criteria - Future of this epic.
- Define feature review process for Self-Managed concerns:
  - Add review checklist for Infrastructure Platfor... (gitlab-org/gitlab!191122 - closed) is workflowin review. It has received a lot of feedback and working towards merging it.
  - More than 70 questions that capture ~"group::Self Managed" requirements and concerns are formulated and organized in 6 different categories (for example "Installation and configuration", "Upgrade", "External dependencies", etc).
  - We are generalizing the process that was used to formulate ~"group::Self Managed" requirements in order to establish a framework to engage other ~"section::infrastructure platforms" groups.
- Implementation work for OpenBao: Evaluation method for inclusion into Helm deployment (https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/1751) leans to implementing a dedicated chart under gitlab-org/cloud-native/charts, with the work described in OpenBao: Implement dedicated chart under cloud-native umbrella, which will provide for inclusion into gitlab/gitlab.
- Navigating a path towards Cloud Native: Approach Evaluation: The PoC that moves the KAS to an embedded Kubernetes (K8s) is complete. This PoC will serve as the foundation for the upcoming evaluation. Preparing a demo and starting evaluation next
Framework
- Cloud Native First Reference Architectures (Project Flow):
  - First Reference Architecture designs and specs proposed and being discussed with stakeholders
  - Solidified proposal for new Sidekiq recommendations based on Dedicated learnings and validated designs - Delivers increased worker counts and separated queues for improved processing efficiency
  - Researched and proposed new comprehensive test strategy - New decentralised approach focuses testing on scalability and specification validation for the new dynamic architectures.
Release & Deploy:
- Container Registry Post-Deployment Migrations Automation for GitLab.com: Gathering information and designing a solution in #21188 (closed).
- Transform release environments into production-like environments:
  - A GET cloud hybrid installation with two clusters running in Geo was set up successfully, proved that we can use it for release environments.
  - Demoed running the Terraform part of GET on a CI pipeline https://youtu.be/e9NZb8PGA3s?t=58

Review 2025-06-05

Recording 📹

Blockers ⚠️

To Be Closed 📕

Highlights 🎉

Build
- UBT: Lots of granular progress. Confirmed that Omnibus can build Docker images based on other distros locally
Self-managed:
- Support Rollout of Container Registry for Self-Managed Instances: ~"group::Self Managed" and groupcontainer registry agreed on how to best split the work on the Gaps. We used group labels to clarify which teams will work on which issues. This issue distribution was also added to the Exit Criteria - Future of this epic.
- Implementation work for OpenBao: Beginning work to add OpenBao as optional component to the charts
Framework
- Define Delivery Framework Operational Readiness Process: Contributed to the self-managed requirements list to add Framework specific questions. Outlined current understanding on the process and requirements needed for readiness generally.
- Cloud Native First Reference Architectures (Project Flow): Started discovery items in close collaboration and discussion with various teams including Dedicated and Self Managed including Sidekiq queue approaches, machine types, specification levels and more.
Release & Deploy:
- We successfully remediated a critical vulnerability on all Dedicated instances without a public patch, using internal releases 🚀
- Extend the maintenance policy to account for three releases: ⚙️ Release tooling ready to update stable branch permissions when maintenance policy extension rolls out (implementation, rollout plan); Pilot ongoing, collaborating with the pilot groups
- Earlier branch creation for monthly releases: Collaboration with DevEx to encourage engineers to backport flaky failures on stable branches and brainstorming on testing efforts to support the monthly release

Review 2025-05-29

Recording 📹

Blockers ⚠️

To Be Closed 📕

Highlights 🎉

Build
- UBT: Currently Gitaly compilation is blocked. This is not in present critical path, but it's worth mentioning.
- Packagecloud replacement: Legal and compliance discovery has been started for pulp evaluation
- https://gitlab.com/groups/gitlab-org/distribution/-/epics/94+: Work is going to to make UBI pipelines as similar to Debian pipelines as possible
Self-managed
- Self managed: Support Rollout of Container Registry for Self-Managed Instances: We've completed the feature gaps investigation and closed the related issue Investigate and identify gaps for Rollout of Co... (gitlab-org/gitlab#525473 - closed). We'll continue to work on alignment with the registry team
- Navigating a path towards Cloud Native: Approach Evaluation: PoC now deploys Cloud-Native KAS in the embedded K8s. Wiring between traditional Omnibus components and Cloud-Native is WiP.
Framework
- Project Flow is a proposal to fundamentally shift the approach with the GitLab's Reference Architectures 0 moving from static, VM-based configurations to dynamic, cloud-native deployments that embrace autoscaling, workload-based sizing and more.
Release & Deploy
- Decommissioning release.gitlab.net, shifting to upgrade path testing - The update path job ran successfully during the release of 18.0.1 🎉, so we have the update path ran successful for a monthly release and a patch release.
- Extend the maintenance policy to account for three releases: 🚀 The pilot for extending the bug fix backporting policy is starting with the upcoming patch release, with groupsource code and groupcode review. https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21044

Review 2025-05-22

Recording 📹

Blockers ⚠️

To Be Closed 📕

Highlights 🎉

Build
- UBT: Aligned with Marin on project progress on plan. The team will evaluate how to unlock benefits early and iteratively by rolling out UBT component by component.
- Packagecloud replacement: We will proceed with Pulp evaluation in Q2, to start implementation in Q3. We will also meet with the Pulp team to discuss a potential partnership.
Self-managed
- Self managed: Support Rollout of Container Registry for Self-Managed Instances: We continue to drive discussions to refine Geo support and Backup/Restore in close collaboration with the Container Registry team
- Navigating a path towards Cloud Native: Approach Evaluation: Progress on KAS as PoC for the strangler fig pattern (using k3s)
Release & Deploy
- Analyse gaps in our current development feature flag solution and process: We had a successful kickoff meeting with DevEx. Lots of feedback and discussion in the Feature Gate proposal doc. We'll go on to support DevEx with technical guidance
- Extend the maintenance policy to account for three releases: Created transparency for customers of maintained versions on our website. We are working with Product Marketing to further iterate on that page.

Review 2025-05-15

Recording 📹 https://www.youtube.com/watch?v=7DcTSn157iA

Blockers ⚠️

https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/24+: Design still under heavy discussion by Package Registry team, no progress, scope still unclear by the team. We are having another round of discussion today to build clarity and understand the need to move this into risk mitigation

To Be Closed 📕

Add support for PostgreSQL 16 (gitlab-org&12172 - closed): Postgres 16 is now the default and only supported version for GitLab 18.0.
Deprecation of 32-bit Raspberry Pi OS (gitlab-org/distribution&91 - closed): We no longer build packages for Raspberry Pi OS 32 or 64 bit. Users are now pointed to Debian ARM64 install instructions going forwards.
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/framework/-/epics/19+: Was reviewed before but not closed so far. Results.
Delivery: Rails 7.1 upgrade rollout to .com and... (&1513 - closed)

Highlights 🎉

Build
- UBT: PG16 now builds and smoke-tested using Omnibus Container. This is arguably the hardest component to cross compile. We also started on-boarding two new self-managed team members to the project. We also started to actually close issues to provide visibility of prgoress
Self-managed
- Navigating a path towards Cloud Native: Approach Evaluation: (PoC) Tooling to manage Cloud Native GitLab added to Omnibus.
Framework
- Define Delivery Framework Operational Readiness Process: The goal of this epic is to build a standardized readiness process for GitLab Delivery Framework, ensuring quality validation for Reference Architectures and GitLab Environment Toolkit while aligning with broader feature readiness initiatives.
Release & Deploy
- Extend the maintenance policy to account for three releases: Increasing visibility of maintained versions for both self-managed users and internal users. We got some initial ideas and suggestions, so we've started to implementing them, for example in the releases page. We are further going to iterate on this page with product marketing.
- Analyse gaps in our current development feature flag solution and process: The feature gates document has been refined to set a direction for the project, touchpoints with DevEx were established
- Earlier branch creation for monthly releases.: First implementation issues completed. Good collaboration with DevEx to guarantee stability on release branches ([1], [2], [3])
- Transform release environments into production-like environments: With the extension of the maintenance policy and the earlier creation of the stable branch, engineers need to be enabled to validate their changes in a production-like environment. This epic will make release environments usable for engineers for that purpose.

Review 2025-05-08

Recording 📹 -

Blockers ⚠️

UBT: while we were able to build and run POC PG14 deployment, PG16 requires further work due to changes in requirements between two versions.

To Be Closed 📕

Highlights 🎉

Build:
- Deprecation of 32-bit Raspberry Pi OS: MR that removes Raspberry Pi package builds in progress, link
- UBT: Portion of Epic descriptions have beend reviewed and populated. Remainder is being worked on.
- CNG: Deprecate Debian based images, in favor of UBI and micro pattern: Some progress, for improvements in vulnerabilities at the example for the workhorse image, see here.
Self-Managed:
- Support Rollout of Container Registry for Self-Managed Instances: After having identified everything that needs to be implemented, we're working on refining issues.
- Define feature review process for Self-Managed concerns: The first draft of the feature checklist is prepared. In its current state, it is a structured list of questions. It requires validation and in need of a better form for its presentation.
- Navigating a path towards Cloud Native: Approach Evaluation: We're currently developing a proof of concept that implements the Strangler Fig Pattern. This prototype will migrate one service to a Kubernetes cluster embedded within Omnibus and will serve as the foundation for evaluating and comparing the different proposals.
Framework:
- Upgrade data internal handbook page
- Dedicated Tenant on ARM architecture benchmark docs and Video walkthrough 9 min
Release:
- Earlier branch creation for monthly releases.: The plan for the epic was completed. Issues were created.
- Extend the maintenance policy to account for three releases: Started scoping, discussions, and analysis. Looking

Review 2025-05-02

Recording 📹 - https://www.youtube.com/watch?v=YWT-2gUu3YE

Blockers ⚠️

Automated Cells Deploy and Change Orchestration: We are not blocked, however, reviews from Environment Automation are slower than desired

To Be Closed 📕

Review .org independency from .com: All the images from distribution projects that were being pulled from .com in .org are now mirrored in .org and being pulled from .org. With that we have a more secure place to build the packages that GitLab uses to deploy GitLab.com and that we provide to our self-managed users and Dedicated.
Expand GitLab Environment Toolkit testing

Highlights 🎉

Build:
- Universal Build Toolchain: we have been able to build and to run PostgreSQL using POC toolchain. This is the first component we manage to cross-compile using the new toolchain!
- Reevaluate Package Cloud replacement: The Package Team proposed a design to help reduce the dependency on gitlab.com, which is very similar to the design proposed by the Build Team initially. We are evaluating our options to accelerate this project and understand the feasibility of delivering it by end of Q3.
Self managed:
- Support Rollout of Container Registry for Self-Managed Instances: We've found multiple gaps in different areas that we believe are key to the success of the registry database rollout. These were documented at gitlab-org/gitlab#525473 (closed). Geo support will be very complex and will need to be evaluated separately
Framework:
- Framework General Maintenance - FY26Q1:
  - It has been concluded to transition Staging Ref to the GitLab Delivery: Deployments team. This transition will be concluded in FY26-Q2.
  - 9 internal requests on reference architecture and GET were handled (4 from PS, 3 from Support, 1 CSM, 1 SA)
- Reevaluating Priorities for Upgrade Path Testing: Finished up with key insights and testing strategy analysis- gitlab-com/content-sites/internal-handbook!6691 (merged) is ready for review, team is reviewing and sharing feedback
- Support Dedicated rollout of a test Tenant running on ARM: Finished up calculation and refinements for MR with results of Dedicated Tenant on ARM architecture benchmark. Current calculated total environment savings would be 5.34-6.28%. We'll have a further sync with FinOps team on actual rates for EC2 instances with GitLab's savings plan.
Release:
- Earlier branch creation for monthly releases: The monthly release process does not allow enough time for QA or emergency fixes before the final release, it is tightly coupled with the auto-deploy process. This quarter, we'll move the active stable branch generation one week earlier in the release schedule to allow for thorough testing and timely backports.
Deploy:
- Automated Cells Deploy and Change Orchestration: We've decided to go with the approach of setting feature flags from the Rails console of each Cell. See https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/20589#note_2465012521.
- Delivery FY2026Q1 technical debt projects: We have addressed 18 tech debt issues this quarter - 15 issues are closed and 3 are just about done. Notably, dev.gitlab.org has been upgraded to Ubuntu 22.04 to be able to keep OS patches up to date and has an updated readiness plan for future work.

Review 2025-04-17

Recording 📹 https://www.youtube.com/live/wxW1aVHzke0

Blockers ⚠️

https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/framework/-/epics/17+: Worked with FinOps team on extracting costs metrics and it turned out that GitLab doesn't have access to hourly cost granularity, only Daily. And Daily cost results are not compatible as sandbox environments were destroyed or stopped after testing which didn't end up in full day. I raised Rerun tests for x86 and ARM to extract daily costs (software-delivery-framework-issue-tracker#71) to rerun the test and ensuring environments are available for a full day for fair comparison.

To Be Closed 📕

Expand GitLab Environment Toolkit testing (gitlab-com/gl-infra/software-delivery/framework&25 - closed)

Highlights 🎉

Build:
- UBT POC will pursue build of PostgreSQL attempting to use produced binary along with Omnibus to prove viability of the approach. We are already seeing some early success.
- Project plan refined, epics restructuring in progress, using roadmaps for visibility
Framework:
- Extraordinary, unplanned, priority support for https://gitlab.com/gitlab-com/request-for-help/-/issues/2674+ in order to unblock Duo testing
- Upgrade issues support ticket data
  - After reclassification of almost 500 tickets, it was identified that about 32% are upgrade related and 68% are not directly related to upgrade. The 68% were actually regressions (17%), license issues (17%), configuration problems (10%), self-inflicted issues(7%), etc. See graphs and details here
  - Top 3 sub-categories in re-classified Upgrade-Related Issues are: 1) Upgrade Path & Planning (40%), 2) Upgrade Migration Issues (36%) 3) Pre/post upgrade issues (11%, where 7% is backup/restore issues)
Internal Releases:
- Rollout phase 2 of internal release to dedicated instances (on Switchboard Production instances) during the upcoming patch release (https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/20957)
- 📝 There's a draft of the internal announcement issue. It'd going through review, then once the Switchboard Production rollout above is completed successfully, it'll be announced in slack channels.

Review 2025-04-11

Recording 📹 No Recording this week

Blockers ⚠️

To Be Closed 📕

Address IngressNightmare vulnerability work is complete - https://gitlab.com/groups/gitlab-org/distribution/-/epics/105
GET 3.6.0 is release - gitlab-com/gl-infra/software-delivery/framework&23 (closed)

Highlights 🎉

RPi 32 Bit images have been stopped and removed from the page - we are finalizing the last pieces in removing RPi 32bit documentation in 17.11 and point users to Debian ARM documentation gitlab-org/distribution&91 (closed)
Upgrade Path Testing finalized the categorization of ~500 support tickets so Upgrade Path Testing can be more data driven and focus on where we can create the most value
Framework team supporting the ARM on Dedicated - we have a 3k CNH sandbox running. GitLab application is working as expected, however monitoring setup used in Instrumentor doesn't work in ARM due to underlying images like Tamland and config-manager not supporting the architecture
Internal releases:
- Rollout phase 1 of internal release to dedicated instances (on Switchboard Test instances) has been successfully completed https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/20957
- Rollout phase 2 of internal release to dedicated instances (on Switchboard Production instances) https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/20957 is about to start

Review 2025-04-03

Recording 📹 https://www.youtube.com/live/Y00uy7Tr9Ho

Blockers ⚠️

To Be Closed 📕

Docker Hub limitations mitigation: great job, no disruption: https://gitlab.com/groups/gitlab-org/distribution/-/epics/104+
Vault Token Automation: slow burned down token-related tech debt: Improve token related automation for Delivery p... (&1335 - closed)

Highlights 🎉

Collaboration with package team on https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/24+ has started. We'll meet biweekly to keep close track of progress and support the team.
Framework team supporting the ARM on Dedicated - https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/framework/-/epics/17+: Have onboarded to Dedicated and started provisioning a test environment.
Internal releases:
- Established two-phase rollout plan of the internal release for Dedicated instances
- Information about the release is available on the handbook now
- We are planning to perform the first rollout of the internal release to Dedicated during the upcoming patch release

Review 2025-03-27

Recording 📹 https://youtube.com/live/74VcWVRW2gU

Blockers ⚠️

Self managed: Support Rollout of Container Regi... (gitlab-org&17005) has been paused in order to prioritize mitigating https://gitlab.com/groups/gitlab-org/distribution/-/epics/105+
Vulnmapper automation broke on Security-side, causing a backlog of 440 vulnerabilities to triage and remediate. This is blocking UBT short-term as the team needs to swarm the CVEs. We are also evaluating how to make that automation more reliable.
DockerHub Limitations that will come into

To Be Closed 📕

nothing

Highlights 🎉

Dockerhub pull limits have been addressed for distribution teams. Other delivery teams have no major dependencies on dockerhub. We should be clear 🤞
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/framework/-/epics/19+ is progressing and surfacing interesting insights on Upgrade issues from support ticket data. See here for some data.
- we are noticing that data quality is not great (only 35% of "upgrade related" tickets are actually upgrade related), but relevant tickets highlight a lot of issues that are self-inflicted (incorrect configuration or feature use) or down to insufficient documentation or upgrade process transparency.

Review 2025-03-20

Recording 📹

Blockers ⚠️

nothing substantial

To Be Closed 📕

Explore options to expand GitLab Environment To... (gitlab-com/gl-infra/software-delivery/framework&20 - closed)

Highlights 🎉

Internal releases:
- We have the ability to create internal packages
- We are discussing communication and automation with the Dedicated team #20161 (closed)
- We are working on internal and external announcement #20956 (closed)

Lights 💡

We worked on understanding our options for the dockerhub limitations
GET 3.6.0 Release has been delayed to first week of April
Operator V2 will be paused in order to focus resources on Build team projects

Review 2025-03-13

Recording 📹 https://www.youtube.com/watch?v=8MXGqzWxP7o

Blockers ⚠️

UBT resourcing:
- had to divert attention to other pressing matters within ~"group::distribution"
- increased team workload (due to reduced team size) prohibits fast onboarding of new resources

To Be Closed 📕

~"group::distribution" self-managed IaC Distribution managed infrastructure and env... (gitlab-org/distribution&67 - closed)
~"team::GitLab Delivery Framework" https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/framework/-/epics/13+
~"team::GitLab Delivery Framework" Explore options to expand GitLab Environment To... (gitlab-com/gl-infra/software-delivery/framework&20 - closed)
~"team::Delivery-Releases" Ability to create internal packages to remediat... (&1373 - closed)

Highlights 🎉

Four (4!) items to close this week. Thank you everyone for the great !
Packagecloud replacement: We have aligned with the Package Registry and the Product Team to invest in building a solution in house that is fully integrated with the product, as part of our goals of competing with Artifactory.
Internal releases
- We successfully tested all the upgrade and installation paths for internal releases on Dedicated instances using Switchboard Test.
- We can now create internal packages and deploy them on Dedicated! We are still working on the end to end strategy including customer comms.
- After discussion with ~"group::GitLab Dedicated", we now have a list of information and correspondents surrounding communication with Dedicated during the internal releases cycle https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/20906

Review 2025-03-06

Recording 📹

Blockers ⚠️

Cells orchestration:

Work on Cells orchestration has slowed down due to work on the new SDLC proposal

To Be Closed 📕

nothing

Highlights 🎉

Build & Self-Managed

Work started on Universal Build Toolchain (UBT). UBT is the first part of our Build system overhaul targeting build efficiency and will decrease build times significantly, with associated MTTP and RTO reduction. We are working on a higher level description of the plan and the value it creates.
Work started on deprecating Debian images in favor of UBI. These have a reduced footprint and security surface and will speed up CNG pipelines while providing more secure images.
GitLab Package Tools. After meeting with Stan and the Packages Team, we agreed to collaborate on https://gitlab.com/gitlab-org/gitlab/-/issues/523482+ to determine if we can build a solution that will be part of the product.

Framework

GET 3.6.0 Release planned for end of March focused on adding more Cloud Service restore options for Dedicated, Cells and Self Managed customers along with other new features and improvements.
GET testing advanced with a blueprint. We'll directly go on to implement this blueprint.

Deploy & Release

Internal releases:
- All upgrade paths tested
- Successfully generated internal release package using release automation
- Testing a fresh install of an internal release on Dedicated instance next
- started discussion with Dedicated about how to communicate internal releases

Review 2025-02-27

Recording 📹

Blockers ⚠️

Nothing major, only mentioning it:

Operator: QA work blocked by provisioning CI cluster
Review .org independency from .com: Access to dev issue bot missing

To Be Closed 📕

nothing

Highlights 🎉

Progress on internal releases:
- Upgrade testing making good progress
- Testing on Dedicated is starting
GET now uses milestones to improve the transparency in collaboration with internal and external customers
On the goal of separating .com and .org to avoid a single point of failure and to make .org .org a safer place to build and sign our packages and images, we've refined the Epic’s scope by completing all investigation issues and defining clear follow-ups. Thanks to this refinement and planning we have a clearer and sharper scope to deliver

Not a highlight but worth surfacing

Implementation of package tools is going back into the design phase, as packagecloud has been successfully renewed :wi

Review 2025-02-20

Recording 📹 https://www.youtube.com/watch?v=Uc4D02MSIxQ

Blockers ⚠️

FYI Only ~"group::distribution" SIRT 6534 activities widely consumed the resources of the team to mitigate and remediate the incident.

To Be Closed 📕

None this week

Highlights 🎉

~"group::distribution" Watch our first operator demo 🍿
~"team::GitLab Delivery Framework" Explore options to expand GitLab Environment To... (gitlab-com/gl-infra/software-delivery/framework&20 - closed) Ruled out Kitchen CI and have looked into Molecule. Initial look at Molecule seems promising but I'm currently running into issues trying to get GitLab installed inside a Docker image. This is more of an issue with Docker and not Molecule specific.
~"team::Delivery-Releases" Ability to create internal packages to remediat... (&1373 - closed)
- 🧪 Testing scenario to upgrade a test Dedicated instance from a patch to an internal release version was successfully completed https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/20837#note_2352580124
- 📓 Discussion on how to indicate Dedicated folks that an internal release version is available is in progress #20898 (closed)
- The jobs to create an internal package are being transformed to a downstream pipeline so the creation of packages can be idempotent and executed in parallel #20865 (closed)
~"team::Delivery-Deployments" Automated Cells Deploy and Change Orchestration (&1436 - closed)
- Blueprint for helm chart auto-deployments is merged
- Planning completed for helm chart auto-deployments, this is currently slated for Q1 as part of Phase 4.4

Review 2025-02-13

Readout

Business as usual this week with tech summit, incidents and other operational items. Closing the GET 3.5.0 release on schedule.

Recording 📹

Blockers ⚠️

FYI only - ~"group::distribution" [PACKAGING PROJECT] https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/2 - We are about to reach a GCP storage quota limit due to the large number of pre-release packages we're hosting. I've already requested a quota increase, but it might be better to simply delete most of those pre-release packages if they're not needed.
FYI only - ~"group::distribution" [UNIFIED BUILDS] We are down to four Go-based projects that require response. See open merge requests attached to https://gitlab.com/gitlab-org/distribution/build-architecture/framework/utilities/core/-/issues/1+s

To Be Closed 📕

~"team::GitLab Delivery Framework" GET - 3.5.0 Release - The work for this Epic was completed last week. This epic can be closed in the Grand Review.

Highlights 🎉

~"group::distribution" https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/2 - We are copying packages from PackageCloud! The first run will take a long time to complete, but the repositories will be live when they finish and will update every 2 hours. About 4.2 TiB is copied so far, most of which are pre-release packages.
~"team::Delivery-Deployments" &1335 (closed) - One more token was moved to vault and there are two more in review
~"team::Delivery-Releases" - 🧪 For testing purposes, internal release packages for a fake version (42.2) were successfully created #20859 (comment 2340637477). Omnibus and CNG pipelines were triggered, successfully completed (1, 2, 3, 4), and packages were available on the pre-release channel
- - Automation was added to the internal release pipeline to review the stable branches before merging security fixes #20866 (closed) / gitlab-org/release-tools!3876 (merged)

Review 2025-02-06

Readout

4 items to close this week along with some end of quarter summaries!

Recording 📹 https://www.youtube.com/watch?v=e_-MEvPXyqs

Blockers ⚠️

No Blockers

To Be Closed 📕

~"team::GitLab Delivery Framework" Add GitLab Operator support to GitLab Environme... (gitlab-org/quality/quality-engineering&95 - closed) - closing summary
~"group::distribution" Automate creation, maintenance, and rotation of access tokens used in Distribution projects - closing summary
~"team::GitLab Delivery Framework" Switch 10k cloud native hybrid static environme... (gitlab-com/gl-infra/software-delivery/framework&15 - closed) - this should unblock the rollout of gitaly on k8s for Cells and Dedicated when we are ready
~"team::Delivery-Deployments" Decouple auto_deploy tagging from rollout - closing summary

Highlights 🎉

~"group::distribution" we'll make an official announcement in %17.9 and stop building RPi 32-bit packages in %18.0
~"group::distribution" We optimized our CI pipelines with off-cluster standalone unit tests. Operator pipelines currently run under 15 minutes.
~"team::Delivery-Releases" 💻 Switchboard has been updated to support internal release packages #20627 (closed)
- 🎥 New demo showing the adaptations required on the Dedicated tooling (Instrumentor and Switchboard) to support internal releases
~"team::Delivery-Deployments" Cells Q4 recap

Edited Oct 23, 2025 by Martin Brümmer