FY26 Q4 GitLab Delivery Stage Highlights
Group Top Level Summary: Infrastructure Platforms - GitLab Delivery (&1451)
Review 2026-02-05
Recording 📹
Blockers ⚠️
To Be Closed 🎉
-
Document Zero Downtime Upgrades Support with Gi... (gitlab-com/gl-infra/software-delivery&16 - closed)
- Traditionally, Zero Downtime Upgrades were not supported in Cloud Native deployments, only in Omnibus HA. This has actively driven customers to adopt VM-based deployments instead of Cloud Native ones. The absence of public support and documentation for this process was hard to understand, as we have been updating our CNH instances with Zero Downtime for years
- With this documentation, this gap has been resolved and we now have public support and documentation of ZDU in Cloud Native Deployments.
- We'll go on to provide automation for this process in GET
- We'll publish a release post item and announce with Support and CS
- Thank you @nwestbury for delivering this
-
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/epics/17+
- So far, our reference architectures present Omnibus-based scaled out deployments as the primary choice, without a public preference for Cloud Native deployments. This is an issue hampering our messaging for a migration to CNG
- Additionally, those reference architectures were signalling user counts to assess their size, which has been misleading users and has been a problem for Dedicated migrations
- A new reference architecture documentation has now been published for CNG deployments. It is still in Beta as long as Gitaly on Kubernetes is not GA.
- The documentation eliminates user counts as size indicator, opting for a t-shirt size approach instead and paves the way for our CNG transformation
- Thank you @grantyoung for delivering this
-
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/epics/30+
- We didn't have much knowledge about how the small kubernetes distribution candidates (k0s, k3s, microk8s) work. We had to explore service exposure, resource consumption, OS support, security feature, network setup, Zero Downtime upgrade support, etc.
- We've verified that all these 3 distributions can be used within the OAK architecture. We also acquired enough knowledge to support taking the next decisions regarding what OAK should support.
- We documented in-depth findings and will go on with the technical design phase for OAK
- Thank you @Alexand for driving
Highlights 🚀
- Build:
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Landed 2 UBT components in Omnibus.
- Second CI iteration has been merged allowing for automated (unattended) UBT package builds.
- FIPS spike completed documenting existing gaps.
- We are on-track according to current timelines.
-
🕐 total hours spent this week by all contributors: 56h
-
https://gitlab.com/groups/gitlab-org/build/-/epics/6+:
- TL;DR: We are on track with our deliverables and things are progressing according to plan.
-
gitlab-runnerpackages are now served from Pulp. This means every package we ship to customers are now served from Pulp.🚀 - Pulp application metrics are integrated into monitoring stack
-
🕐 total hours spent this week by all contributors: 94 hours
- BAU:
- DRI work, MR reviews/testing, Renovate
-
🕐 total hours spent this week by all contributors: 10 hours
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Operate:
-
Cloud Native GitLab: Move from NGINX Ingress to... (gitlab-com/gl-infra/software-delivery/operate&5)
- Support for GitLab Geo via Envoy is in review.
- Exposing Envoy Gateway in GET is work in progress.
- Initial Gateway API support for GitLab Operator is in review.
- More flexible Gateway configuration for GitLab chart is in review.
-
🕐 total hours spent this week by all contributors: 15 hours
-
Operate Team BAU - FY26Q4 (gitlab-com/gl-infra/software-delivery&29 - closed):
- Interviews for Delivery stage
- Breaking Change approvals
- Relative URL spike work
- GET, Omnibus and Charts BAU work
- Distribution DRI
- PREP reviews, specifically AUTH (GATE) advice
- Handled questions and requests from other teams
-
🕐 total hours spent this week by all contributors: 64 hours
-
Cloud Native GitLab: Move from NGINX Ingress to... (gitlab-com/gl-infra/software-delivery/operate&5)
- Release & Deploy:
-
⚠️ Releases and Deployments FY26Q4 KTLO and techni... (&1772 - closed):- A lot of coordination of internal releases and potential patches for DAP related activities. Not sure all hours are there yet.
🕐 total hours spent this week by all contributors: 88 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/epics/20+:
- The GitLab Release Platform design-doc is finally merged
🚀 - We'll now go on to break this down and start implementation
- The GitLab Release Platform design-doc is finally merged
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
- Another slower week due to RM duties
- Only a handful of open issues now, so we are close.
-
🕐 total hours spent this week by all contributors: 14 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1753+:
- PDMs were successfully executed on all cells via the ring based deployment system
- We are now working on hooking up PDM execution on cells to the regular procedure that RMs perform to execute PDMs on gitlab.com.
🕐 total hours spent this week by all contributors: 9 hours
-
Delivery: Ruby 3.3 rollout to .com and self-man... (&1698 - closed):
- Update Ruby upgrade runbook from the experience we learned during this upgrade (gitlab-org/release/docs!993 (merged))
🕐 total hours spent this week by all contributors: 4 hours
-
Previous Statuses
Review 2026-01-29
Recording 📹
Blockers ⚠️
To Be Closed 🎉
Highlights 🚀
- Build:
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Landed 3 UBT components in Omnibus (3 more in-flight). This brings us to 16 out of ~100 components total. Merged first iteration for UBT CI/CD implementation. Currently observing slight uptick in workload as version updates now (temporarily) span across Omnibus, UBT and CNG.
- We are on-track according to current timelines.
-
🕐 total hours spent this week by all contributors: 53h
-
https://gitlab.com/groups/gitlab-org/build/-/epics/6+:
- TL;DR: We are on track with our deliverables and things are progressing according to plan.
-
gitlab-runnernow publish packages to Pulp in addition to Packagecloud. - All
pre-releaseconsumers are now getting their packages from Pulp. - Cloudflare cost estimation completed - https://gitlab.com/gitlab-org/build/team-tasks/-/work_items/96
-
gitlab-fipspackages now served from Pulp. - Pulp UI deployment progressing well.
-
⚠️ PubSec isn’t using Pulp just yet: https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/instrumentor/-/merge_requests/6868#note_3021613035 -
🕐 total hours spent this week by all contributors: 80 hours
- BAU:
- DRI work, MR reviews/testing, Renovate
-
🕐 total hours spent this week by all contributors: 57 hours
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Operate:
-
Cloud Native GitLab: Move from NGINX Ingress to... (gitlab-com/gl-infra/software-delivery/operate&5)
- Initial Gateway API support for GitLab Operator is in review.
- More flexible Gateway configuration for GitLab chart is in review.
- Breaking change exception still pending approval by Senior Director of Infra Platforms
-
🕐 total hours spent this week by all contributors: 8 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/epics/30+:
- We're close to wrapping up the discovery work for all our 3 initially proposed small kubernetes distribution: k0s, k3s, and microk8s.
-
🕐 total hours spent this week by all contributors: 47 hours
-
Document Zero Downtime Upgrades Support with Gi... (gitlab-com/gl-infra/software-delivery&16 - closed)
- Actioned initial feedback on gitlab-org/charts/gitlab!4744 (merged) from multiple reviewers (technical writers and maintainers). Started received approvals and just need to get this merged for completion.
- This issue will deliver on a long-standing customer request for Zero Downtime Upgrade support in Cloud Native deployments. The lack of this support has actively driven customers to adopt Omnibus HA for their scaled deployments. Delivering this support is a major signal for customers to adopt CNG.
-
🕐 total hours spent this week by all contributors: 4 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/epics/17+:
- Further numerous updates and polish to the documentation based on feedback
- Like the ZDU support for CNG, this issue will deliver a strong signal for customers to adopt CNG and lay the foundation to consolidate all GitLab instance towards the Unified Deployment Unit
-
🕐 total hours spent this week by all contributors: 6 hours
-
Operate Team BAU - FY26Q4 (gitlab-com/gl-infra/software-delivery&29 - closed):
- Interviews for Delivery stage
- Breaking Change approvals
- Helping Support with a large escalation
- Advising on notable performance issue
- GET, Omnibus and Charts BAU work
- Helm 4 support in Charts and Operator
- Distribution DRI
- PREP reviews, specifically AUTH (GATE) advice
- Handled questions and requests from other teams
-
🕐 total hours spent this week by all contributors: 55 hours
-
Cloud Native GitLab: Move from NGINX Ingress to... (gitlab-com/gl-infra/software-delivery/operate&5)
- Release & Deploy:
-
⚠️ Releases and Deployments FY26Q4 KTLO and techni... (&1772 - closed):- A lot of coordination of internal releases and potential patches for DAP related activities. Not sure all hours are there yet.
🕐 total hours spent this week by all contributors: 88 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
- Another slower week due to RM duties
- Worked on a runbook which will serve as the entrypoint for RMs when things go wrong in the Cells world: Resolving Patch Failures
-
🕐 total hours spent this week by all contributors: 14 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1753+:
- We are upgrading the version of Instrumentor used in Tissue, to a version that includes a couple of required bug fixes. After that we can re-introduce a patch to execute PDMs on cells.
🕐 total hours spent this week by all contributors: 11 hours
-
Delivery: Ruby 3.3 rollout to .com and self-man... (&1698 - closed):
- GitLab.com is now running on Ruby 3.3.10
🎉 production#21065 (closed) - GitLab documents and CI are updated to prepare for Ruby 3.3 from the milestone 18.9
🕐 total hours spent this week by all contributors: 11 hours
- GitLab.com is now running on Ruby 3.3.10
-
Time Spent 🕐
We are still iterating on how to capture BAU more accurately, as this is still underrepresented.
| Team | Projects | BAU, KTLO, RFH, Release Management, Interrupts |
|---|---|---|
| Build (includes Deploy support for Pulp) | 133h | 57h |
| Operate | 65h | 55h |
| Release & Deploy | 58h | 88h |
Review 2026-01-23
Recording 📹
Blockers ⚠️
To Be Closed 🎉
Highlights 🚀
- Build:
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- We are on track to deliver the "Automated compiler and toolchain build" and "source code sidecache" milestones, which will finally allow us to automatically build UBT and its components via CI, in a reproducible manner and facilitate the the onboarding and integration of UBT-built components into Omnibus.
- We are on-track according to current timelines.
-
🕐 total hours spent this week by all contributors: 56h
-
https://gitlab.com/groups/gitlab-org/build/-/epics/6+:
- TL;DR: We are on track with our deliverables and things are progressing according to plan.
- Deployments to GitLab.com now uses packages from Pulp - https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/21081
- Deployments to pre.gitlab.com and release.gitlab.net now uses packages from Pulp - https://gitlab.com/gitlab-org/build/team-tasks/-/work_items/114
- We can now generate repository configuration scripts for all the repos, like Packagecloud does, for easier consumption by users
- We are progressing with deploying Pulp UI using Runway.
-
⚠️ PubSec isn’t using Pulp just yet: https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/instrumentor/-/merge_requests/6868#note_3021613035 but should be resolved next week -
🕐 total hours spent this week by all contributors: 36 hours
- BAU:
- DRI work, MR reviews/testing
-
🕐 total hours spent this week by all contributors: 30 hours
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Operate:
-
Cloud Native GitLab: Move from NGINX Ingress to... (gitlab-com/gl-infra/software-delivery/operate&5)
- Initial Gateway API support for GitLab Operator is in review.
- Support for OpenBao via Gateway API is merged.
- Breaking change exception still pending approval
-
🕐 total hours spent this week by all contributors: 20 hours
-
Omnibus Adjacent Kubernetes (OAK) - Operate Imp... (gitlab-com/gl-infra/software-delivery#22):
- We've concluded the discovery work of
k0s, and validated that, likek3s, it's also a valid option as a small kubernetes distribution for OAK. -
microk8sdiscovery is upcoming -
🕐 total hours spent this week by all contributors: 15 hours
- We've concluded the discovery work of
-
Document Zero Downtime Upgrades Support with Gi... (gitlab-com/gl-infra/software-delivery&16 - closed)
- Opened gitlab-org/charts/gitlab!4744 (merged) updating the documentation for Zero Downtime. Sent for initial review.
- Opened gitlab-org/gitlab!218945 (merged) to remove warnings about zero downtime with the charts
-
🕐 total hours spent this week by all contributors: 20 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/epics/17+:
- Further numerous updates and polish to the documentation draft
-
🕐 total hours spent this week by all contributors: 4 hours
-
Operate Team BAU - FY26Q4 (gitlab-com/gl-infra/software-delivery&29 - closed):
- Interviews for Delivery stage
- Breaking Change approvals
- Helping Support with a large escalation
- Advising on notable performance issue
- GET, Omnibus and Charts BAU work
- Distribution DRI
- PREP reviews
-
🕐 total hours spent this week by all contributors: 72 hours
-
Cloud Native GitLab: Move from NGINX Ingress to... (gitlab-com/gl-infra/software-delivery/operate&5)
- Release & Deploy:
-
⚠️ Releases and Deployments FY26Q4 KTLO and techni... (&1772 - closed):- We have done the monthly (18.8) release, 18.8.1 special release, patch release (18.8.2, 18.7.2, 18.6.4), and 3 internal releases in support of DAP rollout
- We continue to support, planning an out-of-band patch next week to fix critical DAP bugs
- We are urging caution for Dedicated as we have various tenants in various states of internal releases and are moving outside of established processes, causing widespread customer impact
🕐 total hours spent this week by all contributors: 133 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
- Slower week due to RM duties
- An MR for implementation of auto-deploy in Cells is proceeding through review
- Jobs for tracking deployments to Cell are stable now: chore: disable allow_failure on record deployme... (ringctl!448 - merged)
🎉 -
🕐 total hours spent this week by all contributors: 26 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1753+:
- An Instrumentor MR was merged to fix a bug that was affecting GCP cells. After we upgrade the Instrumentor version in Tissue, we will retry adding a patch to execute PDMs in Cells.
🕐 total hours spent this week by all contributors: 17 hours
-
Time Spent 🕐
We are still iterating on how to capture BAU more accurately, as this is still underrepresented.
| Team | Projects | BAU, KTLO, RFH, Release Management, Interrupts |
|---|---|---|
| Build (includes Deploy support for Pulp) | 92h | 30h |
| Operate | 59h | 72h |
| Release & Deploy | 43h | 133h |
Review 2026-01-15
Recording 📹
Blockers ⚠️
To Be Closed 🎉
-
Readiness Rollout: Limited Adoption (gitlab-org/distribution#127 - closed):
-
GitLab's existing Production Readiness Review only covered GitLab.com, leaving no systematic validation for Self-Managed and Dedicated platforms. This resulted in features reaching GA with critical issues blocking customer adoption.
PREP fills this gap by providing a self-service framework for product teams to validate feature readiness across the entire GitLab platform ecosystem before release.
-
We have finished our limited adoption phase with measurable progress:
- 3+ infrastructure teams engaged beyond initial sectiongitlab delivery scope, establishing PREP as cross-functional infrastructure practice
- 3 production features assessments are underway or completed, identifying critical infrastructure gaps before production
- PREP Assistant automation reduced assessment time by 60-70%, exceeding our expectations (40% manual work reduction target)
- LevelUp course, comprehensive documentation, and quick-start guides enable self-service adoption
- Repeatable process established that shifts infrastructure validation left in development cycle, reducing post-release firefighting
-
We will continue this work in a dedicated PREP team.
-
We have moved the enablement rollout out of the scope of this epic and into the PREP team scope. We continue to work on that enablement.
-
Thank you @pursultani for driving this project end to end and @niskhakova for your support
-
Highlights 🚀
- Build:
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- UBT-FIPS built with minimal issues, confirming that we should be able to produce UBT-FIPS builds.
- CI to automate UBT builds is underway and progressing well. We can build packages via CI now rather than locally, which will make collaboration more efficient.
- We are on-track according to current timelines.
-
🕐 total hours spent this week by all contributors: 58h
-
https://gitlab.com/groups/gitlab-org/build/-/epics/6+:
- TL;DR: We are on track with our deliverables and things are progressing according to plan.
- Migrations of gitlab-runner packages completed, thus bringing https://gitlab.com/groups/gitlab-org/build/-/epics/20 to a completion. We will do another final sweep before the cutover to ensure everything has been migrated over.
- Upstream PR by @myacksmith to Pulp Operator to add support for configuring specific log messages has been merged. We are awaiting a release to make use of them.
- Pulp UI MR has been merged, making it ready for production deployment.
- Dedicated work is yet to be completed, with team targeting to complete it by 19th January
-
🕐 total hours spent this week by all contributors: 33 hours
- BAU:
- DRI work, MR reviews/testing
-
🕐 total hours spent this week by all contributors: 10 hours
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Operate:
-
Cloud Native GitLab: Move from NGINX Ingress to... (gitlab-com/gl-infra/software-delivery/operate&5)
- Initial Gateway API support for GitLab Operator is work in progress.
- Support for OpenBao via Gateway API is in review.
- Support for configuration needed by Dedicated is in review
- :warning:FIPS build instructions for Envoy Proxy documented.
- This relates to hardened images. We should consider if we actually want to provide FIPS images for this third party component for free
- Breaking change exception still pending approval
-
🕐 total hours spent this week by all contributors: 15 hours
-
Omnibus Adjacent Kubernetes (OAK) - Operate Imp... (gitlab-com/gl-infra/software-delivery#22):
- We are starting to work on OAK and have a scoped milestone plan. In subsequent Grand Reviews, we'll start tracking individual milestones.
- OAK is going to enable Omnibus customers to operate specific GitLab components in Kubernetes, committing on our segmentation strategy
- The whole scope will be delivered in FY27Q1, with documented runbooks on how to operate GitLab Omnibus with OAK for many use cases
-
🕐 total hours spent this week by all contributors: 14 hours
-
Document Zero Downtime Upgrades Support with Gi... (gitlab-com/gl-infra/software-delivery&16 - closed)
- Updated Process based on existing tooling
- Ran through the process with Operator and Manually multiple times without issue
- We are getting continued requests from customers to deliver ZDU functionality in CNG, reinforcing the market demand for this work
-
🕐 total hours spent this week by all contributors: 16 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/epics/17+:
- Numerous updates and polish to the documentation draft
- Refined specifications across all RA targets
- Streamlined documentation throughout to be more consumable
- Added multiple new sections giving guidance for specific scaling scenarios and how to deploy
-
🕐 total hours spent this week by all contributors: 8 hours
-
Operate Team BAU - FY26Q4 (gitlab-com/gl-infra/software-delivery&29 - closed):
- Interviews for Delivery stage including developing new task for candidates applying again
- GET 3.8.3 released with several bug fixes
- GET 3.9.0 development work
- Working with PS on adding flexibility for AWS deployments with tags
- Dedicated backport fix for Geo edge case
- Community contribution around setting files permissions
- OpenBao MR draft, now paused awaiting further updates
- Reference Architecture pipeline maintenance
- Investigation into wider implications of upcoming Redis EOL
- Omnibus and Charts BAU work
- PREP BAU reviews
-
🕐 total hours spent this week by all contributors: 86 hours
-
Cloud Native GitLab: Move from NGINX Ingress to... (gitlab-com/gl-infra/software-delivery/operate&5)
- Release & Deploy:
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
- Common
ringctloperations can now be run in CI, thanks to @acafefebrissy.🎉 - Tracking has been added for deployments to Cells in auto-deploy codebases such as GitLab, Gitaly, and release/metadata. Thanks for the great work, @madelacruz!
- Auto-deploy manual promotion will trigger a Cells ring 1 deployment for the same package version
- Patch inversion can be done solely using
ringctlwithout any manual intervention. Thanks @anganga! -
🕐 total hours spent this week by all contributors: 66 hours
- Common
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1753+:
- A new type of deployment pipeline meant for deploying PDM patches was added to Tissue - https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21603.
- We are now switching Tissue to use patches for executing PDMs, instead of executing PDMs after every deployment - https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21605.
- We are in the process of adding validation for the new tenant model field - https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/21685.
🕐 total hours spent this week by all contributors: 14.5 hours
-
Delivery: Ruby 3.3 rollout to .com and self-man... (&1698 - closed):
- On 2026-01-08 we made an attempt to roll out Ruby 3.3, but the deployment failed with an issue not related to the new Ruby version. production#20969 (closed)
- Since we are near the monthly release date, we decide to reschedule it to after the monthly release and will roll out in 18.9.
🕐 total hours spent this week by all contributors: 12 hours
-
Releases and Deployments FY26Q4 KTLO and techni... (&1772 - closed):
- Release management efforts this week
🕐 total hours spent this week by all contributors: ?? hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
Time Spent 🕐
We are still iterating on how to capture BAU more accurately, as this is still underrepresented.
| Team | Projects | BAU, KTLO, RFH, Release Management, Interrupts |
|---|---|---|
| Build (includes Deploy support for Pulp) | 100h | 5h |
| Operate | 13h | - |
| Release & Deploy | 36h | 23h |
Review 2026-01-08
Recording 📹
Blockers ⚠️
- groupoperate is facing issues supporting OpenBao in GET. The current state of the component will only allow experimental GET support which isn't acceptable for Dedicated roll-out. We will work on a plan with grouppipeline security. In Q1, only beta customers are planned, which should give us enough time to deliver.
To Be Closed 🎉
-
Readiness Rollout: Limited Adoption (gitlab-org/distribution#127 - closed):
-
GitLab's existing Production Readiness Review only covered GitLab.com, leaving no systematic validation for Self-Managed and Dedicated platforms. This resulted in features reaching GA with critical issues blocking customer adoption.
PREP fills this gap by providing a self-service framework for product teams to validate feature readiness across the entire GitLab platform ecosystem before release.
-
We have finished our limited adoption phase with measurable progress:
- 3+ infrastructure teams engaged beyond initial sectiongitlab delivery scope, establishing PREP as cross-functional infrastructure practice
- 3 production features assessments are underway or completed, identifying critical infrastructure gaps before production
- PREP Assistant automation reduced assessment time by 60-70%, exceeding our expectations (40% manual work reduction target)
- LevelUp course, comprehensive documentation, and quick-start guides enable self-service adoption
- Repeatable process established that shifts infrastructure validation left in development cycle, reducing post-release firefighting
-
We will continue this work in a dedicated PREP team.
-
Thank you @pursultani for driving this project end to end and @niskhakova for your support
-
Highlights 🚀
- Build:
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Work concentrated on bugfixes in existing codebase and basic CI implementation.
- We are on-track according to current timelines.
-
🕐 total hours spent this week by all contributors: 40h
-
https://gitlab.com/groups/gitlab-org/build/-/epics/6+:
- TL;DR: We are on track with our deliverables and things are progressing according to plan.
- All of GitLab CE and EE is now being served from Pulp with an average of ~170k downloads/day and no customer issue so far.
- Still working on runner packages
-
🕐 total hours spent this week by all contributors: 60 hours
- BAU:
- People coming back from vacation, thus the low number this week
-
🕐 total hours spent this week by all contributors: 5 hours
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Operate:
-
Cloud Native GitLab: Move from NGINX Ingress to... (gitlab-com/gl-infra/software-delivery/operate&5)
- Initial Gateway API and Envoy Gateway support merged into GitLab chart.
- Support for configuration needed by Dedicated is in review.
- Proposal for customer facing deprecation and timeline is ready in review.
-
🕐 total hours spent this week by all contributors: 2 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/epics/17+:
- First documentation draft complete with MR raised!
-
🕐 total hours spent this week by all contributors: 6 hours
-
Cloud Native GitLab: Move from NGINX Ingress to... (gitlab-com/gl-infra/software-delivery/operate&5)
- Release & Deploy:
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
- All the demos for onboarding team members to Cells release management are complete. Thanks to all participants for sharing this work!
- This table has a list with timestamped YouTube links: https://gitlab.com/groups/gitlab-com/gl-infra/-/work_items/1707#demos
- Auto-deploy implementation for Cells in line with Handbook ADR is progressing (https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/21678)
- CI pipelines for common operations are in review (https://gitlab.com/gitlab-com/gl-infra/delivery/-/work_items/21519)
- This should let all team members use CI for common operations rather than having to install tools locally
-
🕐 total hours spent this week by all contributors: 29 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1753+:
- Instrumentor has been updated to use the
post_deployment_migration_versiontenant model attribute. We are now in the process of updating Tissue to run a minimal CI pipeline whenpost_deployment_migration_versionis modified. 🕐 total hours spent this week by all contributors: 7 hours
- Instrumentor has been updated to use the
-
Releases and Deployments FY26Q4 KTLO and techni... (&1772 - closed):
- Release management efforts this week
🕐 total hours spent this week by all contributors: 23 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
Time Spent 🕐
We are still iterating on how to capture BAU more accurately, as this is still underrepresented.
| Team | Projects | BAU, KTLO, RFH, Release Management, Interrupts |
|---|---|---|
| Build (includes Deploy support for Pulp) | 100h | 5h |
| Operate | 13h | - |
| Release & Deploy | 36h | 23h |
Review 2025-12-18
Recording 📹
Blockers ⚠️
To Be Closed 🎉
Highlights 🚀
- Build:
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Slow week with team members taking time off etc.
- Completed UBT builds for all 3rd party Golang components (8 total).
- 2 more UBT integration MRs merged into Omnibus (13 merged total).
- We're on-track within recently revised schedule, with final components to be onboarded in Q2.
-
🕐 total hours spent this week by all contributors: 30h
-
https://gitlab.com/groups/gitlab-org/build/-/epics/6+:
- TL;DR: We are on track with our deliverables and things are progressing according to plan.
- All of GitLab CE is now being served from Pulp.
- We successfully tested serving GitLab EE from Pulp for a 12 hour window. Details are in the CR.
- Migration of Dedicated tenants and Cell nodes to fetch packages from Pulp is progressing well, and testing is underway.
- Migration of runner packages has started. Debian family (Debian, Ubuntu, Raspbian, and Linux Mint) has completed, and RedHat family is currently being done.
- @balasankarc got invited as a collaborator with write permissions to upstream Pulp Operator and Helm Chart repositories. This should hopefully result in faster turnarounds for our PRs.
-
🕐 total hours spent this week by all contributors: 164 hours
- BAU:
- Completed work was mostly BAU MRs, dependency updates and build DRI work
-
🕐 total hours spent this week by all contributors: 107 hours
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Operate:
-
Readiness Rollout: Limited Adoption (gitlab-org/distribution&127 - closed):
- LevelUp course first version completed and available for review. Feedback welcome in course comments. The exit criteria for enablement content creation met.
- Impact measurement framework initiated with "Delivery Time" metric proposed to measure PREP effect on readiness assessment and cross-platform availability
-
🕐 total hours spent this week by all contributors: 18 hours
-
Cloud Native GitLab: Move from NGINX Ingress to... (gitlab-com/gl-infra/software-delivery/operate&5)
- Initial Gateway API and Envoy Gateway support merged into GitLab chart.
- Support for configuration needed by Dedicated is in review.
- Breaking change request created, approvals pending
-
🕐 total hours spent this week by all contributors: 15 hours
-
Document Zero Downtime Upgrades Support with Gi... (gitlab-com/gl-infra/software-delivery#16 - closed):
- Worked through manual process to confirm it fundamentally works\
- Reached out to Release & Deploy team to go over production process
-
🕐 total hours spent this week by all contributors: 10 hours
-
Operate Team BAU - FY26Q4 (gitlab-com/gl-infra/software-delivery&29 - closed):
- OpenBao support - Validated Geo configuration, waiting for upstream bug to be fixed by Pipeline Security team.
- Started writing up Charts migration documentation for customers with Postgres, Redis, etc...
-
🕐 total hours spent this week by all contributors: 50 hours
-
Readiness Rollout: Limited Adoption (gitlab-org/distribution&127 - closed):
- Release & Deploy:
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
- Instrumentor upgraded to the latest version on all cells unblocking usage of Pulp on Cells
🎉 - Alert which triggers upon failure of patch deployment implemented and working as expected
- Making progress on recording deployment of auto-deploy packages to Cells in Rails, Gitaly, etc.
-
🕐 total hours spent this week by all contributors: 58 hours
- Instrumentor upgraded to the latest version on all cells unblocking usage of Pulp on Cells
-
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/epics/20+:
- Last week we had a wider alignment effort with Build and Runway teams. We are now working on incorporating their feedback about the tagging strategy.
🕐 total hours spent this week by all contributors: 26 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1753+:
- Instrumentor will now use the
post_deployment_migration_versiontenant model field (if it is present) to decide which PDMs should be executed. 🕐 total hours spent this week by all contributors: 6 hours
- Instrumentor will now use the
-
Releases and Deployments FY26Q4 KTLO and techni... (&1772 - closed):
- Release management efforts this week (this week was a minor release week)
- These efforts are just GitLab RMs, not including contractors
- Also includes work on Ruby 3.3 rollout
🕐 total hours spent this week by all contributors: 30 hours
- Release management efforts this week (this week was a minor release week)
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
Time Spent 🕐
We are still iterating on how to capture BAU more accurately, as this is still underrepresented.
| Team | Projects | BAU, KTLO, RFH, Release Management, Interrupts |
|---|---|---|
| Build (includes Deploy support for Pulp) | 194h | 75h |
| Operate | 43h | 50h |
| Release & Deploy | 90h | 30h |
Review 2025-12-11
Recording 📹
Blockers ⚠️
- Pulp: Runner packages are currently getting fixed at the Packagecloud side. This needs to be completed before we can migrate the packages over to Pulp.
To Be Closed 🎉
-
Self managed: Support Rollout of Container Regi... (gitlab-org&17005 - closed):
- The container registry metadata database lacked comprehensive support for self-managed deployments, with missing configuration support and a lack of automated provisioning, unclear GA readiness and an original rollout strategy that would have introduced a breaking change
- We identified all implementation gaps across Omnibus, Charts, Backup, HA GET, Geo and Dedicated
- We implemented automated registry database provisioning for Omnibus and Charts with GitLab-managed PostgreSQL installations, added complete database migration support with pre/post-deployment migration handling and integrated PgBouncer connection pooling support. We also informed rollout considerations to avoid breaking changes
- With this, we now provide proper self-managed support for this long awaited feature by devopspackage and deliver an item interlocked for Q4 in time.
- @Alexand drove this end-to-end across groupgeo, groupoperate and groupcontainer registry with support by @lucus.li. Thank you very much!
Highlights 🚀
- Build:
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- 5 more Omnibus/UBT integration MRs have been merged
- Started adding Golang-based components to Omnibus
-
🕐 total hours spent this week by all contributors: 80h
-
https://gitlab.com/groups/gitlab-org/build/-/epics/6+:
- TL;DR: We are on track with our deliverables and things are progressing according to plan.
- Latest patch release had packages uploaded to both Pulp and Packagecloud. There were minor hiccups which are already being handled, but overall this was a confidence booster.
- We have routed customer traffic to certain repositories to Pulp to as first iterations, and they have been successful
- GitLab CE - Debian Trixie
- GitLab CE - Ubuntu Focal
- GitLab CE - Amazon Linux 2
- We'll go on to serve all of GitLab CE from Pulp next
-
🕐 total hours spent this week by all contributors: 86 hours
- BAU:
- progress towards [Spike] Segregate buildx build of rails-ee and rails-ce (#2266), DRI Triage rotation, generally MR testing and reviews
-
🕐 total hours spent this week by all contributors: 75 hours
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Operate:
-
Readiness Rollout: Limited Adoption (gitlab-org/distribution&127 - closed):
- LevelUp course development progressing. 6 of 9 modules content ready, on track for mid-January release before All Hands (28th Jan), course preview expected in coming days
- New README with PREP Assistant agent introduction drafted, completion imminent
-
🕐 total hours spent this week by all contributors: 18 hours
-
Cloud Native GitLab: Move from NGINX Ingress to... (gitlab-com/gl-infra/software-delivery/operate&5)
- The decision to integrate Envoy Gateway has been finalized, documented. This decision involved lots of collaboration involving Dedicated, .com infrastructure, Product, and Operate
. - The remaining work for milestone 18.7 will focus on shipping initial Gateway API and Envoy Gateway support with the GitLab helm chart.
-
🕐 total hours spent this week by all contributors: 25 hours
- The decision to integrate Envoy Gateway has been finalized, documented. This decision involved lots of collaboration involving Dedicated, .com infrastructure, Product, and Operate
-
Document Zero Downtime Upgrades Support with Gi... (gitlab-com/gl-infra/software-delivery#16 - closed):
- Continuing analysis of .com and Dedicated ZDU processes
-
🕐 total hours spent this week by all contributors: 8 hours
-
Operate Team BAU - FY26Q4 (gitlab-com/gl-infra/software-delivery&29 - closed):
-
🕐 total hours spent this week by all contributors: 67 hours
-
-
Readiness Rollout: Limited Adoption (gitlab-org/distribution&127 - closed):
- Release & Deploy:
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
- New command to reverse the changes made by a patch:
ringctl patch invertThis has already been used to deal with a failing patch in Tissue - Tissue UI improved to display useful information in pop-overs
-
🕐 total hours spent this week by all contributors: 50,5 hours
- New command to reverse the changes made by a patch:
-
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/epics/20+:
- Further work and reviews on design doc
🕐 total hours spent this week by all contributors: 10 hours
-
Delivery: Ruby 3.3 rollout to .com and self-man... (&1698 - closed):
- We move the plan to roll out Ruby 3.3 to Production on 2026-01-08, milestone 18.8. In 18.7, we roll out Rails 7.2, so it is good to keep separate so we can more easily troubleshoot any issues.
🕐 total hours spent this week by all contributors: 2 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1753+:
- This will allow us to execute a subset of pending PDMs by setting the
post_deployment_migration_versionfield in the tenant model. 🕐 total hours spent this week by all contributors: 11 hours
- This will allow us to execute a subset of pending PDMs by setting the
-
Releases and Deployments FY26Q4 KTLO and techni... (&1772 - closed):
- Release management efforts this week:
🕐 total hours spent this week by all contributors: 48 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
Time Spent 🕐
We are still iterating on how to capture BAU more accurately, as this is still underrepresented.
| Team | Projects | BAU, KTLO, RFH, Release Management, Interrupts |
|---|---|---|
| Build (includes Deploy support for Pulp) | 166h | 75h |
| Operate | 52h | 67h |
| Release & Deploy | 78,5h | 48h |
Review 2025-12-04
Recording 📹
Blockers ⚠️
- Migration of Runner packages from PC to Pulp is blocked due to presence of duplicate packages in Packagecloud - https://gitlab.com/gitlab-org/build/team-tasks/-/issues/80#note_2922970371. We are collaborating with the runner team to resolve it
To Be Closed 🎉
Highlights 🚀
- Build:
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- 2 more Omnibus/UBT integration MRs have been merged
- All C/C++ components (47% of omnibus components) have been validated to match Omnibus build configuration
-
🕐 total hours spent this week by all contributors: 70h
-
https://gitlab.com/groups/gitlab-org/build/-/epics/6+:
- TL;DR: We are on track with our deliverables and things are progressing according to plan.
- Package syncing has been completed for Omnibus GitLab repos
- All omnibus-gitlab pipelines now upload packages to pre-release repo in Pulp
-
🕐 total hours spent this week by all contributors: 133 hours
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Operate:
-
Self managed: Support Rollout of Container Regi... (gitlab-org&17005 - closed):
- We are still finishing up the final MR (just this one) on our side
-
🕐 total hours spent this week by all contributors: 10 hours
-
Readiness Rollout: Limited Adoption (gitlab-org/distribution&127 - closed):
- First success story (OpenBao) completed - one more (SBOM candidate) needed to meet exit criteria
- LevelUp course for PREP in development with outline available, advancing enablement content creation objective
-
⚠️ Feature teams are seeking groupoperate GA approval, but we lack authority to provide it and cannot effectively redirect to responsible teams without degrading experience (this is becoming apparent in Zoekt GA). We will need to clarify internally what GA approval means and who will provide it. -
🕐 total hours spent this week by all contributors: 16 hours
-
Cloud Native GitLab: Move from NGINX Ingress to... (gitlab-com/gl-infra/software-delivery/operate&5)
- We agreed on a replacement for NGINX Ingress within Cloud Native GitLab.
- We will move towards Gateway API and Envoy Gateway, which aligns with the recommendation of the upstream Kubernetes group, our plans for the new Auth stack, and the effort to rollout Envoy on .com.
-
🕐 total hours spent this week by all contributors: 30 hours
-
Document Zero Downtime Upgrades Support with Gi... (gitlab-com/gl-infra/software-delivery#16 - closed):
- ZDU are currently only officially supported and documented in Omnibus HA deployments. This lack of support actively leads customers to decide to deploy Omnibus HA architectures, against our CNG strategy. In this epic, we will create parity between deployment options as a first step to enable customers choosing the correct deployment.
- Started working on this epic and began pulling information to try and understand the process currently being used by Dedicated.
- Worked through a blueprint and corresponding epic that outlines the process
-
🕐 total hours spent this week by all contributors: 20 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/work_items/17+:
- Reference architectures have to be fundamentally reworked to remove user counts as the main sizing signal and align our recommendations with Dedicated operational experience. These architectures will place CNG as the recommended option, further providing guidance to customers on what they should deploy
- This Epic is scoped to release the Cloud Native First Reference Architectures in Beta to introduce them and gather feedback.
-
🕐 total hours spent this week by all contributors: 10 hours
-
Self managed: Support Rollout of Container Regi... (gitlab-org&17005 - closed):
- Release & Deploy:
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
- Documentation that will help RMs deal with failures in patch processing has been added - https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21597
- Work on controlling PDM execution via the tenant model has been reviewed and we've agreed on an approach - https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/instrumentor/-/merge_requests/6601.\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
- Sending notifications to
#announcementschannel in Slack when a patch fails is in progress - https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21304 - Working on recording deployment of auto-deploy packages to rings, so that the deployment widget in MRs shows the rings that it has been deployed to, in addition to the current staging, production environments - https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21654
- Instrumentor version upgrades are being progressively merged as we slowly catch up to the latest Instrumentor version.
-
🕐 total hours spent this week by all contributors: 75 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/epics/20+:
- This week we worked on alignment and refining the design-doc.
🕐 total hours spent this week by all contributors: 24 hours
-
Delivery: Ruby 3.3 rollout to .com and self-man... (&1698 - closed):
- We successfully performed a test rollout to gstg-cny, gprd-cny and gstg-ref production#20918 (closed)
🕐 total hours spent this week by all contributors: 7 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
Review 2025-11-27
Recording 📹
Blockers ⚠️
To Be Closed 🎉
-
Container Registry Post-Deployment Migrations A... (gitlab-org&17933 - closed):
- Container Registry post deployment migrations used to be manually executed, requiring an SRE from the Release&Deploy team to execute commands in a registry pod. This was time consuming, potentially error prone and delayed Container Registry work items that were dependent on the PDMs.
- A helm release was added to https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com that executes Container Registry PDMs using a K8s job. The procedure to execute PDMs now uses a CI pipeline and is documented in runbooks. Executing Container Registry PDMs is now an automated operation that can safely be done by any SRE.
Highlights 🚀
- Build:
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Continued Omnibus integration of C/C++ components, representing around 47% of Omnibus components (~100 functional components total)
- preliminary tests with 20 components shows potential of up to 20% reduction in compute time alone.
- CI automation of UBT component builds is underway.
-
🕐 total hours spent this week by all contributors: 84h
-
https://gitlab.com/groups/gitlab-org/build/-/epics/6+:
- TL;DR: We are on track with our deliverables and things are progressing according to plan. We've added a Gantt chart for increased transparency
- We are now pushing nightly and auto-deploy packages to Pulp in addition to PackageCloud.
-
URL rewriting and downloading has been enabled for nightly packages. That means,
dev.gitlab.orgnow runs a Package installed from Pulp, still accessed throughhttps://packages.gitlab.com - Pulp upstream has been cooperating a lot, and have fixed the bug with package syncing, thus unblocking us. Package syncing has started.
gitlab-cehas been synced over from PackageCloud to Pulp. https://gitlab.com/gitlab-org/build/team-tasks/-/work_items/81+. - Our PR to add HPA support has been merged upstream, and PR to add custom labels is currently under review.
-
🕐 total hours spent this week by all contributors: 139 hours
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Operate:
-
Self managed: Support Rollout of Container Regi... (gitlab-org&17005 - closed):
- We are still finishing up the final MRs on our side
- Just one more week probably, maybe. For real this time.
-
🕐 total hours spent this week by all contributors: 20 hours
-
Readiness Rollout: Limited Adoption (gitlab-org/distribution&127 - closed):
- Shifting toward presentation materials, demos, and success story development to capture rollout learnings and impact (issues created for tracking)
- Two infrastructure team contribution MRs in review process
-
⚠️ Engagement on active assessments is limited this period -
🕐 total hours spent this week by all contributors: 20 hours
-
Cloud Native GitLab: Move from NGINX Ingress to... (gitlab-com/gl-infra/software-delivery/operate&5)
- Cloud Native GitLab uses NGINX Ingress to expose GitLab via HTTP(S) and SSH (for Git). NGINX Ingress was announced to be retired in March 2026. We'll have to actively manage this high impact breaking change.
🕐 Work just kicked off, no time tracking yet
-
Self managed: Support Rollout of Container Regi... (gitlab-org&17005 - closed):
- Release & Deploy:
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
- We did a couple of demos to begin introducing the rest of the Release&Deploy team to Cells - https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1707#demos
- Work on controlling PDM execution via the tenant model is progressing - https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21601#note_2917384503. This will give us better control over when PDMs are executed which gives us better control over rollback windows.
- Documentation that will help RMs deal with failures in patch processing is being added - https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21597
- Additional alerting is being added to alert when patch processing stops - https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21219\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
- A ringctl command is being added to allow us to undo a partially failed change to a ring - https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21502
- The Instrumentor versions in cells have been updated (they were very out-of-date due to misconfigured Renovate)
-
🕐 total hours spent this week by all contributors: 64 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/-/epics/20+:
- This work will enable the Runway to Self-Managed initiative, expanding the release team offering to further components while shifting work and ownershift left to component teams. We are currently designing and scoping the work
- An early priority will be documenting manual procedures to enable security patches without disclosing vulnerabilities, contributing to an important security priority
- Design doc is in review, we got a DRI from security.
- We have been [aligning with DAP and Runway](gitlab-org&19383%20(comment%202909613290) ) to expedite their security posture improvements in lights of the upcoming GA
🕐 total hours spent this week by all contributors: 25 hours
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
Review 2025-11-20
Recording 📹
Blockers ⚠️
To Be Closed 🎉
-
https://gitlab.com/groups/gitlab-org/distribution/-/epics/111+:
- We delivered containers for OpenBao, enabling their beta in Q4. This is the groupbuild work item that should have been closed last week but didn't surface due to automation issues.
Highlights 🚀
- Build:
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Continued work is ongoing to add more components to UBT. Planning has started to add CNG integration, enabling more minimal images
-
https://gitlab.com/groups/gitlab-org/build/-/epics/6+:
- TL;DR: We are on track with our deliverables and things are progressing according to plan. PRR is moving along and we had sign off from InfraSec. Just waiting for final confirmation from ProdEng.
- Ops infrastructure deployed and closed - Production Pulp infrastructure is now fully operational
- Pre-prod CloudFlare validation completed
- Rate limiting implemented for Pulp instances to prevent abuse
- Package upload validation successful - Confirmed PackageCloud uploads still work after enabling URL rewrite for Pulp downloads
- Performance testing completed - Upload times averaging 8-11 minutes per package with stable infrastructure
- URL rewrite solution developed using CloudFlare Workers KV to maintain client sessions for Debian packages
- Migration planning finalized - Created detailed breakdown of package types with separate issues for enabling uploads/downloads (Decision Log)
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Operate:
-
Self managed: Support Rollout of Container Regi... (gitlab-org&17005 - closed):
- We are still finishing up the final MRs on our side
- Just one more week probably, maybe.
-
Readiness Rollout: Limited Adoption (gitlab-org/distribution&127 - closed):
- PREP Agent exceeded 40% manual work reduction target across SBOM, OpenBao, and Zoekt readiness migration. Automation exit criteria achieved
- PREP assessment identified multiple critical gaps and even a bug in Dependency Scanning/SBOM, demonstrating value in production readiness validation
- Domain coverage expanding with contributions from grouprelease-and-deploy and groupenvironment automation , plus the ongoing groupgeo work.
- Still missing contributions from groupObservability, groupgeo
-
Self managed: Support Rollout of Container Regi... (gitlab-org&17005 - closed):
- Release & Deploy:
-
Container Registry Post-Deployment Migrations A... (gitlab-org&17933 - closed):
- We successfully executed Container Registry PDMs using the automation for the first time - #21189 (comment 2899812882)
- We will introduce the team to the new procedure next
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
- Tissue has been updated to skip PDM execution when running Instrumentor's Configure stage - https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21604
- Working on updating Instrumentor to use the new tenant model attribute to control PDM execution - https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21601
- Documenting the workflow to determine if a patch has failed and why - https://gitlab.com/gitlab-com/gl-infra/delivery/-/issues/21597
-
Container Registry Post-Deployment Migrations A... (gitlab-org&17933 - closed):
Review 2025-11-13
Recording 📹
Blockers ⚠️
Not strictly a blocker (more like attention needed):
- DevEx Feature Readiness is no longer involved with PREP or AI-powered automation for readiness. This requires revised automation strategy for rollout continuation and a discussion on the strategic direction in this area
To Be Closed 🎉
-
https://gitlab.com/groups/gitlab-org/distribution/-/epics/112+:
- We delivered charts and containers for OpenBao, enabling their beta in Q4. GET support will be provided in the next GET release in January, enabling Dedicated and GA in FY27Q1. Delivery contributed significant investment to this widespread customer request with potential to improve netARR.
Highlights 🚀
- Build:
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- 46 out of 47 C/C++ components (from Omnibus list of components) are added to UBT builds. 6 MRs covering Omnibus integration of initial set of 10 components are in review. Initial phase of CNG integration has started.
-
https://gitlab.com/groups/gitlab-org/build/-/epics/6+:
- Infra/Platform ready in preprod
✅ - Preprod fully validated with cloudflare
✅ - Ops infra deployed
✅ - PRR still in progress but moving forward
- Infra/Platform ready in preprod
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Operate:
-
Self managed: Support Rollout of Container Regi... (gitlab-org&17005 - closed):
- We are finishing up the final MRs on our side
- There are still further issues regarding backup/restore. Package is coordinating with groupgeo, who owns that feature and should help deliver on that feature.
-
Readiness Rollout: Limited Adoption (gitlab-org/distribution&127 - closed):
- Feature assessments progressing while organizational changes require strategic planning for continued rollout success.
- PREP assessment for Dependency Scanning using SBOM advancing well. Review in progress and on track for completion
- PREP Agent continues demonstrating value in reducing manual assessment work. In the most recent application, groupcomposition analysis has started using it.
-
Self managed: Support Rollout of Container Regi... (gitlab-org&17005 - closed):
- Release & Deploy:
-
Container Registry Post-Deployment Migrations A... (gitlab-org&17933 - closed):
- After feedback received from the grand review, we will not be giving the Container Registry team access to execute PDMs. They have been given access to view the CI pipeline only. An issue has been opened to finalize the prerequisites for giving the Container Registry team access to execute PDMs (#21644). Once the prerequisites are met, the Container Registry team can be given full access in a future iteration.
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
- Progress is being made on improving the procedures and tooling that RMs will have to use for Cells. Progress has also been made on enabling rollbacks on Cells.
-
Container Registry Post-Deployment Migrations A... (gitlab-org&17933 - closed):
Review 2025-11-06
Recording 📹
Blockers ⚠️
- Pulp PRR still requires input from ProdEng and InfraSec. We can still deploy the Ops instance (production) without the full PRR because it is not yet production use, but we need to get attention to the PRR
To Be Closed 🎉
-
Containers: Implement SBOM generation (gitlab-org/distribution&93 - closed):
- Customers have long been asking for SBOMs for our containers as part of their security and compliance requirements.
- With this epic, we are now delivering SBOMs for containers built in our pipelines, including attestation of these SBOMs to the containers. This improves transparency for our customers and delivers on Secure requirements.
-
https://gitlab.com/groups/gitlab-com/gl-infra/software-delivery/framework/-/epics/35+:
- Sizing of reference architectures has been a challenge for a long time.
- Customers and the field where misled by user counts in reference architecture sizes resulting in inaccurate predictions.
- We are now providing comprehensive documentation to support customers and the field in selecting the right reference architecture, as well as giving some valuable direction on scaling.
- We hope this will improve Dedicated margins by landing customers on right-sized architectures and promote efficiency in the field organisation.
- We will explore further automation and inclusion of existing scripting tools in the guidance going forward
-
Schedule deployment migrations as a job on k8s (&1556 - closed):
- We have now removed the last VMs grouprelease-and-deploy has been maintaining, migrating them to k8s jobs.
- This change will foster efficiency in grouprelease-and-deploy and lower the time required to run migrations, which took a long time on these VMs
Highlights 🚀
- Build
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Merged gitlab-org/omnibus!52 (merged) which unlocked our ability to add UBT components "at-will" to Omnibus, resulting in multiple MRs filed targeting initial set of 10 components, with another 20+ ready to go next. Work for enabling multi-OS QA pipeline for Omnibus are underway. We are on-track.
- We'll be able to show improved metrics in the next few weeks and will start tracking data going forward.
-
https://gitlab.com/groups/gitlab-org/build/-/epics/6+:
- Setup pulp metrics catalog for preprod (gitlab-com/runbooks!9571 - merged) by @jennykim-gitlab has been merged. The remaining task is to configure Grafana dashboards, which will enable observability of Pulp
- Multiple upstream PRs merged and new versions released, unblocking parts of our work.
-
https://gitlab.com/groups/gitlab-org/distribution/build-architecture/-/epics/4+:
- Operate:
-
https://gitlab.com/groups/gitlab-org/distribution/-/epics/112+:
-
🎉 Setup Openbao Rails authentication token (gitlab-org/charts/gitlab!4549 - merged) merged in Rails, supporting gitlab-org/cloud-native/charts/openbao#21+ -
▶️ Setup Openbao Rails authentication token (gitlab-org/charts/gitlab!4549 - merged) is under review, and should merge by end of week🏃 - After that,
👏 🎉
-
-
Self managed: Support Rollout of Container Regi... (gitlab-org&17005 - closed):
- Further complications have emerged with the chosen solution for the registry migration. We have a way forward but will take a few more weeks to avoid breaking changes for some customers. The work is mainly on the Container Registry side and we are supporting.
-
Readiness Rollout: Limited Adoption (gitlab-org/distribution&127 - closed):
- PREP Agent gaining traction beyond initial scope with positive user feedback demonstrating measurable reduction in manual work.
- Productive engagement with groupgeo advancing domain coverage expansion with requirements capture progressing well
-
https://gitlab.com/groups/gitlab-org/distribution/-/epics/112+:
- Release & Deploy:
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+:
- We've iterated on improvements for deployment pipeline stability and observability
-
https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1654+: