**TL;DR** The current multivariate entry-point strategy that GitLab has across self-managed, .com and Dedicated creates significant challenges in the maintenance and release lifecycles of these offerings across several dimensions: Communications, Operations & Support. GitLab's commitment to public by default further complicates these things. Currently our releases and maintenance policies do not support the various entry-points adequately and we must adapt them to better fit our SaaS first principle. Finally, as GitLab grows Larger our toolset and processes must be improved to satisfy the scale in demand and velocity, today's processes will soon fall short. ## Context GitLab currently provides 3 different entry-points for users: Self-Managed, GitLab.com and Dedicated. The way that changes are released for each entry-point are slightly different. There is an [overview of our deployments and releases](https://about.gitlab.com/handbook/engineering/deployments-and-releases/) in the handbook (N.B The handbook page only covers self managed and GitLab.com, Dedicated is not covered). Primarily, the [Delivery Group](https://about.gitlab.com/handbook/engineering/infrastructure/team/delivery/) are responsible for making sure that the latest changes are released across these GitLab entry-points, but there is also strong collaboration across [Distribution](https://about.gitlab.com/handbook/engineering/development/enablement/systems/distribution/), [Quality](https://about.gitlab.com/handbook/engineering/quality/), [Infrastructure](https://about.gitlab.com/handbook/engineering/infrastructure/platforms/) & [Development](https://about.gitlab.com/handbook/engineering/development/). <!-- Talk about the release processes and how we know they may not fit all entry-points here ++ comms with customers and how that's not totally ideal --> Outside of the standard, monthly release processes, there are 3 sub level release processes that we follow (can be found in the [maintenance policy](https://docs.gitlab.com/ee/policy/maintenance.html)): 1. Patch Release 1. Security Release 1. Backports Currently we provide patches for the current version only & security releases for the current version and two previous versions. Backports are offered on an exception basis and not guaranteed. We have no firm commitment on an long term support policy as of the latest edit. Changes are deployed to GitLab.com several times a day as part of our Continuous Delivery Process as defined [in the handbook](https://about.gitlab.com/handbook/engineering/deployments-and-releases/). Self-Managed releases on a different schedule and the 3 sub level release processes defined above are mainly relevant for Self Managed GitLab as they are specific, semantically versioned releases. Communicating change has mainly been done on the same cadence on the minor release through methods like the [release posts](https://about.gitlab.com/handbook/marketing/blog/release-posts/), [deprecations page](https://docs.gitlab.com/ee/update/deprecations.html), [blogposts (example)](https://about.gitlab.com/blog/2023/02/28/oidc/), in-app messaging and occasionally direct user communications. We know that this may no longer be sufficient, particularly in relation to the constant improvements made to GitLab.com and our overall SaaS offering, and we've been [working on improving this](https://about.gitlab.com/blog/2023/03/20/enhanced-deprecations-page/). Lastly, an important component of GitLab's identity and principles is [public by default](https://handbook.gitlab.com/handbook/values/#public-by-default) and this is something that has to be supported by our release processes and maintenance policies. As part of regular lifecycle of a software product in 2023, there are scenarios like security fixes/releases where we have to make changes to GitLab whilst temporarily maintaining the confidentiality of those changes to protect ourselves, our partners and our users. This creates high technical complexity and specific pain points that are unique to companies like GitLab that operate in a public first manner. ## Problems <!-- 1. Under the current processes, a security release takes approximately 36 active hours across a minimum of 5 working days to process and deliver 1. More than one security release per month is untenable for Delivery on a regular basis 2. Without any issues or deviations from the happy path, it is likely that the shortest amount of time to process a security release will be 3 days (due to people and process concerns) 3. Security releases are constrained by master, so broken master and other disruptive events have serious consequences for processes downstream, like the security release 4. Due to the tighter SLAs for remediation in order to support FedRAMP, one security release a month is no longer frequent enough to ensure 100% of security releases are delivered within SLA 2. Due to the confidential nature of security releases and the potential for embargoed fixes, Delivery must maintain a private security mirror in order to prevent leaks before the fix is publicly available 2. There is room for human error in this process, which is hard to fully remove 3. Divergence from the master branch of the canonical repo can cause painful merge conflicts 4. Both repositories must be synced back together once the security fixes are released 3. The maintenance policy currently in place does not scale well for the Delivery team and creates invisible work that is hard to support. It is also not consistent with the security release policy, which creates additional confusion for gitlab team members and users around expectations for levels of support 4. The way that we communicate changes and particularly [breaking changes](https://about.gitlab.com/handbook/product/gitlab-the-product/#breaking-changes) has historically been targeted at customers that upgrade on a regular cadence, to a semantically versioned flavour of GitLab (e.g. 15.11.1). Deprecation and removal notices that are scheduled for 16.0 are relatively easy to digest understand and plan for as part of moving from version to version. However, this is not congruent with our continuous delivery process that we use to release changes to gitlab.com multiple times a day. Changes related to the next version of GitLab can flow into gitlab.com at any time after the release of the current version. To use 2023-04-22 and the 15.11 release as an example, changes destined for the 16.0 release, including breaking changes, started to be deployed into production from 2023-04-24. Our [Deprecations Page](https://docs.gitlab.com/ee/update/deprecations.html) does not currently make this distinction. 3. Tracking when & if a commit/feature that has been merged into the master branch of the canonical repo is deployed to production is difficult 4. --> ### GitLab.com - Multi-tenant SaaS Several deployments a day; continuous change. 1. There is no release process for GitLab.com 2. Users & team members don't understand what and when changes arrive. Worst offender are breaking changes 1. Development Teams struggle to release experiments in a repeatable way and label as such 1. Development Teams struggle to manage the life cycle of features. 1. Marketing struggles to highlight changes on GitLab.com ### Self-managed Packaged, versioned releases (patch, security, minor/major) 1. Users have more control over changes because of semantic versioning but also struggle with managing change 1. Development Teams struggle to manage the life cycle of features 1. GitLab team members don't understand how to engage with release processes or maintenance policies 1. Major releases are disruptive 2. Security releases are difficult and require tight coordination due to our public first nature 3. Out of Policy Backports are 'exception only' but are regularly needed and add to toil 4. Several customers desire a Long Term Support option, but we have no offering currently ### Dedicated Similar problem-set to Self Managed, with the caveat that they run N-1 package and a nuance that we are seeking to support highly regulated environments 1. FedRAMP has increased the demand for a more scalable security release process as the current process is at it's limits to meet the new demand ## User Feedback and Consensus This section is intended to provide external user context with a high level summary of the main insights we have gathered from customers 1. Users are not able to easily find more detailed information on breaking changes and who to contact to push back/escalate 1. Breaking changes are not communicated well to gitlab.com users - there is no indicator of exactly when a change is deployed 1. Customers look at many different SSoTs to try and establish what is currently live in gitlab.com 1. [Removals Page](https://docs.gitlab.com/ee/update/removals.html) 1. [Deprecations Page](https://docs.gitlab.com/ee/update/deprecations.html) 1. The [coordination of timing](gitlab-org/gitlab#352957 (comment 1385507748)) between updating SSoTs and releasing features can be critical for gitlab.com customers ## Core Questions <!-- ## Known issues / Risks --> 1. How Many security releases are required in any given month to remediate vulnerabilities? 2. What do users need in order to be able to minimise the impact of breaking changes on gitlab.com? 1. Is our Major, Minor and Patch release strategy still the right one for GitLab in 2023? 1. What is the right combination of "in cycle" patch releases? 1. How many releases is too many for Self-Managed Customers 1. What is the right communication and notification framework for SaaS users? 1. How can we make releases more visible to internal teams (So that they can own their comms and coordination)? ### Relevant Links - Link to [Dovetail project on breaking changes](https://gitlab.dovetailapp.com/projects/1O1Tgtbh23Y4bxc2iCC2Xt/v/7fiGrRvWVocLXBCXh7rYFF) ### Potential Sub-epics / issues * Open Releases and maintenance top level epic * 16.0 Release retro/review * Investigate LTS Opportunity & connection with FedRAMP - #3752 * https://docs.google.com/document/d/12gAZIY--mV2T9aDhrkKsJy9ADIFgDEajI9575FxeVTM/edit# * Interview Customers around SaaS release pain points and experiences * Research on SaaS Changes and delivery of those changes - https://gitlab.com/gitlab-org/ux-research/-/issues/2434 * Review Major milestone cadence and fit for SaaS / Self Managed with `@dorrino` * Pull together insight and create opportunity canvas backed by data for investment case * Update Delivery Direction and roadmap based on insights & Migrate to Direction section`