Commit 9d22d0e6 authored by Dave Smith's avatar Dave Smith Committed by John Skarbek
Browse files

docs: expand releases handbook with detailed explanations and general updates

parent 90a721d3
Loading
Loading
Loading
Loading
+106 −17
Original line number Diff line number Diff line
@@ -8,40 +8,129 @@ Looking for product release information? See [release posts](https://about.gitla

GitLab has three different release types:

1. **[Monthly release](/handbook/engineering/releases/monthly-releases/)**: A GitLab version (XX.YY.0) that [is published on the 3rd Thursday of every month](/handbook/engineering/releases/monthly-releases/#monthly-release-schedule), containing new features, bug fixes and performance improvements from many successful [deployments on GitLab.com](/handbook/engineering/deployments-and-releases/deployments/). From this monthly release, [patch releases](/handbook/engineering/releases/patch-releases/) are created as needed.
1. **[Patch release](/handbook/engineering/releases/patch-releases/)**: Patches addressing bug and vulnerabilities per the [GitLab Maintenance policy](https://docs.gitlab.com/policy/maintenance/).
1. **[Internal release](/handbook/engineering/releases/internal-releases/)**: Private releases reserved for delivering high-severity fixes to GitLab single-tenant SaaS instances like GitLab Dedicated within [remediation SLAs](/handbook/security/product-security/vulnerability-management/sla/).
1. **[Monthly release](/handbook/engineering/releases/monthly-releases/)**: A GitLab version (XX.YY.0) published on the 3rd Thursday of every month, containing new features, bug fixes and performance improvements from successful [deployments on GitLab.com](/handbook/engineering/deployments-and-releases/deployments/).
1. **[Patch release](/handbook/engineering/releases/patch-releases/)**: Bug fixes and security updates for [maintained versions](https://docs.gitlab.com/policy/maintenance/#maintained-versions), published twice monthly.
1. **[Internal release](/handbook/engineering/releases/internal-releases/)**: Private releases for delivering high-severity fixes to GitLab Dedicated within [remediation SLAs](/handbook/security/product-security/vulnerability-management/sla/).

## Release Policy

### Why Release Policies Matter

GitLab follows [semantic versioning](https://semver.org/), which establishes a contract with our customers:

* **MAJOR** versions may contain breaking changes
* **MINOR** versions contain new features (backward compatible)
* **PATCH** versions contain bug fixes only (safe to auto-upgrade)

Customers rely on this contract to make upgrade decisions. Many customers auto-upgrade patch releases without extensive testing because they trust that patches contain only bug fixes. When we violate this contract, we:

* **Erode customer trust** in our entire release process
* **Put customer environments at risk** with unexpected changes
* **Undermine release predictability** that customers depend on for planning
* **Create cascading problems** when rushed releases introduce new bugs requiring additional patches

### What Each Release Type Contains

| Release Type | Contains | Does NOT Contain |
|--------------|----------|------------------|
| **Monthly** | New features, bug fixes, performance improvements | N/A - this is the vehicle for all new work |
| **Patch** | Bug fixes, security fixes, performance regressions | New features, feature flag enables, incomplete work |
| **Internal** | Critical bug fixes, security fixes for Dedicated instances | New features, feature flag enables, incomplete work |

For SLO/SLA commitments on bug and security fix timelines, see [Patch release SLO/SLA Commitments](/handbook/engineering/releases/patch-releases/#slo-commitments).

### Ownership When Deadlines Are Missed

When a feature misses the monthly release deadline, the appropriate path is the **next monthly release**, not an exception to release policy.

| Team | Owns |
|------|------|
| Development | Shipping code in the next policy-compliant release |
| Product | Communicating the delay to stakeholders; adjusting roadmap expectations |
| Customer Success | Managing customer expectations; negotiating timeline changes |
| Release | Executing policy-compliant releases that maintain customer trust in GitLab releases |

Release Managers do not own accommodating missed deadlines.

### Exception Process

Exceptions to release policy are rare and require explicit approval outside the Release team.

**Release Managers are authorized to decline requests that violate release policies.** Pressure from Customer Success, Sales, or other teams does not constitute approval for policy exceptions.

If an exception is believed necessary:

1. **Requestor** documents the business justification and customer impact assessment
2. **Requestor** escalates to their Director/VP for sponsorship
3. **Sponsoring Director/VP** requests exception from Engineering leadership within the Delivery/Platforms organization:
   * **Backport exceptions (N-3+)**: Senior Engineering Manager or above
   * **Out-of-band releases**: Director of Engineering or above
   * **Policy violations (features in patches, monthly redos)**: VP of Engineering or above
4. **Approving leader** provides written approval with acknowledgment of policy violation
5. **Release Manager** executes only after receiving written approval
6. **Post-release:** Retrospective scheduled to address root cause of the exception request

Exceptions approved without this process set precedent that undermines all release policies.

### Escalation Path

| Situation | Contact |
|-----------|---------|
| Process questions | #releases Slack channel |
| Exception requests | Your Director/VP (not Release Managers) |
| Policy clarification | [Delivery/Platforms leadership](/handbook/engineering/infrastructure-platforms/gitlab-delivery/#teams) |

## Release Coordination

### Teams Involved

| Team | Responsibility |
|------|----------------|
| [Release Team](/handbook/engineering/infrastructure-platforms/gitlab-delivery/delivery/) | Deployments, Process coordination; executing policy-compliant releases |
| Infrastructure | Environment updates |
| Security | Vulnerability fixes and disclosure coordination |
| Product | Feature readiness; owning delays when features miss deadlines |
| Marketing | Release communications |

### Communication

* **#releases** (Slack): Release status and questions
* **[Release Post](https://about.gitlab.com/releases/categories/releases/)**: Public announcements

## Maintenance Policy

See the [GitLab maintenance policy](https://docs.gitlab.com/policy/maintenance/) for details

## Terminology

* **[Maintenance policy](https://docs.gitlab.com/policy/maintenance/)**: Describes in detail the release pace of our major, minor and patch releases for self-managed users. The major release yearly cadence was defined [after an all stakeholder discussion](https://gitlab.com/gitlab-com/Product/issues/50).
* **[Maintenance policy](https://docs.gitlab.com/policy/maintenance/)**: Describes the release pace of our major, minor and patch releases for self-managed users.
* **Upcoming version**: [New GitLab release](/handbook/engineering/releases/monthly-releases/#monthly-release-schedule) (XX.YY.0) being developed.
* **Maintained versions**: GitLab versions covered by the [maintenance policy](https://docs.gitlab.com/policy/maintenance/#maintained-versions)
* **Maintained versions**: GitLab versions covered by the [maintenance policy](https://docs.gitlab.com/policy/maintenance/#maintained-versions).
* **[Backports](/handbook/engineering/releases/backports/)**: Bug or security fixes from a recent version applied to an older version.
* **[Stable branches](/handbook/engineering/releases/stable_branches/)**: Source of the GitLab release packages delivered to GitLab customers.
* **[Auto-deploy](/handbook/engineering/deployments-and-releases/deployments/)**: GitLab process to deploy application changes to GitLab.com
* **[Release managers](https://about.gitlab.com/community/release-managers/)**: DRI's to deliver GitLab releases and performing deployments to GitLab.com
* **[Stable branches](/handbook/engineering/releases/stable_branches/)**: Source of the GitLab release packages delivered to customers.
* **[Auto-deploy](/handbook/engineering/deployments-and-releases/deployments/)**: GitLab process to deploy application changes to GitLab.com.
* **[Release managers](https://about.gitlab.com/community/release-managers/)**: DRIs for GitLab releases and deployments to GitLab.com.

## Resources

### Monthly release

* [When do I need to have my MR merged in order for it to be included into the monthly release](/handbook/engineering/releases/monthly-releases#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-in-the-monthly-release)
* [How can I determine if my merge request will make it into the monthly release](/handbook/engineering/releases/monthly-releases#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release)
* [Monthly release schedule and process](/handbook/engineering/releases/monthly-releases/)
* [When do I need to have my MR merged?](/handbook/engineering/releases/monthly-releases#when-do-i-need-to-have-my-mr-merged-in-order-for-it-to-be-included-in-the-monthly-release)
* [How can I determine if my MR will be included?](/handbook/engineering/releases/monthly-releases#how-can-i-determine-if-my-merge-request-will-make-it-into-the-monthly-release)

### Patch release

If you're a GitLab engineer looking:

* To include a bug fix in a patch release, please follow the steps on the [patch release runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/patch/engineers.md).
* To include a security fix in a patch release, please follow the steps on the [security runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/engineer.md).
* [Patch release process and policy](/handbook/engineering/releases/patch-releases/)
* [Patch release runbook for engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/patch/engineers.md)
* [Security runbook for engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/engineer.md)

### Internal release

* If you're a GitLab engineer looking to fix a high-severity via an internal release, please follow the steps on the
[internal release runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/internal-releases/engineers.md?ref_type=heads).
* [Internal release process](/handbook/engineering/releases/internal-releases/)
* [Internal release runbook for engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/internal-releases/engineers.md)

### Links

* [Release Manager schedule](https://about.gitlab.com/community/release-managers/)
* [How to reach the Release and Deploy team?](/handbook/engineering/infrastructure-platforms/gitlab-delivery/delivery/#reaching-our-team)
* [How to reach the Release and Deploy team](/handbook/engineering/infrastructure-platforms/gitlab-delivery/delivery/#reaching-our-team)
+9 −17
Original line number Diff line number Diff line
@@ -17,7 +17,7 @@ graph TD
    C -->|Yes| F[Patch Release]
    click C "https://docs.gitlab.com/policy/maintenance/#maintained-versions"
    click E "/handbook/engineering/releases/backports/#what-if-my-backport-is-not-covered-by-the-maintenance-policy"
    click F "/handbook/engineering/releases/#patch-releases-overview"
    click F "/handbook/engineering/releases/patch-releases/"

```

@@ -52,24 +52,16 @@ You can check the [Maintenance Policy](https://docs.gitlab.com/ee/policy/mainten

### What if my backport is not covered by the maintenance policy?

If your backport is not covered by the maintenance policy then it is not guaranteed. However, we realize that there is sometimes a need to backport a fix to more than one stable release, depending on the severity of the bug. Backports that are outside of the maintenance policy are performed at the sole discretion of the [Release Managers](/handbook/engineering/infrastructure-platforms/gitlab-delivery/delivery/#delivery-domain-ownership-between-delivery-teams) .

You can request a backport by:

1. Raising an MR to the stable branches that you want to backport to
1. Ensuring that the pipelines are green and no new issues are introduced
1. Raising a [backport request](https://gitlab.com/gitlab-org/release/tasks/-/issues/new?issuable_template=Backporting-request) in the [release/tasks](https://gitlab.com/gitlab-org/release/tasks) repository
1. PM DRI shares business impact analysis and justification for exception on backport request
1. Release Managers will accept or reject the backport request based on criteria including
    1. Estimated severity of the bug
    1. Estimated priority of the bug
    1. Potentially incurring data loss and/or security breach.
    1. Potentially affecting one or more strategic accounts due to a proven inability by the user to upgrade to a [maintained version](https://docs.gitlab.com/policy/maintenance/#maintained-versions).
    1. Impact to other planned release tasks
1. Wait for confirmation and approval from the RMs
Backports to versions outside the [maintenance policy](https://docs.gitlab.com/policy/maintenance/#maintained-versions) are exceptions and require approval from Engineering leadership within the Delivery/Platforms organization (Senior Engineering Manager or above). This follows the same [exception process](/handbook/engineering/releases/#exception-process) as other release policy exceptions.

To request a backport exception:

1. Raise a [backport request](https://gitlab.com/gitlab-org/release/tasks/-/issues/new?issuable_template=Backporting-request)
1. Further instructions are located in this template.
1. Release Managers execute only after receiving written approval
1. Communicate to your stakeholders if/when the release is available on our [releases blog](https://about.gitlab.com/releases/categories/releases/)

Please note that this is done on a case by case basis and not covered by any of our Maintenance Policies or releases.
Backport exceptions are evaluated on a case-by-case basis and are not guaranteed.

## Backports FAQ

+42 −13
Original line number Diff line number Diff line
@@ -2,6 +2,13 @@
title: "Internal Releases"
---

## Internal Release Policy

Internal releases adhere to the same policy requirements as [patch releases](/handbook/engineering/releases/patch-releases/#patch-release-policy): they are limited to critical bug fixes and security patches only.
Internal releases do not contain new features, feature flag changes, or incomplete work, nor may be used for testing purposes.

For the general release policy framework including ownership, exception process, and escalation paths, see the [Release Policy](/handbook/engineering/releases/#release-policy) section.

## Internal release overview

Internal Releases are private releases of GitLab for our single-tenant SaaS instances. They allow us to remediate
@@ -15,15 +22,40 @@ high-severity issues on Dedicated instances:
Internal releases are performed according to a specific criteria:

* Addressed [critical (S1)](/handbook/security/product-security/vulnerability-management/sla/#vulnerability-management-slas-and-labels)
  fixes (bug or security vulnerability) that impact GitLab Dedicated availability: Security or bug fixes
  fixes (bug or security vulnerability) that impact GitLab Dedicated availability:
  1. **Security vulnerability**: The SIRT team investigates a vulnerability and deems the issue to be of high
   severity.
  2. **Critical bug**: The Dedicated team reports a high-severity issue causing a performance degradation.
* Target the current minus one (N-1) and current minus two (N-2) GitLab versions
* Deliver fixes through a private channel before public disclosure

If you’re a GitLab engineer looking to fix a high-severity via an internal release, please follow the steps on the
[internal release runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/internal-releases/engineers.md?ref_type=heads).
If you're a GitLab engineer looking to fix a high-severity issue and create an internal release, please follow the steps on the [internal release runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/internal-releases/engineers.md).

### Relationship to Other Release Types

Internal releases follow the same process as [patch releases](/handbook/engineering/releases/patch-releases/) but serve a different purpose:

* Both use stable branches created during [monthly releases](/handbook/engineering/releases/monthly-releases/)
* Internal releases deliver fixes to GitLab Dedicated *before* public patch releases
* After an internal release, the same fixes are included in the next scheduled patch release for all self-managed customers

### Timeline

Internal releases have two phases with different time characteristics:

| Phase | Duration | Notes |
|-------|----------|-------|
| **Request and approval** | Variable | Depends on issue severity validation, stakeholder availability, and fix readiness |
| **Execution** | ~8 hours | Once approved and fix is ready, Release Managers can complete the release process |

The request phase includes:

* Issue detection and Dedicated severity assessment
* Stakeholder notification and alignment
* Fix development and validation on GitLab.com
* Backport preparation with passing pipelines

Only after these prerequisites are met can Release Managers begin the ~8 hour execution phase.

## Internal release process

@@ -36,24 +68,21 @@ The end-to-end internal release process consists of the following stages:
An internal release has the following phases:

1. **Detect**: A high-severity issue (S1) impacting GitLab Dedicated availability is identified.
   * Security vulnerability: The SIRT team investigates the vulnerability and deems the issue to be of a high
     severity.
   * Critical bug: The Dedicated team reports a high-severity issue causing a performance degradation.
   * Security vulnerability: The SIRT team investigates the vulnerability and deems the issue to be of a high severity.
   * Critical bug: The Dedicated team reports a high-severity issue causing a usability degradation.

2. **Prepare**: The first step in the internal release process, when a release issue is created and stakeholders,
   including the [GitLab Dedicated Group](/handbook/engineering/infrastructure-platforms/gitlab-dedicated) are notified.
2. **Prepare**: The first step in the internal release process when a release task issue is created and stakeholders, including the [GitLab Dedicated Group](/handbook/engineering/infrastructure-platforms/gitlab-dedicated) are notified.

3. **GitLab.com remediation**:
   * The group relevant to the vulnerability/bug prepares the security fix on the GitLab security repositories.
   * Release managers merge the fix to the GitLab default branch.
   * The group relevant to the vulnerability/bug prepares the fix on the appropriate GitLab repositories.
   * Release Managers merge the fix to the GitLab default branch.
   * The high-severity fix is deployed to GitLab multi-tenant production environment (GitLab.com).
   * In case of a vulnerability, the [AppSec team](/handbook/security/product-security/security-platforms-architecture/application-security/#contacting-us)
     verifies that the vulnerability/bug has been remediated on GitLab.com.

4. **Backports**: Security merge requests targeting N-1 and N-2 stable branches are prepared by the relevant group
   associated with the vulnerability/bug.
4. **Backports**: Security merge requests targeting N-1 and N-2 stable branches are prepared by the relevant group associated with the vulnerability/bug.
   * Engineers ensure the backports are ready to be merged (approvals, green pipelines, etc).
   * Once the backports are ready to be merged, release managers merge them into the stable branches.
   * Once the backports are ready to be merged, Release Managers merge them into the stable branches.

5. **Release**: The internal CNG images and Omnibus packages are built and uploaded to the pre-release channel.

+62 −7

File changed.

Preview size limit exceeded, changes collapsed.

+88 −37

File changed.

Preview size limit exceeded, changes collapsed.

Loading