Add internal release to release handbook

**Monthly self-managed release**

GitLab version (XX.YY.0) [is published every month](https://gitlab.com/gitlab-org/release/docs/blob/master/general/monthly/process.md). From this monthly release, [planned](/handbook/engineering/releases/patch-releases/#planned-patch-release-process), and [unplanned critical](/handbook/engineering/releases/patch-releases/#unplanned-critical-patch-release-process) patch releases are created as needed.

GitLab version (XX.YY.0) [is published every month](https://gitlab.com/gitlab-org/release/docs/blob/master/general/monthly/process.md). From this monthly release, [patch releases](/handbook/engineering/releases/patch-releases/#patch-release-types) are created as needed.

Our [maintenance policy](https://docs.gitlab.com/ee/policy/maintenance.html) describes in detail the cadence of our major, minor and patch releases for self-managed users. The major release yearly cadence was defined [after an all stakeholder discussion](https://gitlab.com/gitlab-com/Product/issues/50).

Patches fix regressions in the current self-managed release and vulnerabilities in the current and previous two GitLab versions.

If you're a GitLab engineer looking:

- To include a bug fix in a patch release, please follow the steps on the [patch release runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/patch/engineers.md).

- To include a security fix in a patch release, please follow the steps on the [security runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/engineer.md).

Overview of the steps involved with creating a patch release:

![Patch release overview](/images/engineering/releases/patch-releases/patch-release-overview.jpg)

Details of the patch release lifecycle can be found on the [patch release section](/handbook/engineering/releases/patch-releases/)

If you're a GitLab engineer looking:

- To include a bug fix in a patch release, please follow the steps on the [patch release runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/patch/engineers.md).

- To include a security fix in a patch release, please follow the steps on the [security runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/engineer.md).

If you're a GitLab release manager looking:

- To include a bug fix in a patch release, please follow the steps on the [patch release runbook for GitLab release manager](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/patch/release_managers.md).

- To include a security fix in a patch release, please follow the steps on the [security runbook for GitLab release manager](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/release_manager.md).

### Patch release information dashboard

GitLab team members can view the [internal Grafana dashboard "Release Information"](https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1) for the following information about the active patch release:

The metrics used to display this information are updated automatically throughout the [patch release process](#patch-release-process).

## Internal release overview

Internal releases are reserved for delivering high-severity fixes to GitLab single-tenant SaaS instances like

GitLab Dedicated before public disclosure within [remediation SLAs](/handbook/security/product-security/vulnerability-management/sla/)

and without disrupting the regular release process.

### Internal release process

Internal releases address high-severity issues in the applicable versions running on single tenant SaaS instances.

Overview of the steps involved with creating an internal release:

![Internal release overview](/images/engineering/releases/internal-releases/internal-release-overview.jpg)

- [Diagram source](https://docs.google.com/presentation/d/1rI47asPEzIaAGZ6t4rQASv88jnJJ17y55k3yD9IVkVI/edit?usp=sharing)

Details of the internal release lifecycle can be found on the [internal release section](/handbook/engineering/releases/internal-releases/)

If you’re a GitLab engineer looking to fix a high-severity issue via an internal release, please follow the steps on the

[runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/internal-releases/engineers.md).

If you’re a GitLab release manager looking to fix a high-severity issue via an internal release, please follow the steps on the

[runbook for GitLab release manager](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/internal-releases/release_managers.md).

## Backports overview

For information on backports, visit the [backports page](/handbook/engineering/releases/backports).

- Self-managed releases:

  - [Monthly releases](https://gitlab.com/gitlab-org/release/docs/blob/master/general/monthly/process.md)

  - [Planned Patch releases](/handbook/engineering/releases/patch-releases/#planned-patch-release-process)

  - [Unplanned Critical Patch releases](/handbook/engineering/releases/patch-releases/#unplanned-critical-patch-release-process)

  - [Patch release types](/handbook/engineering/releases/patch-releases/#patch-release-types)

- GitLab.com releases:

  - [Auto-deploy releases](https://gitlab.com/gitlab-org/release/docs/blob/master/general/deploy/auto-deploy.md)

  - [Hot patch](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/deploy/post-deployment-patches.md)

Depending on the bug details, follow one of the following processes:

- For [high severity security bugs](/handbook/engineering/releases/patch-releases/#planned-patch-release-process)

- For [high severity bugs affecting self-managed users](/handbook/engineering/releases/patch-releases/#planned-patch-release-process). If the bug has been found close to the [release date](/handbook/engineering/releases/) of the month please also alert the Release Managers in [#releases](https://gitlab.slack.com/archives/C0XM5UU6B).

- For [high severity security bugs](/handbook/engineering/releases/patch-releases/#patch-release-types)

- For [high severity bugs affecting self-managed users](/handbook/engineering/releases/patch-releases/#patch-release-types). If the bug has been found close to the [release date](/handbook/engineering/releases/) of the month please also alert the Release Managers in [#releases](https://gitlab.slack.com/archives/C0XM5UU6B).

- For [high severity bugs affecting GitLab.com](/handbook/engineering/deployments-and-releases/deployments/#gitlabcom-pick-label)

- For [high security bugs affecting security merge requests](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/security/bugs_introduced_by_security_merge_request.md)

---

title: "Internal Releases"

---

## Internal release overview

Internal Releases are private releases of GitLab for our single-tenant SaaS instances. They allow us to remediate

high-severity issues on Dedicated instances:

* As quickly and efficiently as on GitLab.com

  ([SLA driven](/handbook/security/product-security/vulnerability-management/sla/#vulnerability-management-slas-and-labels))

* Without version skips in the public packages

* Without disclosing vulnerabilities before a public patch release

Internal releases are performed according to a specific criteria:

* Addressed [critical (S1)](/handbook/security/product-security/vulnerability-management/sla/#vulnerability-management-slas-and-labels)

  fixes (bug or security vulnerability) that impact GitLab Dedicated availability: Security or bug fixes

  1. **Security vulnerability**: The SIRT team investigates a vulnerability and deems the issue to be of high

   severity.

  2. **Critical bug**: The Dedicated team reports a high-severity issue causing a performance degradation.

* Target the current minus one (N-1) and current minus two (N-2) GitLab versions

* Deliver fixes through a private channel before public disclosure

If you’re a GitLab engineer looking to fix a high-severity via an internal release, please follow the steps on the

[internal release runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/internal-releases/engineers.md?ref_type=heads).

## Internal release process

The end-to-end internal release process consists of the following stages:

![internal release overview](/images/engineering/releases/internal-releases/internal-release-overview.jpg)

* [Diagram source - internal](https://docs.google.com/presentation/d/1rI47asPEzIaAGZ6t4rQASv88jnJJ17y55k3yD9IVkVI/edit?usp=sharing)

An internal release has the following phases:

1. **Detect**: A high-severity issue (S1) impacting GitLab Dedicated availability is identified.

   * Security vulnerability: The SIRT team investigates the vulnerability and deems the issue to be of a high

     severity.

   * Critical bug: The Dedicated team reports a high-severity issue causing a performance degradation.

2. **Prepare**: The first step in the internal release process, when a release issue is created and stakeholders,

   including the [GitLab Dedicated Group](/handbook/engineering/infrastructure/team/gitlab-dedicated) are notified.

3. **GitLab.com remediation**:

   * The group relevant to the vulnerability/bug prepares the security fix on the GitLab security repositories.

   * Release managers merge the fix to the GitLab default branch.

   * The high-severity fix is deployed to GitLab multi-tenant production environment (GitLab.com).

   * In case of a vulnerability, the [AppSec team](/handbook/security/product-security/application-security/#contacting-us)

     verifies that the vulnerability/bug has been remediated on GitLab.com.

4. **Backports**: Security merge requests targeting N-1 and N-2 stable branches are prepared by the relevant group

   associated with the vulnerability/bug.

   * Engineers ensure the backports are ready to be merged (approvals, green pipelines, etc).

   * Once the backports are ready to be merged, release managers merge them into the stable branches.

5. **Release**: The internal CNG images and Omnibus packages are built and uploaded to the pre-release channel.

6. **Final Steps**: Roll out the internal release packages to GitLab single-tenant SaaS instances.

   * The GitLab Dedicated Group is notified.

   * A [Dedicated emergency maintenance](https://docs.gitlab.com/administration/dedicated/maintenance/#emergency-maintenance) process

     starts.