Commit ceaafd82 authored by Mayra Cabrera's avatar Mayra Cabrera 0️⃣
Browse files

Clarify post-release steps for Out-of-band patches and internal releases

parent 8ca5df5d
Loading
Loading
Loading
Loading
+6 −4
Changes for content/handbook/engineering/releases/internal-releases/index.md: 6 added lines, 4 removed lines.
Original line number Diff line number Diff line
@@ -67,8 +67,8 @@ Only after these prerequisites are met can Release Managers begin the ~8 hour ex
## Request process

> [!IMPORTANT]
> Every approved internal release for a **bug fix** requires a [Feature Change Lock (FCL)](/handbook/engineering/#feature-change-locks). Every
> approved internal release requires a post-release [incident review](/handbook/engineering/infrastructure-platforms/incident-review/). **These are not optional.**
> Every approved internal release for a **critical bug fix** requires a [Feature Change Lock (FCL)](/handbook/engineering/#feature-change-locks) and a post-release [incident review](/handbook/engineering/infrastructure-platforms/incident-review/). These are **not optional**. Every
> approved internal release for a **critical security vulnerability** strongly encourages participation in the post-release SIRT/PSIRT retrospective, and **requires** a [Root Cause Analysis (RCA)](/handbook/security/root-cause-analysis/) if the vulnerability is tied to a product feature, including vulnerabilities in upstream third-party code, when a GitLab decision or delay contributed to the exposure. There are no exceptions to the RCA requirement when it applies.

Internal releases are reserved for high-severity bugs or security vulnerabilities that impact Dedicated availability and cannot wait for the next scheduled patch release.

@@ -83,9 +83,11 @@ If an internal release is necessary:
1. **Requestor** contacts the **Engineering Manager** of the affected area, assigns them to the issue, and ensures they start the [Feature Change Lock (FCL)](/handbook/engineering/#feature-change-locks)
   process for bug fixes (**mandatory**).
1. **Dedicated Engineering Manager** (sponsor) provides written approval on the issue.
1. **Software Delivery Engineering Leadership** (Release & Deploy Manager, Senior Product Manager or above) provides written approval.
1. **[Software Delivery Engineering Leadership](/handbook/engineering/infrastructure-platforms/gitlab-delivery/delivery/#software-delivery-engineering-leadership)** (or above) provides written approval.
1. **Requestor** assigns the [active Release Managers](/handbook/engineering/releases/release-managers/) to the issue for final review and execution once all items above are complete.
1. **Post-release:** [Incident review](/handbook/engineering/infrastructure-platforms/incident-review/) to investigate how this escaped detection before release (**mandatory**).
1. **Post-release, for bug fixes:** [Incident review](/handbook/engineering/infrastructure-platforms/incident-review/) to investigate how this escaped detection before release (**mandatory**).
1. **Post-release, for security vulnerabilities:** Participation in the SIRT/PSIRT retrospective issue is **strongly encouraged**. If the vulnerability is tied to a product feature, the team responsible for introducing it must complete a [Root Cause Analysis (RCA)](/handbook/security/root-cause-analysis/). There are no exceptions to this requirement.
   * For vulnerabilities in upstream third-party code, RCA is required only if a GitLab decision or delay, such as not applying an available fix or relying on an end-of-life dependency, contributed to the exposure. Otherwise, RCA is optional.

Internal releases approved outside this process set precedent that undermines release stability and the reliability of GitLab Dedicated.

+21 −14
Changes for content/handbook/engineering/releases/patch-releases/index.md: 21 added lines, 14 removed lines.
Original line number Diff line number Diff line
@@ -88,29 +88,36 @@ At GitLab, there are two types of patch releases processes:
   the [security remediation SLAs](/handbook/security/product-security/vulnerability-management/sla/). Patches that include
   [`critical` vulnerabilities](/handbook/security/product-security/vulnerability-management/sla/) are considered critical patches.
1. **Out-of-band**: A patch outside of the regular [patch release cadence](#patch-release-cadence)
   reserved strictly for mitigating a high-severity bug or critical vulnerability. These ad-hoc
   patches must never be used to accommodate missed deadlines or bypass standard release policy.
   Following the patch release cadence, out-of-band patches are delivered on Wednesdays.
   reserved strictly for mitigating a high-severity bug or critical vulnerability. Out-of-band patches
   must never be used to accommodate missed deadlines or bypass standard release policy, and are delivered
   on Wednesdays, following the regular patch release cadence.

   All out-of-band patches, whether for bug fixes or security vulnerabilities, must satisfy the
   following:
   following, with no exceptions:

   - **Exception process**: An [exception request](/handbook/engineering/releases#exception-process)
     is **mandatory** and must be followed before any out-of-band patch is executed. Release Managers
     are authorized to decline requests that have not completed this process, regardless of business
     pressure.
     is **mandatory** before any out-of-band patch is executed. Release Managers are authorized to
     decline requests that haven't completed this process, regardless of business pressure.
   - **Incident declaration**: An [incident must be declared](/handbook/engineering/infrastructure-platforms/incident-management/#reporting-an-incident)
     with the following attributes:
     - Marked as "Out-of-Band Patch"
     - Severity S1 or S2
     - A Contributing Factor of 'Inadequate testing or QA' or 'Miscommunication or coordination gap'
   - **FCL and incident review**: To uphold the reliability and availability of any GitLab platform,
     completing an [incident review](/handbook/engineering/infrastructure-platforms/incident-review/)
     and an [FCL](/handbook/engineering/#feature-change-locks) is **mandatory**.

   Additionally, for **critical security vulnerabilities**, a [security RCA](/handbook/security/root-cause-analysis/)
   is **required** and must be completed by the team responsible for introducing the vulnerability that forced the
   out-of-band release. There are no exceptions to this accountability.
   - **Incident review**: For out-of-band patches addressing **bug fixes**, completing an
     [incident review](/handbook/engineering/infrastructure-platforms/incident-review/) is
     **mandatory** to uphold the reliability and availability of the GitLab platform. For
     out-of-band patches addressing **security vulnerabilities**, participating in the SIRT
     incident retrospective issue is strongly encouraged instead.
   - **FCL**: An [FCL](/handbook/engineering/#feature-change-locks) is required for out-of-band
     patches addressing **bug fixes**. Does not apply to security vulnerabilities, which are
     instead reviewed by PSIRT at the overall security incident level.

Additionally, a [security RCA](/handbook/security/root-cause-analysis/) is **required**, with no
exceptions, when a **critical security vulnerability** is tied to a product feature and code
owned by an engineering team, including vulnerabilities in upstream third-party code when a
GitLab decision or delay, such as not applying an available fix promptly or relying on an
end-of-life dependency, contributed to the exposure. The team responsible must complete the
RCA. Otherwise, RCA is optional.

## Patch release process