Proposal: Support a 12 month Long Term Support (LTS) release with only backward-compatible upgrade stops between LTS releases

The problem

The current GitLab release and maintenance policy where we apply security updates to the latest three minor releases does not meet the needs of a segment of customers.

  • Customers seem to be happy to apply security patch releases as they are seen as low-risk of causing outages.
  • Only providing security fixes for the last three minor versions means that a customer must update to at least a new minor or major version release at least once every three months to keep their deployments free of known security vulnerabilities.
  • Qualifying new releases every three months can be too much effort for some customers.
  • Customers may value stability over new features, so upgrading to a a new minor or major number release every three months (or less) only adds to their deployment workload for not much perceived benefit.

Proposed solution

Create a Long Term Support (LTS) release that is supported for 12 months based on each year's x.11.x. monthly release. Example:

  • 17.11.x is an LTS release
  • 18.11.x is the next LTS release

The x.11.x monthly release was chosen as it includes both the current PostgreSQL major number release and the next PostgreSQL major number release. PostgreSQL can then be upgraded to the newer release required in the following LTS release. One disadvantage of using such a late monthly release is that a customer on LTS releases will have to wait 11 months to get the new functionality available in the the next major number release.

Six and 24 month LTS release cycles were also considered. A six month LTS release was rejected as being too short to make much of a difference, while a 24 month release would create engineering challenges for backporting code to that old of a release.

Variants of this proposal can be found at:

Requirements

The following requirements must be met to implement the proposed solution: ​

Only backwards-compatible upgrade stops between LTS releases

Upgrade stops are permitted as normal, but all stops must be backwards-compatible between x.11.x releases. No intermediate upgrade stops are required when going directly from one LTS release to the next. The X.2, X.5, X.8, and X.11 upgrade stops remain as normal (other than the backwards-compatibility requirement) for non-LTS installations. This requirement implies:

  • Backwards-compatibility code cannot be removed until the next x.11.x release.
  • All migrations in upgrade stops between LTS releases must be applied when upgrading directly from one LTS release to the next.

Warning: This requirement is a major change from current practice and may be difficult or impractical to implement for the following reasons:

  • Maintaining backwards-compatibility over 4 intervening upgrade stops instead of 1 will create a lot of extra development work.
  • When upgrading from one LTS release to the next, the previous 4 upgrade stop's worth of migrations will have to be applied in one upgrade stop which might cause downtime. ​

LTS releases only get security fixes

LTS releases will only get security and selected bug fixes just like the previous two stable releases as described in Patch releases.

Can convert to non-LTS release from an LTS release

A customer may upgrade from an LTS release to a later normal release to take advantage of new desired features and then convert back to the next LTS release when it becomes available.

LTS release is an upgrade stop

Every LTS release is an upgrade stop. Customers on previous non-LTS releases may upgrade to it using the normal upgrade process (possibly including intermediate upgrade stops).

Licensing and revenue generation

LTS releases may require a license and/or SKU to enable additional revenue generation. One of the following options can be chosen to enforce licensing:

  • Technical licensing

    LTS installations have code the checks for a license key needed to operate fully. Implementing such a scheme would require a separate set of installation artifacts, e.g. the current GitLab-CE and GitLab-EE, with an added GitLab-EE-LTS set which includes the new licensing code.

  • Installation artifact availability

    Limit the availability of LTS artifacts to customers that have installation credentials. The initial x.11.0 and subsequent patches would be available to all customers. But credentials would be needed to access any updates that fall outside the initial x.11.x support window. Example:

    17.11.0 and subsequent patches installation artifacts are available using the conventional repositories. All subsequent 17.11.x patches are only available at credential protected repositories once 18.2.0 is released and 17.11.x drops off the conventional stable releases.

  • Honor system

    No technical protection of any kind is used to prevent customers from using LTS artifacts. Customer support might refuse to support any installation of LTS code unless either purchased or solely to get the customer upgraded to the next supported release.

  • Forgo revenue generation

    No licensing or SKU is required to use LTS versions. We forgo revenue generation, but we get the following advantages:

    • Customers have more incentive to keep systems patched since they can stay on stable releases for longer.
    • No technical licensing code or additional repositories are required.
    • LTS releases are treated just like another stable release.

Linux packages, helm charts, and operator available for LTS

LTS releases will be available for Linux packages, helm charts, and operator deployment methods.

LTS releases never exposed to fixed security issues

A customer's LTS deployment is never exposed to any security issues fixed in later releases even when upgrading as long as they upgrade to the next LTS releases when it becomes available.

No excess downtime required on upgrades

No excess downtime will be required when upgrading from one release to the next. Some downtime is permitted as an upgrade will cross major number releases, but it should not exceed the time needed for a typical maintenance window.

12 + 0-2 months of support

There is one week between a new monthly release and the next patch release. If that release is an x.11.x (the next LTS release) customers will have a single week to both qualify and upgrade from the previous release or risk being exposed to any security/bug fixes in the subsequent patch. We may want to increase the support time for the previous release by one or two months to give customers time for qualification and upgrade. This extension will increase the number of releases to patch by one for the length of the extension.

Example:

With a zero-length extension

18.11.0 is released on the third Thursday of of the month

  • 17.11.20 LTS release
  • 18.9.6 next to last stable release patch
  • 18.10.4 last stable release patch
  • Latest current minor number 18.11.0/next LTS release

One week later, the next set of patches are released. LTS 17.11.x is no longer supported, the customer's system is vulnerable unless upgraded to 18.11.1.

  • 17.11.20 LTS release (does not get patched)
  • 18.9.7 next to last stable release patch
  • 18.10.5 last stable release patch
  • Latest current minor number 18.11.1/next LTS release

With a one or two month extension, the previous LTS release continues to get security and bug fixes along with the current LTS release and the customer has a cushion of time to qualify and upgrade their installation at the expense of GitLab having to support 5 instead of 4 patch releases for the length of the extension.

Impact

Implementing this proposal has potential or given impacts to many areas within Gitlab.

GitLab

  • Possible additional revenue generation.

Customer

  • Potential downtime during upgrade from one LTS version to the next.

General development

  • Requiring backwards-compatibility between upgrade stops may greatly slow development of new features as backwards compatibility must be retained for much longer periods (12 minor releases instead of 3).
  • Backporting fixes to up to a year old release may be difficult.
  • 5-6 back-ports are now required instead of 4 for each patch cycle.
  • Adding technical licensing code.

Build

  • Pipelines will have to be modified to create new LTS artifacts.
  • Runner use will increase to create additional patches.
  • Storage will increase to hold additional artifacts.
  • Additional work may be needed to put LTS artifacts in separate repositories.

Release team

  • The release team will now have to patch 4 or 5 releases instead of the current 3.
  • Tooling will have to be adjusted to support the additional patch releases.
  • Additional release environments may have to be created to support the additional releases.
  • Tooling may need to be created to ensure that we don't forget to apply fixes to LS releases.

QA team

  • Upgrade testing would need to be expanded to involve LTS versions.
  • We might need to expand our current scheduled pipelines for self managed upgraded, done using upgrade-tester, to include the patches we support. If not running on a schedule, we need to run them on a specific cadence to get a confidence that the upgrade for those patches is working fine.
  • We may need to have separate test/pipelines setup if new Gitlab-EE-LTS versions are required.

Charts and Operator

  • Coordinating LTS releases with Kubernetes releases may be difficult as the Kubernetes releases initially tested with an LTS release go end-of-life.
  • Current operator releases support the last three GitLab releases. This will have to change to the last 3 releases plus any LTS releases.

Implementation schedule

The next 17.11.0 release is in April 2025. The 17.11.0 stops getting patches two months later when 18.2.0 is released. Work to implement this solution needs to be completed by then or will have to wait until April 2026. Implementation should thus be started in FY26-Q1 or FY26-Q2.

Edited by Andrew Patterson