Skip to content

Implementation of Breaking changes in .com deployments

NOTE - 13.0 IS SCHEDULED FOR MAY 22 AT THIS TIME (NO CHANGES) - Delivery will be pushing features sooner on .com

Presently GitLab is planning to release our next major release on May 22nd: https://docs.gitlab.com/ee/policy/maintenance.html#versioning

Delivery will be rolling features using CD. So during the month breaking changes could be implemented before May 22nd.

These interruptions could have an impact on our customer experience and perception of inadequate communications.

Release process for breaking changes

Feature flags should be our first option that we consider. If you put a changing behavior behind a feature flag, you can remove it and the possible code in a patch release.

Breaking changes behind a feature flag

  1. Set feature flag such that the breaking change is enabled by default
  2. Notify release managers - exact process TBD
  3. Once the update is published to .com, disable the breaking change
  4. On the 22nd during release, we re-enable the feature flags to enable the breaking changes

Breaking changes that cannot be behind a feature flag

  1. BEFORE the MR changing behavior is merged, include @gitlab-org/release/managers group as approvers for the MR, label the MR with "breaking change" label.
  2. Raise an issue at https://gitlab.com/gitlab-org/release/tasks/
  3. In the issue, provide a link to MR(s) and a short description of impact on GitLab.com users.
  4. Provide details of how communication to users of GitLab.com will be done.
  5. Ping @gitlab-org/release/managers so they can make plans on whether they can do a controlled release to .com.
  6. Release managers will attempt to provide an estimated timeline, but be advised that due to timezones the timeline for release might be very short.
  7. It is crucial that all "breaking change" MR's are fully ready, which means reviewed, with passing pipelines NO LATER THAN 2020-05-15. We will not be considering this special process for breaking change past that date. Breaking change MR's past this date will go as regular MR's through our regular process.

Stolen from #897 (closed)

Changes which require a major release

  • Must Have: Yes if the change is required for %13.0 due to the impact of waiting another year
  • On Track: ✅ if progress is going well, ⚠ if at risk, ❌ if it will not make it without a delay.
  • .com affected: ✅ if it affects .com users, ❌ if it does not affect .com users
  • MR-timeframe: List of MR changing compatibility and approximate timeframe for merge (week).

It would be great if we could update the items in the list by Monday of next week so we can get a sense of where we stand.

Enablement ✅

Currently devopsenablement is on track with breaking changes that require %13.0.

DRI Change (Issue/Epic) Must Have? On Track? .com affected MR-timeframe Affect on customers
@mendeni PG11 and removal of 9.6 & 10 (gitlab-org/omnibus-gitlab#5099 (closed) gitlab-org&2414 (closed)) Yes ✅ No GitLab.com is scheduled to upgrade on May 10th gitlab-com/gl-infra&106 (closed) >= PG11 is required
@mendeni Puma opt-out (gitlab-org&954 (closed)) Yes ✅ No Puma was fully rolled to .com on Feb 19 gitlab-com/gl-infra/production#1684 (closed) gitlab-org/omnibus-gitlab!4141 (merged) gitlab-org/omnibus-gitlab!4165 (closed) Puma or Unicorn is supported
@mendeni Remove node['gitlab'] monitoring deprecation in next major release(gitlab-org/omnibus-gitlab#4687 (closed)) No ✅ No gitlab-org/omnibus-gitlab!4139 (merged) Old settings are removed
@mendeni Remove protected paths throttle from Omnibus (gitlab-org/omnibus-gitlab#4750 (closed)) No ✅ No gitlab-org/omnibus-gitlab!4149 (merged) Protected paths throttling doc and logic removed
@mendeni Remove support for and references to postgresql['data_dir'] (gitlab-org/omnibus-gitlab#4806 (closed)) No ✅ No Customers need to use postgresql['dir'] instead
@mendeni Remove Grafana reset logic during upgrade in GitLab 13.0 (gitlab-org/omnibus-gitlab#4891 (closed)) No ✅ No gitlab-org/omnibus-gitlab!4155 (merged) Should be no impact
@mendeni Rename Redis slave to replica (gitlab-org/omnibus-gitlab#5141 (closed)) No ✅ No gitlab-org/omnibus-gitlab!4168 (merged) Will require a redis restart

Secure stage

Currently devopssecure is on track with breaking changes that require %13.0.

DRI Change (Issue/Epic) Must Have? On Track? .com affected MR-timeframe Affect on customers
@gonzoyumo Remove Docker in Docker requirement: gitlab-org/gitlab#37278 (closed) - Dep Notice Strong Desire ⚠ Yes Need to update CI config when customization is based on job name
@gonzoyumo Drop support for license_management CI job, artifact and template License-Management.gitlab-ci.yml: gitlab-org/gitlab#8912 (closed) ? ✅ Yes None if already migrated to new template, otherwise need to update CI to point to new file
@twoodham Transition to rules syntax for Secure's vendored templates: gitlab-org&2300 (closed) Yes ✅ Yes Task to merge all our template changes expected week of May 11th Customers overriding Secure templates AND using only/excludes [will need to migrate to rules](https://docs.gitlab.com/ee/user/application_security/#transitioning-your-onlyexcept-syntax-to-rules)
@gonzoyumo Remove deprecated variable PIP_DEPENDENCY_PATH in gemnasium-python: gitlab-org/gitlab#13305 (closed) ? ✅ Yes Use new variable DS_PIP_DEPENDENCY_PATH instead
@gonzoyumo Remove deprecated proprietary report format (clair scanner) for Container Scanning: gitlab-org/gitlab#35442 (closed) ? ✅ Yes Make sure to use new template and new job definition for Container Scanning
@sethgitlab Remove the legacy way of using DAST: gitlab-org/gitlab#31866 (closed) Yes ✅ Yes After 2020-05-22 Need to update old templates
@gonzoyumo Remove undefined value from Severity and Confidence: gitlab-org/gitlab#202685 (closed) ? ⚠ Yes No longer rely on or expect undefined value but use unknown instead
@twoodham Drop usage of x-y-stable docker images and replace with pining to Major version: gitlab-org/gitlab#207128 (closed) - Dep Notice Strong Desire ✅ No
@gonzoyumo Remove the legacy cve property from the JSON report: gitlab-org&2791 Strong Desire ❌ Yes No long use the cve property but use id instead and make sure to provide location properties

Defend stage

Currently ~"devops::defend" is on track with breaking changes that require %13.0. DRI: @whaber

No breaking changes from the defend stage.

Summarized list of Runner deprecation issues for 13.0

DRI Change (Issue/Epic) Must Have? On Track? .com affected MR-timeframe Affect on customers
@erushton Remove backported os.Expand() implementation Yes ✅ Yes Merged gitlab-org/gitlab-runner!1892 (merged) None.
@erushton Remove support for array of strings when defining services for Docker Executor Yes ✅ No Merged gitlab-org/gitlab-runner!2035 (merged) Customers registering runners with docker executor must use list format
@erushton Remove windows Batch from shell executor Yes ✅ No Merged gitlab-org/gitlab-runner!2037 (merged) Windows Batch not supported as a shell option. Must use powershell and if needed call batch from powershell
@erushton Remove /debug/jobs/list?v=1 endpoint Yes ✅ No Merged gitlab-org/gitlab-runner!1894 (merged) None
@erushton Remove support for --docker-services flag on register command Yes ✅ No In Review gitlab-org/gitlab-runner!2036 (merged) Services must be registered in the toml file
@erushton Remove FF_USE_LEGACY_VOLUMES_MOUNTING_ORDER feature flag Yes ✅ No Merged gitlab-org/gitlab-runner!1889 (merged) Customers must switch to recommended format for mounting volumes
@erushton Remove legacy build directory caching Yes ✅ No Expected within next week
@erushton Remove Windows 1803 support Yes ✅ No Merged gitlab-org/gitlab-runner!2033 (merged) Packaging for Windows 1803 no longer distributed
@erushton Remove fedora/29 packages Yes ✅ No Merged gitlab-org/gitlab-runner!1905 (merged) RPMs for Fedora 29 no longer distributed

Create:Editor ✅

DRI Change (Issue/Epic) Must Have? On Track? .com affected MR-timeframe Affect on customers
@dsatcher Remove Snippet Content from search (UI and API breaking change) Yes ✅ Yes Merged Snippet content search API will return 500

Release:Progressive Delivery

DRI Change (Issue/Epic) Must Have? On Track? .com affected MR-timeframe Affect on customers
@csouthard Legacy Feature Flags Deprecation Plan No ⚠ Yes Multiple MRs; Targeting 13.1 Old "V1" Feature Flags will become read-only in 13.4

Release:Release Management

DRI Change (Issue/Epic) Must Have? On Track? .com affected MR-timeframe Affect on customers
@sean_carroll Deprecate Pages Legacy Architecture Yes ✅ Yes MR Closed Pages architecture will be opt -out so most people will already have the API config enable and disk source will be removed in 14.0
@sean_carroll Remove deprecated Release Evidence endpoints Yes ✅ Yes MR In Review Removed fields accessed differently in API
@sean_carroll Pages Secrets Files No ✅ Yes MR In Review Won't be able to pass secrets on CLI

Configure:Configure

DRI Change (Issue/Epic) Must Have? On Track? .com affected MR-timeframe Affect on customers
@nicholasklick Default Auto DevOps Postgres install to V2 Yes ✅ Yes MR In Review With gitlab-org/gitlab#211890 (closed) implemented, the impact on pipelines utilizing the legacy option is that the pipeline will refuse to proceed. User will have to additionally opt-in to switch from legacy to new option as this actively deletes the legacy DB.
@nicholasklick Moving Auto DevOps jobs syntax to rules Yes ✅ Yes MR In Review Impacted users (small percentage?) will see their pipeline fail to start with something like: jobs:<job-name> config key may not be used with rules
@nicholasklick Change k8s api endpoints from deprecated beta to apps/v1 Yes ✅ Yes MR In Review If user is using Kubernetes 1.9 and below, they will need to upgrade their Kubernetes cluster in order to get apps/v1. Note in https://docs.gitlab.com/ee/topics/autodevops/#requirements, we require Kubernetes 1.12+. If the user's cluster is Kubernetes 1.9 or below, this breaking change will cause them to an error message

Monitor:APM

DRI Change (Issue/Epic) Must Have? On Track? .com affected MR-timeframe Affect on customers
@mnohr Elastic Helm chart deprecation Yes ✅ No MRs merged If a user upgrades to the supported chart, their existing data in ElasticStack will be removed
@mnohr Deprecate influxDB Yes ✅ No MRs merged Only support prometheus going forward

Verify:Continuous Integration

DRI Change (Issue/Epic) Must Have? On Track? .com affected MR-timeframe Affect on customers
@cheryl.li Remove JenkinsDeprecatedService Yes ✅ Yes Merged gitlab-org/gitlab!31607 (merged) Removes the deprecated CI service, JenkinsDeprecatedService and associated DB records

Manage: Access ✅

DRI Change (Issue/Epic) Must Have? On Track? .com affected MR-timeframe Affect on customers
@lmcandrew Remove deprecated /admin/application_settings redirect Yes ✅ Yes Merged No effect unless a self-managed administrator had the endpoint bookmarked

cc: @marin @joshlambert

Edited by Alessio Caiazza