Skip to content

Test list rendering in MRs

Marcel Amirault requested to merge test-approvals into main

Proposals

Change from separate deprecation/removal pages to "Important changes in GitLab XX" pages.

The current pages are too big, and the lists will grow forever, so we should have pages for each major version and all minor versions until the next major version. So one page for GitLab 14, one for GitLab 15, and so on.

Most breaking changes happen in the major milestone, so if the new pages are sorted by milestone, starting with XX.0, the breaking changes will be near the top of the page.

Only use "deprecation"/"removal" terminology if it's truly a deprecation or a removal.

The documentation should be flexible enough that a PM/EM can write the exact information that users need to know, without needing to massage the language to fit the "deprecation -> removal" concept.

Other possible important changes that might need to be explained with different language:

  • Feature flag status changes.
  • Limit additions/changes.
  • Changes to a feature, like where it is in the UI, the tier changing, etc.
  • API/GraphQL changes might warrant different language as well, or even a different page as these are very common but clutter the pages. Perhaps one page for REST deprecations/changes, one for GraphQL?
  • Feature that are no longer recommended, or perhaps no longer being developed, but not yet deprecated. "Feature A is no longer in development, but current users can continue using it and receiving support."

Deprecations should be actionable

This is a suggestion to improve the deprecation process itself.

Right now, we're often treating deprecations like notifications of an incoming removal. But users see a deprecation as a warning that you need to take action, to prevent a breaking change in the future. It's a subtle difference, but important. When they see a deprecation announcement, they expect to be able to do something immediately.

We should make it a requirement that all deprecations are fully actionable. Some examples:

  • Deprecation: types keyword being removed from CI/CD configuration.
    Action: Start using the stages keyword, which already exists.
  • Deprecation: Certificate-based Kubernetes integration being removed from GitLab.
    Action: Start using the Agent for Kubernetes (already exists).
  • Deprecation: Pseudonymizer feature is being removed from GitLab, with no replacement.
    Action: Stop using Pseudonymizer.

Then our requirement that breaking changes be announced at least 2-3 milestones before the major milestone means:

  • We're not giving users 2-3 milestones to prepare for the change. We're giving them time to make the change.
  • It isn't good enough to just make an annoucement, we must have the deprecation action available at least 2-3 milestones before the breaking change as well. In fact, the deprecation process could be essentially: "Step 1, prepare the replacement, step 2, create the deprecation, Step 3..."

An example that we should try to avoid would be:

  • Deprecation: Feature A is being removed, to be replaced with a future feature. Action: Wait for the future feature, and switch to it when it becomes available.

In this case, the deprecation should be delayed until the replacement is available.

Non-deprecation breaking changes

We also need to think about the language for non-deprecation breaking changes, as well as the action users should take. For example:

  • A feature being moved to Ultimate-only shouldn't be considered a deprecation, but is a breaking change that should be announced early. The action would be to upgrade to Ultimate, or stop using the feature. We can document examples of several kinds of announcement styles.
  • A new limit being added, especially to .com, isn't a deprecation, but would require action. Like "The maximum numbers of jobs in a pipeline will be set to 1234". The action would then be to bring the project within the limit range before it is added.

Previous issue content

Click me to expand/collapse.

As we've come closer and closer to 15.0, we've increasingly seen problems with the deprecation/removal notification process.

Currently, the process is to "deprecate" and notify users at least 2 releases before we then "remove" and notify users of the removal. Deprecations and removals can be considered Breaking Changes, or not.

But this somewhat simple process is not flexible enough for the variety of important changes that we implement each month, and repeated complaints seem to highlight this.

  • Quotes from the thread in gitlab-org/gitlab!84339 (comment 901244690):
    • That is not what deprecation notices are for. They mean "Check for a replacement NOW, so you don't have issues later."
    • It's a different story for "we are just dropping this without replacement, stop using it" deprecations.
    • The "deprecated" makes it sound like you should not use it anymore, which is why people are concerned about a lack of replacement.
  • From: gitlab-org/gitlab!82646 (comment 874257859)
    • it seems like we're experiencing some scope creep as to what qualifies as a deprecation/removal. I've also been asked by other PMs to add things to the deprecation/removal doc that aren't deprecations or removals. To my knowledge, the deprecations/removals doc isn't for breaking changes that aren't caused by feature deprecations/removals.
  • From: gitlab-com/www-gitlab-com#13230 (comment 923349881)
    • (Paraphrasing) A breaking change notice was late because it was neither a deprecation nor a removal, and it was not intuitive that it should be added to either page.
  • etc...

Expanding on the issues

Breaking changes

Breaking changes can come from all types of changes, including:

  • Removals
  • Limits (New/changed)
  • Feature flag default state changes.
  • Feature defaults changing from to on/off (or off/on).
  • Feature changing tiers.
  • Any other change in behavior that forces a user to take action before they can continue to work as normal.

It's notable that the only page that exists for these breaking changes is the removals page. I'm not considering the deprecations page for this, because a deprecation itself is not actually a breaking change (though it may warn of a future breaking change).

Deprecations:

People generally expect deprecation warnings to mean: "Stop doing X" (often combined with a "Start doing Y instead"). But our deprecation announcements are rarely that clear or simple, and the variability is huge:

  • Sometimes immediately actionable:
    • "The types keyword is deprecated, start using the stages keyword".
    • "The X integration has been discontinued and will not be replaced." (The action is to just stop using the feature) outright.
  • Sometimes not immediately actionable:
    • "The X feature is deprecated, it will be replaced in milestone Y.yy."
    • "This feature's behavior will change in future milestone Y.yy"

We also have trouble with many deprecation notices not really being deprecations, but being written as such to make it fit on the page, because there is no other place for these announcements. For example (paraphrased):

  • Using this feature in the Premium tier is deprecated and will be removed in 15.0. It is not deprecated for use in Ultimate.
  • Using more than X jobs in a pipeline is deprecated and will be removed in 15.0.

Removals

Similar to the issues raised in the breaking changes section above, the removals page will have a huge variety of types of removals in 15.0 (if we use the earlier deprecation notices as a prediction). But many of these will not be things we'd consider deprecations, but are listed there because they are breaking changes that we want people to be aware of.

Product documentation

Outside of the deprecation/removals pages, we have a lot of feature docs that may or may not also have details about the upcoming changes, and I'm not sure we've got a solid standard for what to do with the feature docs.

  • Should deprecation/removal warnings be in the feature docs too?
  • Should removed features continue to be documented to support self-managed users on older versions that are still supported by GitLab support?

Brainstorming

I believe we should make significant changes to our processes for deprecations/removals/breaking changes, and have some ideas.

Idea 1 - Limit use of Deprecation terminology

We should only use the term Deprecation when we mean: "Stop doing A, start doing B."

In many cases, it seems like "End of development" announcements might make more sense. For a real example: https://docs.gitlab.com/ee/ci/yaml/#only--except. This is even more useful when there is no replacement yet available.

Idea 2 - New page for important announcements

Instead of a deprecation page and a removal page, we could have one "Notable changes" (title TBD) page where we list everything we think people should really know about. It can be important changes to feature in the current milestone, or planned changes to features in future milestones.

Breaking changes of any kind can then be added to the page. New limits, feature changes, removals, etc. If it's important for people to know, we can put it there.

We have badges in the docs (see the red ultimate here: https://docs.gitlab.com/ee/administration/monitoring/gitlab_self_monitoring_project/#take-action-on-prometheus-alerts), with on-hover text, and a link. We could add a new Breaking Change badge to add to all sections on this doc that are considered breaking changes. On-hover text will give a little bit more detail, and clicking it will bring you to a page that gives more details about breaking changes in general. Like major ".0" milestones for most breaking changes, minor milestones for important security fixes, etc.

Sorting up for discussion, but perhaps some kind of "Milestone when you can/need to do something about it". For example, a deprecation could be announced in any milestone, but perhaps should be listed under the milestone when the replacement is expected to be available. A removal warning for an integration being dropped in 15.0 with no replacement should be a breaking change warning listed under 15.0.

Idea 3 - Have boilerplate language for product docs

Similar to our documentation for feature flagged features: https://docs.gitlab.com/ee/development/documentation/feature_flags.html#use-a-note-to-describe-the-state-of-the-feature-flag

We should have a warning box style for important notices about the feature (or we can just reuse FLAG:), along with a collection of standard language that can be copied and pasted into the box and added to the feature docs themselves.

I expect the language to be simple and clear, and easily pasted as needed. You may need to mix and match. Some potential examples:

  • This feature has reached end of development and will no longer receive updates, but can still be used.
  • This feature is deprecated. You should now use Y instead.
  • This feature will be removed in X.xx. You will need to use Y.
  • This limit will be increased/decreased in X.xx. To prepare, you should Y.
  • This feature's feature flag will be enabled by default in XX.x. To prepare, self managed users can X. SaaS users can Y.
  • etc... (these are all just quick ideas, the language is not polished yet)

Extras

  • Should we be thinking about severity of the changes?
  • We should research what other companies are doing for these kinds of announcements.
  • I imagine self managed users would love it if all breaking changes were added 2 releases ahead of a major milestone, but behind a disabled feature flag. They'd have two full releases to enable the flag whenever they want, and they'd be fully ready when the major milestone comes around that sets the flag to enabled by default (because it'll already be enabled on their instance). Wouldn't work for SaaS...
  • How to handle security breaking change announcements?

OLD Previous Issue discussion

In gitlab-org/gitlab!82646 (comment 874257859), @ngaskill points out:

It seems like we're experiencing some scope creep as to what qualifies as a deprecation/removal. I've also been asked by other PMs to add things to the deprecation/removal doc that aren't deprecations or removals. To my knowledge, the deprecations/removals doc isn't for breaking changes that aren't caused by feature deprecations/removals. Something for the TW team to discuss.

Is everything we are adding to the "deprecations" and "removals" pages truly a deprecation or removal?

Some examples of breaking changes that are not deprecations:

  • A project setting changing from default disabled to default enabled. Happens from time to time, and could easily be a breaking change, but not a deprecation.
  • Instance limit updates. Happens quite frequently actually, usually to improve performance or reduce potential abuse of a limit that was too high. You try to rewrite it to make it more like a deprecation by focusing on the ability that is being removed: "The ability to create job names with more than 256 characters is being deprecated". I don't think most people consider that a deprecation though, and it might be more easily understood as simply a new limit: "We are introducing a limit on the length of job names" (not a deprecation).

So how do we handle notifying users about:

Right now, they are all being put in the same deprecations and removals pages due to being similar, but they are not exactly the same.

Proposals

Perhaps we should:

  • Rename the pages? Some initial brainstorming ideas:

    • Deprecations and planned changes / Removals and changes
    • Deprecations and announcements / Removals and important changes
    • Deprecations and planned breaking changes / Removals and breaking changes
    • Planned changes / Important changes by GitLab release version
    • Planned breaking changes / Breaking changes by GitLab release

  • Change what's on the page?

    A lot of the suggestions above relate to changing the focus on the pages from deprecations to essentially "Any important change we really want users to be aware of". So perhaps, along with a rename, we should change what kind of content is on these pages. Looking at deprecations, for example, these are some of the options:

    • List all deprecations only, both breaking and non-breaking. <- Current

    • List all deprecations, and also non-deprecation changes that are breaking.

    • List only breaking changes. That includes deprecations, but does NOT include deprecations that are not breaking changes.

      This is an interesting idea, because I suspect what people really need/want to know are the breaking changes, no matter the type. Deprecations that are not breaking, like the removal of support of some integration that already didn't work because the related product itself is gone, doesn't need to be documented here, does it? It could be handled in the regular feature docs?

Edited by Marcel Amirault

Merge request reports