[Feature flag] Rollout of `remove_description_html_in_release_api`

Summary

This issue is to rollout the feature on production, that is currently behind the :remove_description_html_in_release_api feature flag.

Related !60380 (merged)

Owners

  • Team: ~"group::release"
  • Most appropriate slack channel to reach out to: #s_release
  • Best individual to reach out to: @shinya.maeda
  • PM: @kbychu

Stakeholders

The Rollout Plan

  • Partial Rollout on GitLab.com with testing groups
  • Rollout on GitLab.com for a certain period (How long)
  • Percentage Rollout on GitLab.com
  • Rollout Feature for everyone as soon as it's ready

Testing Groups/Projects/Users

  • gitlab-org/gitlab project
  • gitlab-org/gitlab-com groups
  • ...

Expectations

What are we expecting to happen?

This removes the redundant description_html field from the Release LIST API. We can expect a significant improvement for the duration of API.

See the actual performance improvement on Grafana https://dashboards.gitlab.net/d/api-rails-controller/api-rails-controller?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-controller=Grape&var-action=GET%20%2Fapi%2Fprojects%2F:id%2Freleases.

What might happen if this goes wrong?

If a user depends on this specific field, they might report us that it's affecting their tools.

What can we monitor to detect problems with this?

We should keep looking at the GitLab Issue Tracker that the there are no bug reports.

Rollout Steps

Rollout on non-production environments

  • Enable the feature globally on non-production environments
    • /chatops run feature set remove_description_html_in_release_api true --dev
    • /chatops run feature set remove_description_html_in_release_api true --staging
  • Verify that the feature works as expected. Posting the QA result in this issue is preferable.

Preparation before production rollout

  • Check if the feature flag change needs to be accompanied with a change management issue. Cross link the issue here if it does.
  • Ensure that documentation has been updated (More info)
  • Announce on the feature issue an estimated time this will be enabled on GitLab.com
  • If the feature flag in code has an actor, enable it on GitLab.com for testing groups/projects.
    • /chatops run feature set --project=gitlab-org/gitlab remove_description_html_in_release_api true
  • Verify that the feature works as expected. Posting the QA result in this issue is preferable.

Global rollout on production

  • Incrementally rollout the feature. If threre is no risk that the feature affects usability or server loads, skip to the global rollout.
    • If the feature flag in code has an actor, perform actor-based rollout.
      • /chatops run feature set remove_description_html_in_release_api 25 --actors
      • /chatops run feature set remove_description_html_in_release_api 50 --actors
      • /chatops run feature set remove_description_html_in_release_api 75 --actors
    • If the feature flag in code does NOT have an actor, perform time-based rollout a.k.a random rollout.
      • /chatops run feature set remove_description_html_in_release_api <rollout-percentage>
  • Enable the feature globally on production environment
    • /chatops run feature set remove_description_html_in_release_api true
  • Announce on the issue that the feature has been globally enabled
  • Cross post chatops slack command to #support_gitlab-com (more guidance when this is necessary in the dev docs) and in your team channel
  • Wait for at least one day for the verification term.

(Optional) Release the feature with the feature flag

If you're still unsure whether the feature is deemed stable but want to release it in the current milestone, you can change the default state of the feature flag to be enabled. To do so, follow these steps:

  • Create a merge request to set the default_enabled attribute in the feature flag definition to true. Ask for review and merge it.
  • Wait until the above MR has landed on production, that usually takes one working day. To check if it has been deployed, run /chatops run auto_deploy status 26691998b0a7ae097386e43009ced94e0c6180aa. If the merge request was deployed before the code cutoff, the feature can be officially announced in the release blog post.
  • Close the main feature issue to indicate the feature has been implemented.

WARNING: This approach has the downside that it makes harder situation for us to cleanup the flag. For example, on-premise users could disable the feature on their GitLab instance, but when you remove the flag at some point, they suddenly see the feature as enabled that they can't roll it back to the previous behavior. To avoid this potential breaking change, use this approach only for urgent matters.

Release the feature

After the feature has been deemed stable, the cleanup should be done as soon as possible in order to permanently enable the feature and reduce complexity in the codebase.

  • Create a merge request to remove :remove_description_html_in_release_api feature flag. Ask for review and merge it.
    • Remove all references to the feature flag from the codebase
    • Remove the YAML definitions for the feature from the repository
    • Create a changelog entry
  • Wait until the above MR has landed on production, that usually takes one working day. To check if it has been deployed, run /chatops run auto_deploy status 7617ca65cf8a408a00257a7f7351f3dc6225d8c6. If the merge request was deployed before the code cutoff, the feature can be officially announced in the release blog post.
  • Close the main feature issue to indicate the feature has been implemented.
  • Clean up the feature flag from all environments by running these chatops command in #production channel:
    • /chatops run feature delete remove_description_html_in_release_api --dev
    • /chatops run feature delete remove_description_html_in_release_api --staging
    • /chatops run feature delete remove_description_html_in_release_api
  • Close this rollout issue.

Rollback Steps

  • This feature can be disabled by running the following Chatops command:
/chatops run feature set remove_description_html_in_release_api false
Edited by Shinya Maeda