<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Label this issue](https://contributors.gitlab.com/manage-issue?action=label&projectId=278964&issueIid=600746) </details> <!--IssueSummary end--> ## Summary This issue is to roll out the `duo_add_otel` feature flag on production. The feature flag was converted from `wip` to `gitlab_com_derisk` to prepare for production rollout. Related feature issue: https://gitlab.com/gitlab-org/embody-team/experimental-observability/documentation/-/work_items/57 ## Owners - Most appropriate Slack channel to reach out to: `#embody-team` - Best individual to reach out to: @dakotadux ## Expectations ### What are we expecting to happen? The feature adds a button to the project page that instruments OpenTelemetry. This flag gates the `duo_otel_workflow_available` policy condition in `ee/app/policies/ee/project_policy.rb`. ### What can go wrong and how would we detect it? Users could see the feature before it's ready for their environment. Monitor error rates on dashboards. ## Rollout Steps ### Rollout on non-production environments - Verify the MR with the feature flag is merged to `master` and has been deployed to non-production environments with `/chatops run auto_deploy status <merge-commit-of-your-feature>` - [ ] Enable the feature globally on non-production environments with `/chatops run feature set duo_add_otel true --dev --pre --staging --staging-ref` - [ ] Verify that the feature works as expected. ### Global rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted (with the command results) to the responsible team's Slack channel. - Ensure that the feature MRs have been deployed to both production and canary with `/chatops run auto_deploy status <merge-commit-of-your-feature>` - [ ] [Incrementally roll out](https://docs.gitlab.com/development/feature_flags/controls/#process) the feature on production. - Example: `/chatops run feature set duo_add_otel <rollout-percentage> --actors`. - Between every step wait for at least 15 minutes and monitor the appropriate graphs on https://dashboards.gitlab.net. - [ ] After the feature has been 100% enabled, wait for [at least one day before releasing the feature](#release-the-feature). ### Release the feature - [ ] Create a merge request to remove the `duo_add_otel` feature flag. - [ ] Ensure that the cleanup MR has been included in the release package. - [ ] Once the cleanup MR has been deployed to production, clean up the feature flag from all environments by running these chatops command in `#production` channel: `/chatops run feature delete duo_add_otel --dev --pre --staging --staging-ref --production` - [ ] Close this rollout issue. ## Rollback Steps - [ ] This feature can be disabled on production by running the following Chatops command: ``` /chatops run feature set duo_add_otel false ``` - [ ] Disable the feature flag on non-production environments: ``` /chatops run feature set duo_add_otel false --dev --pre --staging --staging-ref ```