[FF] `security_inventory_stale_analyzer_status` -- Stale analyzer status in Security Inventory (#597929) · Issues · GitLab.org / GitLab

[FF] `security_inventory_stale_analyzer_status` -- Stale analyzer status in Security Inventory

WIP Details to be clarified ## Summary This issue is to roll out the feature on production, that is currently behind the `security_inventory_stale_analyzer_status` feature flag. Introduces a **Stale** analyzer status in Security Inventory UI, and takes the status into configuration when calculating `not_configured` ## Owners - Most appropriate Slack channel to reach out to: `#g_security_platform_management` - Best individual to reach out to: @vpedak1 ## Expectations ### What are we expecting to happen?  * Security Inventory shows * a **Stale** status pill for analyzers that have been absent from the last 3 consecutive default-branch pipelines. * **Stale** status in the project coverage popup * Group-level rollup of `not_configured` and `stale` is incremented/decremented correctly as project statuses transition in and out of stale. * No change to behavior for analyzers that run normally or for settings-owned types (`secret_detection`, `container_scanning`). ### What can go wrong and how would we detect it? * **Worker failures / performance degradation**: * `PipelineAnalyzersStatusUpdateWorker` errors would leave stale counts stale indefinitely (no scheduled per-project recalculation exists) - [Dashboard](https://dashboards.gitlab.net/d/sidekiq-worker-detail/sidekiq3a-worker-detail?var-worker=Security::PipelineAnalyzersStatusUpdateWorker&orgId=1&from=now-6h%2Fm&to=now%2Fm&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main) * `RecalculateWorker` - [Dashboard](https://dashboards.gitlab.net/d/sidekiq-worker-detail/sidekiq3a-worker-detail?var-worker=Security::AnalyzerNamespaceStatuses::RecalculateWorker&orgId=1&from=now-6h%2Fm&to=now%2Fm&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main) * `AdjustmentWorker` - [Dashboard](https://dashboards.gitlab.net/d/sidekiq-worker-detail/sidekiq3a-worker-detail?var-worker=Security::AnalyzerNamespaceStatuses::AdjustmentWorker&orgId=1&from=now-30d&to=now&timezone=utc&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&var-stage=main) * **In case of any UI issues** * Roll back the feature flag * **In case of analyzer coverage counts drift, incorrect values:** * Revert the FF * Revert the code changes * Run AdjustmentWorker   ## Rollout Steps Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command. ### Rollout on non-production environments * [ ] Verify the MR with the feature flag is merged to `master` and has been deployed to non-production environments with `/chatops gitlab run auto_deploy status <merge-commit-of-your-feature>` * [ ] Deploy the feature flag at a percentage (recommended percentage: 50%) with `/chatops gitlab run feature set security_inventory_stale_analyzer_status 50 --actors --dev --pre --staging --staging-ref` * [ ] Monitor that the error rates did not increase (repeat with a different percentage as necessary). * [ ] Enable the feature globally on non-production environments with `/chatops gitlab run feature set security_inventory_stale_analyzer_status true --dev --pre --staging --staging-ref` * [ ] Verify that the feature works as expected. The best environment to validate the feature in is [`staging-canary`](https://about.gitlab.com/handbook/engineering/infrastructure/environments/#staging-canary) as this is the first environment deployed to. Make sure you are [configured to use canary](https://next.gitlab.com/). * [ ] If the feature flag causes end-to-end tests to fail, disable the feature flag on staging to avoid blocking [deployments](https://about.gitlab.com/handbook/engineering/deployments-and-releases/deployments/). - See [`#e2e-run-staging` Slack channel](https://gitlab.enterprise.slack.com/archives/CBS3YKMGD) and look for the following messages: - test kicked off: `Feature flag security_inventory_stale_analyzer_status has been set to true on **gstg**` - test result: `This pipeline was triggered due to toggling of security_inventory_stale_analyzer_status feature flag` If you encounter end-to-end test failures and are unable to diagnose them, you may reach out to the [`#s_developer_experience` Slack channel](https://gitlab.enterprise.slack.com/archives/C07TWBRER7H) for assistance. Note that end-to-end test failures on `staging-ref` [don't block deployments](https://about.gitlab.com/handbook/engineering/infrastructure/environments/staging-ref/#how-to-use-staging-ref). ### Before production rollout - [ ] ~~If the change is significant and you wanted to announce in ~~[~~#whats-happening-at-gitlab~~](https://gitlab.enterprise.slack.com/archives/C0259241C)~~, it best to do it before rollout to `gitlab-org/gitlab-com`.~~ ### Specific rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted (with the command results) to the responsible team's Slack channel. - [ ] Ensure that the feature MRs have been deployed to both production and canary with `/chatops gitlab run auto_deploy status <merge-commit-of-your-feature>` - [ ] Enable for internal users first (group actor): * `/chatops gitlab run feature set --group=gitlab-org,gitlab-com security_inventory_stale_analyzer_status true` - [ ] Verify that the feature works for the specific actors (check Security Inventory for `gitlab-org` and `gitlab-com` groups). ### Preparation before global rollout - [ ] Set a milestone to this rollout issue to signal for enabling and removing the feature flag when it is stable. - [x] ~~Check if the feature flag change needs to be accompanied with a ~~[~~change management issue~~](https://about.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process)~~. Cross link the issue here if it does~~. Change management not needed. - [ ] Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production. If a different developer will be covering, or an exception is needed, please inform the oncall SRE by using the `@sre-oncall` Slack alias. - [ ] Ensure that documentation exists for the feature, and the [version history text](https://docs.gitlab.com/development/documentation/feature_flags/#add-history-text) has been updated. - [x] ~~Ensure that any breaking changes have been announced following the ~~[~~release post process~~](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations-removals-and-breaking-changes)~~ to ensure GitLab customers are aware.~~ No breaking changes - [ ] Notify the [`#support_gitlab-com` Slack channel](https://gitlab.slack.com/archives/C4XFU81LG) and your team channel ([more guidance when this is necessary in the dev docs](https://docs.gitlab.com/development/feature_flags/controls/#communicate-the-change)). - [x] ~~If this flag is or may be queried by external API consumers (for example, IDE extensions, Duo CLI, or CI integrations), follow the ~~[~~external API consumer guidance~~](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers)~~ and ensure a fail-open mechanism is in place before the rollout milestone is finalised.~~ N/A the flag is not queried. ### Global rollout on production For visibility, all `/chatops` commands that target production must be executed in the [`#production` Slack channel](https://gitlab.slack.com/archives/C101F3796) and cross-posted (with the command results) to the responsible team's Slack channel. - [ ] [Incrementally roll out](https://docs.gitlab.com/development/feature_flags/controls/#process) the feature on production. ( Between every step wait for at least 15 minutes and monitor the appropriate graphs on https://dashboards.gitlab.net ) * [ ] `/chatops gitlab run feature set security_inventory_stale_analyzer_status 25 --actors` * [ ] `/chatops gitlab run feature set security_inventory_stale_analyzer_status 50 --actors` * [ ] `/chatops gitlab run feature set security_inventory_stale_analyzer_status 100 --actors` - [ ] After the feature has been 100% enabled, wait for [at least one day before releasing the feature](#release-the-feature). ### Release the feature After the feature has been [deemed stable](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#including-a-feature-behind-feature-flag-in-the-final-release), the [clean up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up) should be done as soon as possible to permanently enable the feature and reduce complexity in the codebase. You can either [create a follow-up issue for Feature Flag Cleanup](https://gitlab.com/gitlab-org/gitlab/-/issues/new?description_template=Feature%20Flag%20Cleanup) or use the checklist below in this same issue.  - [ ] Create a merge request to remove the `security_inventory_stale_analyzer_status` feature flag. The MR should include the following changes: - Remove all references to the feature flag from the codebase. - Remove `ee/config/feature_flags/wip/security_inventory_stale_analyzer_status.yml` - Make `AnalyzerNamespaceStatus#not_configured` unconditionally account for `stale`. - Remove conditional GraphQL field exposure for `stale` on `AnalyzerNamespaceStatus`. - [ ] Ensure that the cleanup MR has been included in the release package. If the merge request was deployed before [the monthly release was tagged](https://about.gitlab.com/handbook/engineering/releases/#self-managed-releases-1), the feature can be officially announced in a release blog post: `/chatops gitlab run release check <merge-request-url> <milestone>` - [ ] Close the feature issue to indicate the feature will be released in the current milestone. - [ ] Once the cleanup MR has been deployed, clean up the feature flag from all environments: `/chatops gitlab run feature delete security_inventory_stale_analyzer_status --dev --pre --staging --staging-ref --production` - [ ] Close this rollout issue. ## Rollback Steps - [ ] This feature can be disabled on production by running the following Chatops command: ``` /chatops gitlab run feature set security_inventory_stale_analyzer_status false ``` - [ ] Disable the feature flag on non-production environments: ``` /chatops gitlab run feature set security_inventory_stale_analyzer_status false --dev --pre --staging --staging-ref ``` - [ ] Delete feature flag from all environments: ``` /chatops gitlab run feature delete security_inventory_stale_analyzer_status --dev --pre --staging --staging-ref --production ```

issue