`approval_settings` property is appearing in policies in production

Summary

Context

• The project Prevent branch modification when a policy disab... (&9705 - closed) aims to add additional yaml to the scan result policies in the form of approval_settings
• The "scan_result_policy_settings" ff controls the frontend for the policy settings
• The "scan_result_policies_block_unprotecting_branches" ff controls the backend (yes, we should have only used one; it is being fixed)

Issue

• Grant has seen policies with the additional setting in policies in two projects where neither feature flag was on in. The policies were created as far back as Sept 14th and some as recent as 4 days ago. How is this possible?

Steps to reproduce

1. Navigate to a project => Secure => Policies => New policy => Scan Result Policy
2. Select a policy type

Example Project

• https://gitlab.com/gitlab-org/gl-security/license-compliance-dogfooding-trial
• https://gitlab.com/haven-group/security-group/security-policy-project-management

What is the current bug behavior?

The following yaml is added to the policy even when the feature flag is turned off

approval_settings:
  block_protected_branch_modification:
    enabled: true

What is the expected correct behavior?

No approval_settings yaml should be added to the policy

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)

Implemenation Plan

• frontend prevent approval_settings from being added without the feature flags (Fix issue where settings showed up without feat... (!134471 - merged))
• frontend check for property in the yaml on load and remove it if it exists
  • This solves the two requests from product below. Luckily the incorrect yaml that was added was block_protected_branch_modification: { enabled: true }, which has been replaced with block_unprotecting_branches: true, so it will be easy to detect
  • frontend for policies that have been impacted, ensure the setting is not enabled when the feature flag is turned on
  • frontend ensures the policy editor is not disabled when this setting is in the yaml (Allow permitted invalid approval settings for p... (!134622 - merged))

Obsolete additional info

Theories that have been proven false

• The "scan_result_policy_settings" ff was accidentally turned on for these project in production
  • This cannot have happened because the rollout issue for the feature flag does not show that the feature flag was ever on production and global searching slack for the feature flag shows that no one ever ran the command
• The "scan_result_policies_block_unprotecting_branches" ff was accidentally turned on for these projects in production
  • This cannot have happened because the rollout issue for the feature flag does not show that the feature flag was ever on production and global searching slack for the feature flag shows that no one ever ran the command
• The values were added via yaml
  • Grant says he did not do that

Potential Explanations (needs investigating)

• The frontend accidentally let the additional value be added via a bug that did not hide the setting
  • This could be tracked via searching the history of the frontend code and reviewing it at various stages
  • The policies with the values span a large time frame (1mo -> 4days ago), so for it not to be caught in that timeframe seems unlikely
  • Though perhaps it is a specific action that adds the value (for example: maybe new policies would not have the value, but there could be a bug where editing a policy adds the value)