## Problem An issue was raised in https://gitlab.com/gitlab-org/gitlab/-/issues/348496#note_856625881 that we saw causing a small number of 500 errors -- which appear to be triggered by some underlying issue inside GitLab Experiment. The exceptions raised can be seen in https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+free_indicator We believe this has something to do with the caching mechanism in GitLab Experiment that seems to be only affecting us when using `variant` in the [Experiment class](http://gitlab.com/gitlab-org/gitlab/blob/faea16cc89460f5d55a6148a81f05e1cb44f3929/app/experiments/new_project_sast_enabled_experiment.rb#L3-3) and/or when using [multiple variants](http://gitlab.com/gitlab-org/gitlab/blob/faea16cc89460f5d55a6148a81f05e1cb44f3929/app/views/projects/_new_project_fields.html.haml#L67-67). We are currently only seeing errors that point back to the [new_project_sast_enabled experiment](https://gitlab.com/gitlab-org/gitlab/-/issues/340929) which has both of the conditions above. ## Immediate actions We will take each action only as needed, executing the first action and if our issue is resolved, not move onto the next as they increase in impact/effort. - [x] Disable `new_project_sast_enabled` and clear experiment cache for all experiments on `.com`. @jejacks0n - impact: only `new_project_sast_enabled` experiment as it will turn it off and send it down the `control` path. - [x] Execute a cleanup/promotion of `new_project_sast_enabled` experiment via MR. @jejacks0n to prep and anyone to help it through review if he is offline. - impact: only `new_project_sast_enabled` as it will remove the experiment and promote the 'winner' to production. - [ ] Turn off all experimentation on `.com` via making [this method](http://gitlab.com/gitlab-org/gitlab/blob/faea16cc89460f5d55a6148a81f05e1cb44f3929/lib/gitlab/experiment/rollout/feature.rb#L14-14) return `false`. - [x] Circuit breaker to be created to enable ability to turn off GLEX experiments - @dstull https://gitlab.com/gitlab-org/gitlab/-/merge_requests/81834 - [x] Document the circuit breaker @dstull https://gitlab.com/gitlab-org/gitlab/-/issues/354107 - impact: all experiments running on `.com` will be disabled, sending everything down the `control` paths. ## Long term action Figure out why this is happening and fix it.