[FF] `graphql_increased_validate_timeout` -- Increase GraphQL validation timeout to 0.5s

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Label this issue

Summary

This issue is to roll out the feature on production, that is currently behind the graphql_increased_validate_timeout feature flag.

Owners

Most appropriate Slack channel to reach out to: #g_api-platform
Best individual to reach out to: @abignotti

Expectations

What are we expecting to happen?

Large legitimate GraphQL queries that currently fail validation due to the 0.2s timeout will succeed with the increased 0.5s timeout. This is still 6x below the gem default of 3s, and validate_max_errors 5 remains as the primary DoS protection.

What can go wrong and how would we detect it?

Maliciously complex queries could take slightly longer to reject (up to 0.5s instead of 0.2s). This is mitigated by validate_max_errors 5 which caps the number of errors before aborting. Monitor GraphQL request latency on https://dashboards.gitlab.net.

Rollout Steps

Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command.

Rollout on non-production environments

Verify the MR with the feature flag is merged to master and has been deployed to non-production environments with /chatops gitlab run auto_deploy status <merge-commit-of-your-feature>
Deploy the feature flag at a percentage (recommended percentage: 50%) with /chatops gitlab run feature set graphql_increased_validate_timeout 50 --actors --dev --pre --staging --staging-ref
Monitor that the error rates did not increase (repeat with a different percentage as necessary).
Enable the feature globally on non-production environments with /chatops gitlab run feature set graphql_increased_validate_timeout true --dev --pre --staging --staging-ref
Verify that the feature works as expected. The best environment to validate the feature in is staging-canary as this is the first environment deployed to. Make sure you are configured to use canary.
If the feature flag causes end-to-end tests to fail, disable the feature flag on staging to avoid blocking deployments.
- See #e2e-run-staging Slack channel and look for the following messages:
  - test kicked off: Feature flag graphql_increased_validate_timeout has been set to true on **gstg**
  - test result: This pipeline was triggered due to toggling of graphql_increased_validate_timeout feature flag

If you encounter end-to-end test failures and are unable to diagnose them, you may reach out to the #s_developer_experience Slack channel for assistance. Note that end-to-end test failures on staging-ref don't block deployments.

Before production rollout

If the change is significant and you wanted to announce in #whats-happening-at-gitlab, it best to do it before rollout to gitlab-org/gitlab-com.

Specific rollout on production

For visibility, all /chatops commands that target production must be executed in the #production Slack channel and cross-posted (with the command results) to the responsible team's Slack channel.

Depending on the type of actor you are using, pick one of these options:
- For project-actor: /chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss,gitlab-com/www-gitlab-com graphql_increased_validate_timeout true
- For group-actor: /chatops gitlab run feature set --group=gitlab-org,gitlab-com graphql_increased_validate_timeout true
- For user-actor: /chatops gitlab run feature set --user=abignotti graphql_increased_validate_timeout true
- For all internal users: /chatops gitlab run feature set --feature-group=gitlab_team_members graphql_increased_validate_timeout true
Verify that the feature works for the specific actors.

Preparation before global rollout

Set a milestone to this rollout issue to signal for enabling and removing the feature flag when it is stable.
Check if the feature flag change needs to be accompanied with a change management issue. Cross link the issue here if it does.
Ensure that you or a representative in development can be available for at least 2 hours after feature flag updates in production. If a different developer will be covering, or an exception is needed, please inform the oncall SRE by using the @sre-oncall Slack alias.
Ensure that documentation exists for the feature, and the version history text has been updated.
Ensure that any breaking changes have been announced following the release post process to ensure GitLab customers are aware.
Notify the #support_gitlab-com Slack channel and your team channel (more guidance when this is necessary in the dev docs).
If this flag is or may be queried by external API consumers (for example, IDE extensions, Duo CLI, or CI integrations), follow the external API consumer guidance and ensure a fail-open mechanism is in place before the rollout milestone is finalised.

Global rollout on production