Failure in ee/api/3_create/code_suggestions_spec.rb | ModelOps Code Suggestions on the GitLab API with PAT auth returns a suggestion

Test metadata (don't modify)

Field	Value
File	`qa/qa/specs/features/ee/api/14_model_ops/code_suggestions_spec.rb#L40`
Description	`ModelOps Code Suggestions on the GitLab API with PAT auth returns a suggestion`
Test level
Hash	`6d6e335edb9f2d6e74d6cf84b9398a62e689f254867bf0ad6fc4cd1bc8cdcdf0`
Duration	1.76 seconds
Expected duration	< 240 seconds
Test case	https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/420973

Stack trace

Failure/Error: expect(response.code).to be(200), "Request returned (#{response.code}): `#{response}`"
  Request returned (401): `{"message":"401 Unauthorized"}`

Reports (11)

Timeline

2023-11-09 05:43:51 UTC test passed on staging in full pipeline https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/11833840#L61
2023-11-09 07:41 UTC gstg deploy started: https://ops.gitlab.net/gitlab-com/gl-infra/deployer/-/pipelines/2504421 - note that this is the deploy that includes the test as a smoke test, although it ran previously in the full suite.
2023-11-09 08:09 UTC deploy finished
2023-11-09 08:16:52 UTC first failure
The execution failure history issue shows that it passed at least a few times after that first failure.
2023-11-09 09:43:51 UTC example pass: https://ops.gitlab.net/gitlab-org/quality/staging/-/jobs/11836474#L208
2023-11-09 11:52:00 UTC Quarantine MR merged: !136438 (merged)
2023-11-09 13:43:00 UTC Nailia noted it passed when she ran it from her machine against Staging.
2023-11-10 02:00 UTC test passes on each environment in each pipeline: #431317 (comment 1642037321)

Data

2023-11-21: https://ops.gitlab.net/gitlab-org/quality/staging-canary/-/jobs/11982618 (https://ops.gitlab.net/gitlab-org/quality/staging-canary/-/pipelines/2551750)

Investigation summary

Rails rejected authentication without sending the request to the AI-gateway.
The API token has the api scope and hasn't changed.
No changes were made to the user profile or group settings.
The test was running successfully until I added it to the smoke test suite. No change was made to the test itself, or the pipelines that run the test.
There was a Ruby 3.1 test rollout around the time of the failures, but the tests passed many times in subsequent Ruby 3.1 test rollouts.
The test passed several hundreds of times when run every 30 minutes the same CI conditions that failed before (in staging, staging-canary, canary, and production pipelines).

I think it's reasonable to add the tests back to the smoke suite, given the inconclusive investigation above and the many successful executions since the failures.

Plus, fast quarantine works in those pipelines now so we can more easily unblock them if needed.

Edited Nov 28, 2023 by Mark Lapierre