Answer refused tokens on the MCP endpoints with their challenge

What does this MR do and why?

The MCP endpoints, /api/v4/mcp and /api/v4/orbit/mcp, answer a request with no token with a 401 whose challenge API::Helpers::Mcp::AuthChallenge builds: realm="GitLab", the scope an OAuth client requests for the endpoint (mcp or mcp_orbit), and resource_metadata, which is how an MCP client finds the authorization server. The other refusals of a token on these endpoints said something else. An expired or revoked token, or an impersonation token while impersonation is disabled, got the guard's invalid_token challenge, with realm="Protected by OAuth 2.0" and no resource_metadata. The insufficient_scope 403, which carries a challenge since !260668 (merged), named every scope the guard accepts for the request: mcp api for a POST to /api/v4/mcp and mcp_orbit mcp read_api ai_workflows api on the Orbit endpoint. And on /api/v4/mcp, a token that has api and not mcp passed the guard, because API::API registers api for every route (and read_api for a GET, so a read_api token's GET went the same way), and was refused afterwards with forbidden!: {"message":"403 Forbidden"} and no challenge at all.

The MCP authorization specification asks for that 403 to carry error="insufficient_scope", the scopes needed and resource_metadata, "for consistency with 401 responses", and its step-up flow has the client request the scopes the challenge names. A client told mcp api could request api, which /api/v4/mcp then refuses with a 403 that names nothing.

This MR makes the invalid_token 401 and the insufficient_scope 403 on these endpoints carry the challenge of their 401, with error and error_description added:

WWW-Authenticate: Bearer realm="GitLab", error="insufficient_scope", error_description="The request requires higher privileges than provided by the access token.", scope="mcp", resource_metadata="https://gitlab.example.com/.well-known/oauth-protected-resource/api/v4/mcp"

The guard's error handler runs in the endpoint (rescue_from with a proc runs it through instance_exec), so it can ask the endpoint. APIGuard::HelperMethods gets two private hooks whose defaults change nothing: bearer_challenge_scopes, the scope list the 403 names in its body and its challenge, and bearer_challenge(error:, error_description:, scope: nil), the challenge for an error RFC 6750 defines, or nil to keep the guard's. AuthChallenge overrides bearer_challenge and API::Mcp::Base overrides bearer_challenge_scopes, so only the two MCP endpoints change, and every other endpoint answers byte for byte as before. The DPoP and language server 401s, whose error codes are not RFC 6750's, and the insufficient_granular_scope 403 keep what they have.

On /api/v4/mcp the 403 names mcp alone when the guard's list holds it, because the endpoint's own scope check refuses the other scopes in that list, api included: naming api there sends a client to a token that is refused again. mcp is also what the 401 there names, and the scope Oauth::AuthorizationsController#pre_auth_params sets on an authorization request for that resource and for a dynamically registered client limited to it. A list that asks for another scope is kept: sudo! raises InsufficientScopeError with sudo, and naming mcp there would tell an administrator to request the scope the token already has. The Orbit endpoint, which the issue left open, keeps the guard's list, mcp_orbit mcp read_api ai_workflows api: every scope in it authorizes the request there, and a project or group access token, which can't get mcp_orbit, can have read_api or api. The list already starts with mcp_orbit, and an OAuth step-up still ends with mcp_orbit, which pre_auth_params sets for that resource whatever the client asks for. The body's scope comes from the same list, so the body and the challenge agree, as they do after !260668 (merged).

On /api/v4/mcp, the endpoint's own scope check now raises Gitlab::Auth::InsufficientScopeError with mcp instead of calling forbidden!, so a token with api and not mcp (or, on a GET, read_api) gets the same 403 from the same handler, and so does a request signed in through the session. The status is 403 as before. Like the guard, it first records the refusal of a token in the request's log context, so the API log shows insufficient_scope as meta.auth_fail_reason and mcp as meta.auth_fail_requested_scopes, which forbidden! did not. A request signed in through the session has no token, so nothing is recorded for it.

The commits:

1. Answer refused tokens on the MCP endpoints with their challenge: the two hooks, the overrides in AuthChallenge and API::Mcp::Base, and specs for both endpoints, for the guard with an endpoint that defines the hooks, and for the helper. It carries the Changelog: fixed trailer and describes the main change, so it is the right message if the MR is squashed.
2. Refuse an MCP token without the mcp scope as insufficient_scope: the raise in lib/api/mcp/base.rb, the log context before it, and their specs.
3. Document the challenge of the MCP endpoints' 403: doc/api/rest/authentication.md, which describes the guard's challenge, now says what the MCP endpoints send instead, and the GitLab MCP server page says what its 403 Forbidden carries.

This comes out of my work on gitlab-mcp-server, an MCP server for GitLab. The 403 without a challenge was the first row of the record it keeps of everything done on its dependencies and on sibling projects, and this MR is the follow-up agreed in the review of !260668 (merged).

References

Closes #633417 (closed)

Builds on !260668 (merged), which gave the insufficient_scope 403 a challenge, and on !261131 (merged), which added the Authentication section of the GitLab MCP server page.

Screenshots or screen recordings

Not a UI change. What the endpoints answer, from the request specs of this MR run against master and against this branch (the WWW-Authenticate header, with the long error_description and URL shortened):

Request	Before	After
/api/v4/mcp, expired or revoked token, or impersonation token while impersonation is disabled	401, Bearer realm="Protected by OAuth 2.0", error="invalid_token", error_description="..."	401, Bearer realm="GitLab", error="invalid_token", error_description="...", scope="mcp", resource_metadata="..."
/api/v4/mcp, token with only read_user	403, Bearer realm="Protected by OAuth 2.0", error="insufficient_scope", ..., scope="mcp api", body scope mcp api	403, Bearer realm="GitLab", error="insufficient_scope", ..., scope="mcp", resource_metadata="...", body scope mcp
/api/v4/mcp, token with api and not mcp	403, {"message":"403 Forbidden"}, no header	403, insufficient_scope body and challenge as above
/api/v4/mcp, administrator token with mcp and not sudo, Sudo header	403, guard realm, scope="sudo"	403, realm="GitLab", scope="sudo", resource_metadata
/api/v4/orbit/mcp, token with only read_user	403, guard realm, scope="mcp_orbit mcp read_api ai_workflows api"	403, realm="GitLab", the same scope, resource_metadata
/api/v4/orbit/mcp, revoked token	401, guard realm, invalid_token	401, realm="GitLab", invalid_token, scope="mcp_orbit", resource_metadata
Either endpoint, no token or an unknown one	401 with the endpoint's challenge	unchanged
Any other endpoint	the guard's challenges	unchanged

Checks I ran

All on top of master at f235ea85be5, which holds !260668 (merged) and !261131 (merged), in the test environment, with bundle exec rspec over the four spec files the MR changes (spec/lib/api/helpers/mcp/auth_challenge_spec.rb, spec/requests/api/api_guard_spec.rb, spec/requests/api/mcp/base_spec.rb and ee/spec/requests/api/orbit/mcp_spec.rb):

• The branch's specs with the three lib files from master: 169 examples, 15 failures, every one of them an example this MR adds: the three helper examples, the guard's invalid_token and insufficient_scope examples with an endpoint that defines the hooks, the eight /api/v4/mcp examples and the two Orbit ones. The endpoint examples fail on the guard's challenge (Bearer realm="Protected by OAuth 2.0", ..., scope="mcp api"), on scope mcp api in the body, and, for the token with api and not mcp and for the request signed in through the session, on a body with no error at all, and for that token on a log line with no meta.auth_fail_* fields.
• The branch's specs with lib/api/mcp/base.rb from the first commit: 169 examples, 3 failures, the two examples for a token with api and not mcp (its challenge and its log line) and the one for a request signed in through the session, which the second commit makes pass.
• The first commit with lib/api/mcp/base.rb from master, so /api/v4/mcp has no bearer_challenge_scopes of its own: 166 examples, 1 failure, the read_user example, which then gets mcp api.
• The first commit alone: 166 examples, 0 failures. The whole branch: 169 examples, 0 failures. Each run has 8 pending examples, skips inside the granular token shared examples (namespace has no top-level group, only meaningful on Project/Group boundaries). The helper spec also passes under fast_spec_helper alone: 7 examples, 0 failures.
• The rest of the MCP request specs (spec/requests/api/mcp, ee/spec/requests/api/mcp, spec/requests/api/api_spec.rb, ee/spec/requests/api/orbit/data_spec.rb, ee/spec/requests/api/internal/orbit_redaction_spec.rb) on the branch: 599 examples, 22 failures, the same 22 that fail with the lib files from master in my environment, in call_tool_spec.rb and scope_route_restriction_spec.rb: 20 on a NoMethodError raised from Gitlab::Graphql::Subscriptions::ActionCableWithLoadBalancing in my local setup, and the two save_pipeline examples, whose tool call answers with an error there.
• RuboCop on the Ruby files of the branch: no offenses. Both commit message linters (scripts/lint/commit_linter.rb and the Danger one) on each commit: no problems. git diff --check: clean. markdownlint-cli2 and Vale, with the image the docs lint jobs use, on the two pages: no markdownlint issues, and the same Vale suggestions as on master.

How to set up and validate locally

1. Allow access to the MCP server for the instance.

2. Create a personal access token with only the api scope, and another with only read_user.

3. Send an MCP initialize request to /api/v4/mcp with each:

   curl --include --request POST --header "Authorization: Bearer <token>" \
     --header "Content-Type: application/json" \
     --data '{"jsonrpc":"2.0","method":"initialize","id":1,"params":{"protocolVersion":"2025-06-18"}}' \
     --url "http://gdk.test:3000/api/v4/mcp"

   Both answer 403 with "error":"insufficient_scope" and "scope":"mcp" in the body, and a WWW-Authenticate header with realm="GitLab", scope="mcp" and resource_metadata. Before this MR, the api token got {"message":"403 Forbidden"} with no header, and the read_user token got the guard's challenge with scope="mcp api".

4. Revoke the api token and send the request again: the 401 now carries the same challenge with error="invalid_token".

5. Send the read_user request to /api/v4/orbit/mcp on an instance that serves it: the challenge names mcp_orbit mcp read_api ai_workflows api and the Orbit endpoint's metadata URL.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.