Answer an unknown token on the MCP endpoints with invalid_token

What does this MR do and why?

On the MCP endpoints, /api/v4/mcp and /api/v4/orbit/mcp, an unknown token gets the 401 of a request without one: no error, and {"message":"401 Unauthorized"}. !261813 (merged) gave these endpoints their challenge with error="invalid_token" for an expired or revoked token, which got invalid_token with the OAuth realm before. RFC 6750 section 3.1 defines it for any token that is not valid.

The finders raise UnauthorizedError for an unknown token (auth_finders.rb#L439), and initial_current_user answers that with unauthorized! before the guard can. API::Helpers::Mcp::AuthChallenge, which only the MCP endpoints include, now overrides find_current_user! to raise TokenNotFoundError instead when a token was sent. The guard answers it with invalid_token, "Bad Access Token." and the endpoint's challenge. A token with no user or an invalid composite identity gets that too; a request without a token keeps its 401. The REST API authentication page names the unknown token too.

It is not in unauthorized!, the issue's proposal, because /api/v4/mcp also calls that for a valid mcp token in PRIVATE-TOKEN; a new example pins it.

The review of !261813 (merged) asked for this. It comes from my work on gitlab-mcp-server, whose upstream-bugs.md records its work on dependencies and sibling projects.

Checks I ran

All on master at b7c44b3b2a1, with Ruby 3.3.11, the second version .tool-versions lists, rather than the 3.4.10 of .ruby-version:

• The new examples with master's lib/api/helpers/mcp/auth_challenge.rb: in the request specs, 4 examples, 3 failures, the two unknown-token examples of /api/v4/mcp and the Orbit one, each on the body {"message"=>"401 Unauthorized"} and the challenge without error; in spec/lib/api/helpers/mcp/auth_challenge_spec.rb, 9 examples, 1 failure, the find_current_user! example for a request with a token. The examples for a valid token in PRIVATE-TOKEN and for a request without a token pass there by design.
• With the change made in unauthorized! instead, the example for the valid token in PRIVATE-TOKEN fails, since that request then gets error="invalid_token". Without the check that a token is present, the find_current_user! example for a request without a token fails.
• On the branch, spec/requests/api/mcp/base_spec.rb, ee/spec/requests/api/orbit/mcp_spec.rb and spec/requests/api/api_guard_spec.rb: 170 examples, 0 failures, 8 pending (skips inside the granular token shared examples). auth_challenge_spec.rb under fast_spec_helper: 9 examples, 0 failures.
• RuboCop on the four Ruby files: no offenses. Both commit message linters: no problems. markdownlint-cli2 0.23.2 and Vale 3.21.0 on the page: no issues, and the same Vale suggestions as on master.

References

Related to #634370 (closed)

The issue also reports other endpoints, such as GET /api/v4/user; this MR leaves them out.

Follow-up for them: #637638

Screenshots or screen recordings

Not a UI change.

How to set up and validate locally

1. Allow access to the MCP server.

2. Send an MCP request with an unknown token:

   curl -i -H "Authorization: Bearer glpat-notarealtokenatall0000" -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"initialize","id":1,"params":{"protocolVersion":"2025-06-18"}}' "http://gdk.test:3000/api/v4/mcp"

   The 401 carries error="invalid_token" in its challenge and body.

3. Without the Authorization header, the 401 is unchanged.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.