Add WWW-Authenticate challenge to MCP 401 responses

What does this MR do and why?

Unauthenticated requests to POST /api/v4/mcp returned 401 Unauthorized without a WWW-Authenticate response header. The MCP authorization spec (2025-06-18+) and RFC 9728 §5.1 require this header so MCP clients (MCPJam Inspector, gemini-cli, Claude Desktop via mcp-remote) can discover the OAuth protected-resource metadata URL directly from the 401 challenge. Without it, each client must add GitLab-specific fallback logic or is blocked outright.

GitLab already serves the metadata at /.well-known/oauth-protected-resource/api/v4/mcp; this MR makes the 401 advertise it.

How

Introduces a shared API::Helpers::Mcp::AuthChallenge helper that overrides unauthorized! to emit:

WWW-Authenticate: Bearer realm="GitLab", resource_metadata="<gitlab_url>/.well-known/oauth-protected-resource/api/v4/mcp"

This mirrors the existing packages BasicAuthHelpers pattern and is wired into API::Mcp::Base.

The 403 insufficient-scope path (see the existing FIXME in lib/api/api_guard.rb) is a separate defect and out of scope here.

A follow-up MR wires the same helper into the Orbit MCP endpoint (/api/v4/orbit/mcp).

References

• Resolves #597976 (closed)

Manual verification (GDK)

The 401 is raised before any feature-flag/licence check (authenticate! runs first in the before block), so an unauthenticated request is enough — no MCP feature setup is required. $GDK is your GDK base URL (e.g. http://gdk.test:3000).

Reproduce the bug (on master, before this MR):

curl -si -X POST "$GDK/api/v4/mcp" \
  -H 'Content-Type: application/json' \
  -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"probe","version":"1"}},"id":1}' \
  | grep -iE '^HTTP/|^www-authenticate'

Output on master — 401 with no WWW-Authenticate header:

HTTP/1.1 401 Unauthorized

With this MR the same request returns the RFC 9728 challenge:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="GitLab", resource_metadata="$GDK/.well-known/oauth-protected-resource/api/v4/mcp"

Confirm the advertised metadata URL resolves (already served on master):

curl -s "$GDK/.well-known/oauth-protected-resource/api/v4/mcp" | jq

Returns the resource metadata JSON (resource, authorization_servers, scopes_supported).

Automated coverage: the request specs assert the exact header value on both the POST and GET unauthenticated paths.

RFC 9728 §5.1 compliance

Per RFC 9728 §5.1, a 401 from a protected resource advertises the metadata location via the resource_metadata parameter in WWW-Authenticate. This MR emits:

WWW-Authenticate: Bearer realm="GitLab", resource_metadata="<gitlab_url>/.well-known/oauth-protected-resource/api/v4/mcp"

• resource_metadata — quoted URL of the protected-resource metadata ✅ (the parameter introduced by §5.1)
• realm — optional per §5.1 (governed by RFC 6750); included for clarity ✅
• Metadata URL uses the §3 / §3.1 path-insertion form — the well-known segment inserted between host and resource path ✅ — matching the route already registered in config/routes.rb.

Subpath (relative URL root) install

For an install on a relative URL root (e.g. external_url 'https://git.example.org/gitlab'), request.path includes the root (/gitlab/api/v4/mcp) and Gitlab.config.gitlab.url already contains it, so the helper strips the root to avoid duplicating it. Expected header:

WWW-Authenticate: Bearer realm="GitLab", resource_metadata="https://git.example.org/gitlab/.well-known/oauth-protected-resource/api/v4/mcp"

The /gitlab prefix must appear exactly once. Covered by the unit spec spec/lib/api/helpers/mcp/auth_challenge_spec.rb.

Known limitation (follow-up)

This MR adds the resource_metadata challenge to unauthenticated (no-credentials) 401 responses — the primary OAuth discovery flow. Requests with an invalid/expired/revoked bearer token are handled earlier by the shared APIGuard bearer error handler (lib/api/api_guard.rb:190) and return WWW-Authenticate: Bearer error="invalid_token" without resource_metadata. Extending that (shared) handler was kept out of scope here; tracked in #604654 (closed) (which also overlaps the existing InsufficientScopeError FIXME in api_guard.rb).