Job router: only a forbidden runner is unhealthy

What does this MR do?

On the job router path, a failed job request now only marks the runner unhealthy when the router returns PermissionDenied (403), the same as the direct HTTP path. Every other request error (409, 422, 429, 401, 5xx as mapped by KAS) keeps the runner healthy and polling.

Why was this MR needed?

handleRouterError treated every non-transport gRPC error as unhealthy. KAS maps a 409 from Rails to FailedPrecondition. Rails returns 409 when another runner picked the build first, which is routine at scale, and the direct path has always treated it as "poll again". Through the router it counted towards unhealthy_requests_limit (default 3), so a burst of three 409s turned the manager off for unhealthy_interval (1h):

Error requesting a job  rpc error: code = FailedPrecondition desc = HTTP status code: 409 for path /api/v4/internal/ci/job_router/jobs/request with reason 409 Conflict
Runner "https://us-east1.ci-gateway.int.gprd.gitlab.net:8989/KVJMJq9V9" is unhealthy and will be disabled for 1h0m0s seconds!

On the gitlab.com private shards, which run with FF_USE_JOB_ROUTER, 24 of 27 p-s managers were disabled at least once in a 6 hour window today, some four times. Each is a manager taking no jobs for an hour.

What's the best way to test this MR?

TestRequestJob_RequestErrorHealth fails on main for FailedPrecondition, ResourceExhausted, Unauthenticated, Internal and Unknown, and passes with the change. PermissionDenied stays unhealthy.

What are the relevant issue numbers?

Found while rolling out gitlab-com/gl-infra/production-engineering#29652 (closed).