Run Orbit query streams in Rails instead of Workhorse

Problem

Orbit queries run through Workhorse (#349 (closed)). Every change to the query path needs Go code and Workhorse maintainer review, and redaction needs a second API request (/api/v4/internal/orbit/redaction). That slows Orbit work, including the planned api/v4/orbit/context endpoint (kg!2677).

Proposal

Run the Orbit query stream and redaction in Rails, behind orbit_query_in_rails. gitlab!261318 shipped this. Production results for one user so far:

  • All surfaces work: REST, agent commands, MCP, and DAP chat.
  • Redaction round trip p50 154 ms to 61 ms, p90 683 ms to 93 ms.
  • No redaction callbacks.

Tasks

  • Roll out the flag: gitlab#633423
  • Set max_io_threads so Orbit queries do not hold regular Puma threads:
    • CNG: read PUMA_MAX_IO_THREADS in the webservice puma.rb
    • GitLab.com: k8s-workloads api extraEnv (gstg-cny, gstg, gprd-cny, gprd)
    • Dedicated: instrumentor webservice extraEnv
    • Self-managed: Omnibus puma['max_io_threads'] and docs
  • Build new query endpoints, such as context, on the Rails path
  • After rollout: remove the flag, workhorse/internal/orbit, send_orbit_query, and /api/v4/internal/orbit/redaction