[FF] orbit_query_in_rails -- Run Orbit queries in Rails instead of Workhorse

Summary

Roll out orbit_query_in_rails (gitlab!261318). With the flag on, Rails runs the Orbit query stream and redaction in one request, so Workhorse is not involved. Response bodies do not change.

The flag actor is the user, so gate with --user, --feature-group, or --actors.

What to watch

  • Each Orbit query holds an api Puma thread for 1 to 5 seconds until max_io_threads is set. Peak traffic is about 18 queries in flight across the api fleet.
  • api service overview, and 5xx on /api/v4/orbit/*.
  • Enabled users make zero /api/v4/internal/orbit/redaction calls.

Rollout

Run production commands in #production and post results to #f_orbit.

/chatops gitlab run feature set orbit_query_in_rails true --staging
/chatops gitlab run feature set --user=michaelangeloio orbit_query_in_rails true
/chatops gitlab run feature set --feature-group=gitlab_team_members orbit_query_in_rails true
/chatops gitlab run feature set orbit_query_in_rails 10 --actors
/chatops gitlab run feature set orbit_query_in_rails 50 --actors
/chatops gitlab run feature set orbit_query_in_rails true

Rollback

/chatops gitlab run feature set orbit_query_in_rails false

Cleanup

Remove the flag, the Workhorse Orbit path, and /api/v4/internal/orbit/redaction. Then:

/chatops gitlab run feature delete orbit_query_in_rails --staging --production