Validate Duo Workflow client events against the run state
What does this MR do and why?
Workhorse checks every Duo Workflow client event against the state of the run before forwarding it to the Duo Workflow Service. This is option 2 of the plan in the issue.
An event is invalid if it is:
- any event other than a start request sent before the start request (
before_start) - an
actionResponsewithout a request ID (missing_request_id) - an
actionResponsewhose request ID Workhorse didn't write to this client, or that was already answered (unknown_request_id)
Workhorse tracks the request IDs of the client-bound actions it writes in actionDispatcher.writeToClient. It skips newCheckpoint, which needs no answer (it matches ACTION_TYPES_WITHOUT_RESPONSES in the DWS outbox). An action Workhorse answers itself because the transport can't run it is removed again. Tracking always runs, so log-only mode shows what enforcement would reject.
Rails sends the new RejectInvalidClientEvents setting in the DuoWorkflow pre-authorize config. It is controlled by the duo_workflow_reject_invalid_client_events flag (gitlab_com_derisk, default off).
- Flag off (default): the event is forwarded as before, and each reason is logged once per connection with the event type and mode. The log has no payload and no client-chosen request ID.
- Flag on: the event is not forwarded. The teardown reason is
client_invalid_event(visible ingitlab_workhorse_duo_workflow_session_errors_total), and the WebSocket is closed with1008andInvalid client event: <reason>.
Not in this MR: no per-connection rate budget and no change to MaxMessageSize. Those depend on the client message size and rate metrics and will follow separately.
Notes for reviewers
errDuplicateStartRequestkeeps its current behavior: no close code and no teardown reason. I left it unchanged to keep this MR small. It could move to the same1008path in a follow-up.- Responses after a reconnect to a different Workhorse, and clients that still answer checkpoints, count as
unknown_request_id. The log-only rollout is meant to measure exactly that before enforcement. - The pending set is bounded by what DWS sends. DWS waits for each of these actions to be answered.
- The Go tests and the RSpec spec were not run locally: the agent environment couldn't download Go modules and had no Postgres. CI will be the first real run.
References
Screenshots or screen recordings
Not applicable, no UI changes.
How to set up and validate locally
- Recompile and restart workhorse:
make -C workhorse && gdk restart gitlab-workhorse rails-web - Grab the test script at $6073233
- In a separate terminal, monitor for the log entries:
gdk tail gitlab-workhorse | grep 'not valid in the current state of the run' - Send some invalid requests, with the FF off an on, e.g.
GITLAB_TOKEN=$GITLAB_LOCAL_TOKEN GDK_URL=http://gdk.test:3000 PROJECT_ID=1000000 bundle exec ruby invalid_events_client.rb before_startWith the FF off, you only get a warning log entry, with the FF on you get an error and a closed 1008 socket
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.