Duo Workflow Service should be able to handle gRPC responses out of order

Problem

Our current Duo Workflow Service code handling gRPC messages only works if the request and response are strictly in order and this is blocking some enhancements in gitlab-org/gitlab#543044 gitlab-org/gitlab#545037 gitlab-org/gitlab#543908 .

There is some discussion at !2513 (comment 2511636997) explaining how our current architecture is forced to wait for a response every time it sends a message down the gRPC channel. The code at https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/blob/12d8044cc5a5f676ceb6f5f42f78ec2a77fae98b/duo_workflow_service/executor/action.py shows the problem because we are expecting the next message in the inbox to correspond to the message we just put in the outbox.

This approach won't deal well with passing messages concurrently.

Solution

We'll need to introduce a new mechanism to match the inbox messages to their corresponding outbox messages using the requestID in the payload. In async Python this should likely make use of Futures and some kind of map. We can store a future per message and in a map by the requestID and then when the response comes in we locate that Future and call set_result.

This makes a nice interface for calling code as they can just call await _execute_action and they will be able to wait specifically on that action to finish.