Make the Bots service send Operation messages to a RabbitMQ exchange
Context
Currently there's a lot of contention for database connections, especially when we scale up and/or are under heavy load. One of the various reasons for that is the need to regularly poll the database to check for updates to Operation/Job state. For PostgreSQL we use LISTEN/NOTIFY in the JobWatcher thread (this is the thread which monitors the db for those updates), which means we're always using a minimum of one connection per Execution service.
The Bots service should support directly generating an Operation message (with no name set) when an UpdateBotSession request is received, and serialising the message to publish it onto a RabbitMQ exchange. This message should contain as much of the current state of the Operation as is possible to be deduced from the context the Bots service has. We shouldn't add any database interaction to support this.
We shouldn't drop the existing database update and PostgreSQL notification until the other services can consume from the same exchange properly.
Solution starting point
See the Operation construction code in the RabbitMQ prototype.
Most of the state for the Operation can be easily deduced from the Lease state as in that prototype. We shouldn't do lookups in a shared Redis in the final solution, that is an artefact of trying to reduce the number of updates we sent.
We should serialise the Operation message we create, and publish it to a RabbitMQ topic exchange, called something like operation-updates
. The routing key should include the Action digest the Operation is for, and also maybe some indication of the type of update (maybe the Operation stage makes sense?). Deciding on a format for this routing key is part of this task, and will influence how the updates are consumed by the ActionCache, Execution, and Operations services.
Acceptance Criteria
The Bots service publishes Operation messages to a RabbitMQ topic exchange to be consumed by other services.
Consider whether the following are required, and complete if so:
-
Unit tests -
Metrics -
Documentation update(s)