Proposal: Use a web hook for mailroom instead of pushing directly to redis-sidekiq
In &490 (closed) we applied compression to our Sidekiq job payloads, which meant that they are now much smaller than they were before. We do have one set of jobs to which this doesn't apply, though: incoming emails. https://log.gprd.gitlab.net/goto/5d6ce52b36242583e9369ecc7472ba83 shows that these have a relatively high median job size as a result.
Incoming emails are handled by a mail_room process that pushes jobs directly to Redis. We generate the config for this in two places:
- For source and Omnibus installs, we use https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/mail_room.yml (via https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/6158de5dbb748f57db3cfecca62d57b0bc986392/files/gitlab-cookbooks/gitlab/recipes/mailroom.rb#L25 for Omnibus).
- For charts, we use https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/mail_room.yml.
While we could attempt to add compression to mail_room itself, that seems to be adding complexity where a conceptually simpler solution exists. This solution is more complicated to migrate to, though.
The mail_room gem also supports a postback method, where we can have it hit an HTTP endpoint with the body of the mail. This is basically what we do for git pushes triggering our PostReceive worker: gitlab-shell hits an internal
post_receive endpoint (https://gitlab.com/gitlab-org/gitlab/-/blob/509e4ffb7626999af33406638bb80cd0de695d85/lib/api/internal/base.rb#L278-284) that then triggers the job. This gives us the job compression for free, as we're just a regular Sidekiq client with all the middleware from the Rails application.
Having mail_room work differently here is confusing and is probably technical debt.
This would also give us some other benefits from running other client middleware:
- We'd have context metadata applied automatically.
- We'd be able to apply worker routing rules automatically (#1087 (closed)).
In Rails' codebase:
- Implement an internal API:
POST /api/v4/internal/mailroom/*address. The API is a thin wrapper to receive the request, classify the address based on
addressURL parameters, then push to
ServiceDeskEmailReceiverWorkeraccordingly. These endpoints should also capture the size limiter rejection exceptions and return 400 status.
JwtAuthenticableauthentication to the aforementioned endpoint.
- The JWT authentication needs a shared secret file between mailroom gem and Rails code base. As there are two addresses with different configurations, we must add a
secret_filefield to both configuration.
- Implement an internal API:
In the mailroom gem:
In the mailroom Helm chart:
authTokenconfiguration to mailroom helm chart. The secret is mounted as files in both webservice and mailroom deployment definitions.
- The secret file path is mounted as
service_desk_emailconfiguration of the webservice configmap
- The secret file path is mounted as delivery options of mailroom configmap.
- Generalize mailroom chart, support both the old sidekiq and postback method. We can use
authTokento conditionally generate the configuration.
workhorseconfiguration to mailroom. This is similarly to gitlab-shell's configuration.
At the end of this phase, GitLab.com instance now lets MailRoom use WebHook instead of Sidekiq strategy. The sidekiq strategy is still used by self-managed instances and GDK.
This phase is to replace the sidekiq strategy by the webhook strategy completely.
- In the Gitlab Development Kit: we need to switch from sidekiq strategy to the postback strategy. During the migration phase, GDK must support both delivery strategies.
- In Omnibus: It's quite tricky because the final configuration is composed from
service_desk_email_*, and mailroom dedicated config. The final goal is still to support 2 versions. I'm not super clear about the details, but I'm sure it's hard.
- In Rails code base: replace the delivery options in config/mail_room.yml by the webhook equivalence. As GDK and Omnibus both support two delivery strategies, they should work well afterward.
- Remove Sidekiq strategy in GDK and Omnibus. This should be done in a later major release.