Proposal: Use a web hook for mailroom instead of pushing directly to redis-sidekiq
In &490 (closed) we applied compression to our Sidekiq job payloads, which meant that they are now much smaller than they were before. We do have one set of jobs to which this doesn't apply, though: incoming emails. https://log.gprd.gitlab.net/goto/5d6ce52b36242583e9369ecc7472ba83 shows that these have a relatively high median job size as a result.
Incoming emails are handled by a mail_room process that pushes jobs directly to Redis. We generate the config for this in two places:
- For source and Omnibus installs, we use https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/mail_room.yml (via https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/6158de5dbb748f57db3cfecca62d57b0bc986392/files/gitlab-cookbooks/gitlab/recipes/mailroom.rb#L25 for Omnibus).
- For charts, we use https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/mail_room.yml.
While we could attempt to add compression to mail_room itself, that seems to be adding complexity where a conceptually simpler solution exists. This solution is more complicated to migrate to, though.
The mail_room gem also supports a postback method, where we can have it hit an HTTP endpoint with the body of the mail. This is basically what we do for git pushes triggering our PostReceive worker: gitlab-shell hits an internal post_receive
endpoint (https://gitlab.com/gitlab-org/gitlab/-/blob/509e4ffb7626999af33406638bb80cd0de695d85/lib/api/internal/base.rb#L278-284) that then triggers the job. This gives us the job compression for free, as we're just a regular Sidekiq client with all the middleware from the Rails application.
Having mail_room work differently here is confusing and is probably technical debt.
This would also give us some other benefits from running other client middleware:
- We'd have context metadata applied automatically.
- We'd be able to apply worker routing rules automatically (#1087 (closed)).
Proposal
Phase 1
-
In Rails' codebase:
- Implement an internal API:
POST /api/v4/internal/mailroom/*address
. The API is a thin wrapper to receive the request, classify the address based onaddress
URL parameters, then push toEmailReceiverWorker
andServiceDeskEmailReceiverWorker
accordingly. These endpoints should also capture the size limiter rejection exceptions and return 400 status. - Add
JwtAuthenticable
authentication to the aforementioned endpoint. - The JWT authentication needs a shared secret file between mailroom gem and Rails code base. As there are two addresses with different configurations, we must add a
secret_file
field to both configuration.
- Implement an internal API:
-
In the mailroom gem:
- Although mail_room supports postback, it uses Faraday and its token_auth configuration. As a result, we need to update the gem so that the
postback
delivery strategy supports JWT authentication.
- Although mail_room supports postback, it uses Faraday and its token_auth configuration. As a result, we need to update the gem so that the
-
In the mailroom Helm chart:
- Add
authToken
configuration to mailroom helm chart. The secret is mounted as files in both webservice and mailroom deployment definitions. - The secret file path is mounted as
secret_file
filed ofincoming_email
andservice_desk_email
configuration of the webservice configmap - The secret file path is mounted as delivery options of mailroom configmap.
- Generalize mailroom chart, support both the old sidekiq and postback method. We can use
authToken
to conditionally generate the configuration. - Add
workhorse
configuration to mailroom. This is similarly to gitlab-shell's configuration.
- Add
At the end of this phase, GitLab.com instance now lets MailRoom use WebHook instead of Sidekiq strategy. The sidekiq strategy is still used by self-managed instances and GDK.
Phase 2
This phase is to replace the sidekiq strategy by the webhook strategy completely.
- In the Gitlab Development Kit: we need to switch from sidekiq strategy to the postback strategy. During the migration phase, GDK must support both delivery strategies.
- In Omnibus: It's quite tricky because the final configuration is composed from
mailroom.yaml
,incoming_email_*
configs,service_desk_email_*
, and mailroom dedicated config. The final goal is still to support 2 versions. I'm not super clear about the details, but I'm sure it's hard. - In Rails code base: replace the delivery options in config/mail_room.yml by the webhook equivalence. As GDK and Omnibus both support two delivery strategies, they should work well afterward.
- Remove Sidekiq strategy in GDK and Omnibus. This should be done in a later major release.