Meta: Replace mail_room email ingestion with scheduled Sidekiq jobs
As mentioned in #329990 (comment 1201344150) in Configurable e-mail address for service desk (#329990 - closed) I would like to start a conversation on consolidating our email ingestion infrastructure.
Design Document
There is a Design Document (Blueprint) in the Architecture handbook section called "Replace mail_room
email ingestion with scheduled Sidekiq jobs". This is the SSOT and it will be updated regularly.
Proposal
I would like to propose that we stop ingesting emails via a separate mail_room
process and instead use scheduled Sidekiq workers to do this directly.
We will regularly update this description as we have new findings and arguments.
How could that be achieved:
- Strip out necessary logic from
gitlab-mail_room
gem, so we can make IMAP connections and MS Graph call from within GitLab - Add a new worker that polls
incoming_email
andservice_desk_email
mailboxes e.g. every minute - Collect all new and unread messages and schedule jobs to process these messages, like we do today.
- Remove the dependency to
gitlab-mail_room
entirely, remove the internal API endpoint, removemail_room.yml
dynamically generated static configuration file formail_room
(respect deprecation periods of course)
Advantages:
- Easier set up in GDK (no need to run separate processes, no need to manually change and install a different gem version. See also this issue on the story of mail_room. It also seems like we could solve the version inconsistency with ruby 3)
- Reduced infrastructure (less to maintain)
- Improved scalability for multi-node installations (see this issue and this MR on multi-node issues)
- Better use of computation resources (
mail_room
uses one thread per mailbox thatfetches -- sleeps -- etc
. - Easier monitoring (just sidekiq jobs, logging inside GitLab, no separate process) see also Service Desk config errors visible to non-admins (#384530) and &9407 (comment 1197193652)+
- Easier to integrate other service provider adapters like GMail via API etc.
- Foundation for custom email for Service Desk ingestion via IMAP or API as described as long term effort in #329990 (comment 1201344150), we could directly poll custom email addresses like
support@example.com
for a given Service Desk project instead of relying on forwarding to theservice_desk_email
in the long run. See also this #329990 (comment 1227384943) comment on the current approach to fully customizable email addresses for Service Desk. Current architecture and infrastructure would not allow this (see static configuration above)
Disadvantages:
- Will need change management and will probably impact large installations as they might need to change infrastructure (but to the better)
- Adding to the above. Probably might be a breaking change.
- Probably decent changes in Omnibus and Help chart (from
@qmnguyen0711
comment below) - We will be responsible for all IMAP and MS Graph API related code on our own (chance and risk)
- Work under the hood that does not change anything for users (just for admins and ops)
-
mail_room
is independent, so it does not affect the GitLab app (does it?)
Current setup vs. proposed setup
Current setup
thank you @qmnguyen0711, taken from #329990 (comment 1199136101)
webhook
delivery method (recommended)
flowchart LR
User --Send email via IMAP--> Gmail
mail_room --Fetch via IMAP--> Gmail
mail_room --HTTP POST--> Web
Web --Schedule--> Sidekiq
Sidekiq --Create Service Desk Issue--> Sidekiq
sidekiq
delivery method (soon deprecated)
flowchart LR
User --Send email via IMAP--> Gmail
mail_room --Fetch via IMAP--> Gmail
mail_room --directly writes to redis queue--> Sidekiq
Sidekiq --Create Service Desk Issue--> Sidekiq
Proposed setup
flowchart LR
User --Send email via IMAP--> provider[Email provider mailbox]
ingestion[Sidekiq ingestion job] --Fetch via IMAP or Graph API--> provider
ingestion --adds n jobs for each fetched email--> create[Sidekiq create Service Desk issue job]
Next steps
After gathering feedback and perspectives and we find that this is a good idea, we should follow the evolution workflow process and start a blueprint for this. I (@msaleiko
) volunteer to start this document.
Additional notes
This is a blocker for Deprecate sidekiq delivery method for email ing... (#398132) . We need to have an alternative to the sidekiq
delivery method for mail_room
in place before we remove it.