Skip to content

Meta: Replace mail_room email ingestion with scheduled Sidekiq jobs

As mentioned in #329990 (comment 1201344150) in Configurable e-mail address for service desk (#329990 - closed) I would like to start a conversation on consolidating our email ingestion infrastructure.

Design Document

There is a Design Document (Blueprint) in the Architecture handbook section called "Replace mail_room email ingestion with scheduled Sidekiq jobs". This is the SSOT and it will be updated regularly.

Proposal

I would like to propose that we stop ingesting emails via a separate mail_room process and instead use scheduled Sidekiq workers to do this directly.

We will regularly update this description as we have new findings and arguments.

How could that be achieved:

  1. Strip out necessary logic from gitlab-mail_room gem, so we can make IMAP connections and MS Graph call from within GitLab
  2. Add a new worker that polls incoming_email and service_desk_email mailboxes e.g. every minute
  3. Collect all new and unread messages and schedule jobs to process these messages, like we do today.
  4. Remove the dependency to gitlab-mail_room entirely, remove the internal API endpoint, remove mail_room.yml dynamically generated static configuration file for mail_room (respect deprecation periods of course)

Advantages:

  • Easier set up in GDK (no need to run separate processes, no need to manually change and install a different gem version. See also this issue on the story of mail_room. It also seems like we could solve the version inconsistency with ruby 3)
  • Reduced infrastructure (less to maintain)
  • Improved scalability for multi-node installations (see this issue and this MR on multi-node issues)
  • Better use of computation resources (mail_room uses one thread per mailbox that fetches -- sleeps -- etc.
  • Easier monitoring (just sidekiq jobs, logging inside GitLab, no separate process) see also Service Desk config errors visible to non-admins (#384530) and &9407 (comment 1197193652)+
  • Easier to integrate other service provider adapters like GMail via API etc.
  • Foundation for custom email for Service Desk ingestion via IMAP or API as described as long term effort in #329990 (comment 1201344150), we could directly poll custom email addresses like support@example.com for a given Service Desk project instead of relying on forwarding to the service_desk_email in the long run. See also this #329990 (comment 1227384943) comment on the current approach to fully customizable email addresses for Service Desk. Current architecture and infrastructure would not allow this (see static configuration above)

Disadvantages:

  • Will need change management and will probably impact large installations as they might need to change infrastructure (but to the better)
  • Adding to the above. Probably might be a breaking change.
  • Probably decent changes in Omnibus and Help chart (from @qmnguyen0711 comment below)
  • We will be responsible for all IMAP and MS Graph API related code on our own (chance and risk)
  • Work under the hood that does not change anything for users (just for admins and ops)
  • mail_room is independent, so it does not affect the GitLab app (does it?)

Current setup vs. proposed setup

Current setup

thank you @qmnguyen0711, taken from #329990 (comment 1199136101)

webhook delivery method (recommended)

flowchart LR
  User --Send email via IMAP--> Gmail
  mail_room --Fetch via IMAP--> Gmail
  mail_room --HTTP POST--> Web
  Web --Schedule--> Sidekiq
  Sidekiq --Create Service Desk Issue--> Sidekiq

sidekiq delivery method (soon deprecated)

flowchart LR
  User --Send email via IMAP--> Gmail
  mail_room --Fetch via IMAP--> Gmail
  mail_room --directly writes to redis queue--> Sidekiq
  Sidekiq --Create Service Desk Issue--> Sidekiq

Proposed setup

flowchart LR
  User --Send email via IMAP--> provider[Email provider mailbox]
  ingestion[Sidekiq ingestion job] --Fetch via IMAP or Graph API--> provider
  ingestion --adds n jobs for each fetched email--> create[Sidekiq create Service Desk issue job]

Next steps

After gathering feedback and perspectives and we find that this is a good idea, we should follow the evolution workflow process and start a blueprint for this. I (@msaleiko) volunteer to start this document.

Additional notes

This is a blocker for Deprecate sidekiq delivery method for email ing... (#398132) . We need to have an alternative to the sidekiq delivery method for mail_room in place before we remove it.

Edited by Marc Saleiko