Improvements for Incoming Email feature (Mailroom) in scaled setups

Context

After assisting on a support issue for the Incoming Email feature (Mailroom) we noticed there wasn't clear enough guidance for scaled (multi-node) setups, including:

• What settings are required when there's multiple Rails nodes
• That there's different delivery methods and what is recommended

After some investigation we found out the following for the above:

• That gitlab_rails['incoming_email_enabled'] and gitlab_rails['incoming_email_address'] are needed on every Rails node as it needs those settings to render elements in the UI
• That gitlab_rails['incoming_email_enabled'] also enables the Mailroom service. It looks to be the case that only one instance of this service is running so on other Rails nodes the mailroom['enabled'] = false is needed to prevent race conditions
• That gitlab_rails['incoming_email_delivery_method'] is available and that it has two choices, webhook or sidekiq, with webhook being the default. However this requires that the same token is configured on every rails node and traffic going externally through the external ingress. Sidekiq doesn't need tokens, the traffic remains internal and Sidekiq is better suited for this task so it looks like the better option in most (all?) cases and this should be called out.

Acceptance criteria

• Document the findings above in the user docs for Incoming Email
• Look at any changes that can make this process smoother, especially the handling required for the Mailroom service on multiple nodes (perhaps Omnibus roles?), and potentially create follow up issues to address those improvements