Improve `ActionMailer::MailDeliveryJob` performance by using replica
Problem
From gitlab-com/gl-infra/scalability#1811 (comment 1332375768). Amongst the workers set to :always data consistency, ActionMailer::MailDeliveryJob is consistently in the top 5 for sum.db_main_duration_s.
MailDeliveryJob performs the mail_method which is the mailer's logic (Gitlab code) before calling the delivery_method which sends the mail. From kibana, ~20% of jobs have 0 json.db_write_count and the remainder only have 1 json.db_write_count (it shows 3 in kibana but it is a bug that !116218 (merged) fixes).
The insert is performed by SentNotification.record. An example is in https://gitlab.com/gitlab-org/gitlab/-/blob/c0c3de94829c80df91a0d5b070205a2326474334/app/mailers/emails/issues.rb#L6.
It is used in mailers as seen from a ripgrep of the project.
ee/app/mailers/emails/epics.rb
35: @sent_notification = SentNotification.record(@epic, recipient_id, reply_key)
app/mailers/emails/reviews.rb
34: @sent_notification = SentNotification.record(@merge_request, recipient_id, reply_key)
app/mailers/emails/service_desk.rb
55: @sent_notification = SentNotification.record(@issue, @support_bot.id, reply_key)
app/mailers/emails/issues.rb
115: @sent_notification = SentNotification.record(@issue, recipient_id, reply_key)
app/mailers/emails/notes.rb
82: @sent_notification = SentNotification.record_note(@note, recipient_id, reply_key)
app/mailers/emails/merge_requests.rb
176: @sent_notification = SentNotification.record(@merge_request, recipient_id, reply_key)Caveat: jobs with 1-write, the write is performed within a transaction, so the performance gains would not be mind-blowing but the initial chunk of reads will be off-loaded to the replica.
Proposal
Switch ActionMailer::MailDeliveryJob data consistency to :sticky or even :delayed.
Adding This is fake news. Mail jobs passes through our middleware as a data_consistency for ActionMailer::MailDeliveryJob will be done through DummyWorker which is done in sidekiq_config.ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper. We will likely need to update the load balancing middlewares.