Manually expunge huge backlog of deleted messages in mailroom IMAP inbox
C3
Production Change - Criticality 3Per https://github.com/tpitale/mail_room/issues/85, which was fixed in https://github.com/tpitale/mail_room/pull/90, the mail_room gem has not been expunging deleted (processed) messages.
Before we deploy the latest mail_room gem (coming soon, see gitlab-org/gitlab!19186 (merged)), we should manually expunge the backlog (~533K messages) in a controlled fashion outside of mail_room. After this change, newly deleted messages will still not be expunged, but the number should be small and manageable when the upgrade occurs after which expunging will happen naturally over time.
Change Objective | Expunge ~540K historical 'deleted' messages in the imap inbox used by mail_room. |
---|---|
Change Type | Operation |
Services Impacted | mailroom |
Change Team Members | @cmiskell |
Change Severity | C3 |
Change Reviewer or tested in staging | Review-only (TBD); staging does not have any deleted but unexpunged messages, and also cannot connect to imap.gmail.com from the console server |
Due Date | 2019-11-01 01:00 |
Time tracking | 5 minutes; rollback not possible |
-
Start a rails console in a tmux session on console-01-sv-gprd -
Run this to prepare, and keep the session alive
imap = MailRoom::IMAP.new("imap.gmail.com", 993, :ssl => true)
imap.login("incoming@gitlab.com", "REDACTED")
imap.select("inbox")
imap.uid_search("DELETED").length
-
Verify that the number of DELETED messages is around 533K. - If not close to this number (should be larger, but not by much), abort this change and debug the situation.
-
When ready to expunge, in the same rails console, run: imap.expunge()
-
Re-run imap.uid_search("DELETED").length
to verify there are no DELETED messages remaining (or nearly 0) -
Check https://prometheus-app.gprd.gitlab.net/graph?g0.range_input=1h&g0.expr=imap_nb_messages_in_mailbox&g0.tab=0 and confirm that the number of messages in the mailbox has substantially dropped (near 0). There will still be some stuck messages (~50 at this writing), to be dealt with independently.
No alerts are expected to be triggered.
Edited by Craig Miskell