Gitlab Mailroom using UBI image fails to read incoming email
Bug
Gitlab Mailroom pod when deployed using the Big Bang helm chart (related ticket there: https://repo1.dso.mil/big-bang/product/packages/gitlab/-/issues/220) is in CrashLoopBackOff because it is unable to pull and arbitrate incoming emails. We are using microsoft_graph for the inbox.
Description
This happens on startup of the mailroom pod. The pod immediately crashes. The same mailroom configuration was working on older versions of GitLab (pre 15.10).
Appears to be an outdated redis-namespace dependency that is incompatible with Ruby 3 in the UBI mailroom image (see previous related ticket: #657 (closed)).
Dockerfile.build.ubi8 is using redis-namespace v1.6.0 instead of v1.10.0 as specified in the install-dependencies script.
This is the pod logs on startup:
Begin parsing .erb templates from /etc
[2023-11-30 20:19:41] INFO WEBrick 1.7.0
[2023-11-30 20:19:41] INFO ruby 3.0.6 (2023-03-30) [x86_64-linux]
{"severity":"INFO","time":"2023-11-30T20:19:41.175+00:00","context":{"email":"redacted-email@company.onmicrosoft.com","name":"inbox"},"action":"Setting up watcher"}
[2023-11-30 20:19:41] INFO WEBrick::HTTPServer#start: pid=1 port=8080
{"severity":"INFO","time":"2023-11-30T20:19:41.269+00:00","context":{"email":"redacted-email@company.onmicrosoft.com","name":"inbox"},"action":"Retrieving OAuth2 token..."}
{"severity":"INFO","time":"2023-11-30T20:19:41.579+00:00","context":{"email":"redacted-email@company.onmicrosoft.com","name":"inbox"},"action":"Processing started"}
{"severity":"INFO","time":"2023-11-30T20:19:42.128+00:00","context":{"email":"redacted-email@company.onmicrosoft.com","name":"inbox"},"uid":"<redacted>","action":"asking arbiter to deliver","arbitrator":"MailRoom::Arbitration::Redis"}
#<Thread:0x00007fba09c667c0 /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/mailbox_watcher.rb:35 run> terminated with exception (report_on_exception is true):
/usr/lib64/ruby/gems/3.0.0/gems/redis-5.0.7/lib/redis/commands/strings.rb:83:in `set': wrong number of arguments (given 3, expected 2) (ArgumentError)
from /usr/lib64/ruby/gems/3.0.0/gems/redis-namespace-1.6.0/lib/redis/namespace.rb:442:in `call_with_namespace'
from /usr/lib64/ruby/gems/3.0.0/gems/redis-namespace-1.6.0/lib/redis/namespace.rb:328:in `method_missing'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/arbitration/redis.rb:34:in `deliver?'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/mailbox.rb:108:in `deliver?'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:169:in `block in unread_messages'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:169:in `select'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:169:in `unread_messages'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:153:in `block (2 levels) in new_message_ids'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:152:in `loop'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:152:in `block in new_message_ids'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:201:in `each'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:201:in `each_with_object'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:201:in `messages_for_ids'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:144:in `new_messages'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:122:in `process_mailbox'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:26:in `wait'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/mailbox_watcher.rb:37:in `block in run'
[2023-11-30 20:19:42] INFO going to shutdown ...
[2023-11-30 20:19:42] INFO WEBrick::HTTPServer#start done.
{"severity":"INFO","time":"2023-11-30T20:19:42.151+00:00","context":{"email":"redacted-email@company.onmicrosoft.com","name":"inbox"},"action":"Quitting connection..."}
{"severity":"INFO","time":"2023-11-30T20:19:42.151+00:00","context":{"email":"redacted-email@company.onmicrosoft.com","name":"inbox"},"action":"Terminating watching thread..."}
/usr/lib64/ruby/gems/3.0.0/gems/redis-5.0.7/lib/redis/commands/strings.rb:83:in `set': wrong number of arguments (given 3, expected 2) (ArgumentError)
from /usr/lib64/ruby/gems/3.0.0/gems/redis-namespace-1.6.0/lib/redis/namespace.rb:442:in `call_with_namespace'
from /usr/lib64/ruby/gems/3.0.0/gems/redis-namespace-1.6.0/lib/redis/namespace.rb:328:in `method_missing'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/arbitration/redis.rb:34:in `deliver?'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/mailbox.rb:108:in `deliver?'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:169:in `block in unread_messages'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:169:in `select'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:169:in `unread_messages'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:153:in `block (2 levels) in new_message_ids'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:152:in `loop'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:152:in `block in new_message_ids'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:201:in `each'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:201:in `each_with_object'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:201:in `messages_for_ids'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:144:in `new_messages'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:122:in `process_mailbox'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/microsoft_graph/connection.rb:26:in `wait'
from /usr/lib64/ruby/gems/3.0.0/gems/gitlab-mail_room-0.0.23/lib/mail_room/mailbox_watcher.rb:37:in `block in run'And this is the mailbox configuration. Again, this was previously working before the Ruby 3 upgrade in GitLab Mailroom UBI image.
:mailboxes:
-
:email: <redacted>
:name: inbox
:delete_after_delivery: false
:expunge_deleted: false
:logger:
:log_path: "/dev/stdout"
:inbox_method: microsoft_graph
:inbox_options:
:tenant_id: "<redacted>"
:client_id: "<redacted>"
:client_secret: <%= File.read("/etc/gitlab/mailroom/client_id_incoming_email").strip.to_json %>
:poll_interval: 60
:azure_ad_endpoint: https://login.microsoftonline.us
:graph_endpoint: https://graph.microsoft.us
:delivery_method: postback
:delivery_options:
:delivery_url: 'http://gitlab-webservice-default.gitlab.svc:8181/api/v4/internal/mail_room/incoming_email'
:content_type: "text/plain"
:jwt_auth_header: "Gitlab-Mailroom-Api-Request"
:jwt_issuer: "gitlab-mailroom"
:jwt_algorithm: "HS256"
:jwt_secret_path: "/etc/gitlab/mailroom/incoming_email_webhook_secret"
:arbitration_method: redis
:arbitration_options:
:redis_url: redis://:<%= ERB::Util::url_encode(File.read("/etc/gitlab/redis/redis-password").strip) %>@gitlab-redis-master.gitlab.svc:6379
:namespace: mail_room:gitlabAffected versions
- Evidenced on 7.5.1 / GitLab 16.5.1
Edited by Jason Plum