All incoming mail stopped working after update to v15.3.1 from v15.2.2 last night
Summary
Incoming mails worked perfectly fine until I received the v15.3.1 update last night. Since then service desk and ticket creation via e-mail services are not working.
The configuration has not changed since the upgrade.
It appears that a missing secret_file (or equivalent omnibus field
Steps to reproduce
-
git pull gitlab/gitlab-ce:latestwhile you have a v15.2.2 image with the config below docker-compose down && docker-compose up -d
Related config (gitlab.rb)
gitlab_rails['incoming_email_enabled'] = true
gitlab_rails['incoming_email_address'] = "tickets+%{key}@myhost.com"
gitlab_rails['incoming_email_email'] = "tickets@myhost.com"
gitlab_rails['incoming_email_password'] = "[READCTED]"
gitlab_rails['incoming_email_mailbox_name'] = "inbox"
gitlab_rails['incoming_email_idle_timeout'] = 60
gitlab_rails['incoming_email_host'] = "mx.myhost.com"
gitlab_rails['incoming_email_port'] = 993
gitlab_rails['incoming_email_ssl'] = true
gitlab_rails['incoming_email_start_tls'] = false
gitlab_rails['service_desk_email_enabled'] = true
gitlab_rails['service_desk_email_address'] = "support+%{key}@myhost.com"
gitlab_rails['service_desk_email_email'] = "support@myhost.com"
gitlab_rails['service_desk_email_password'] = "[READCTED]"
gitlab_rails['service_desk_email_mailbox_name'] = "inbox"
gitlab_rails['service_desk_email_idle_timeout'] = 60
gitlab_rails['service_desk_email_log_file'] = "/var/log/gitlab/mailroom/mail_room_json.log"
gitlab_rails['service_desk_email_host'] = "mx.myhost.com"
gitlab_rails['service_desk_email_port'] = 993
gitlab_rails['service_desk_email_ssl'] = true
gitlab_rails['service_desk_email_start_tls'] = falseWhat is the current bug behavior?
Gitlab does not process new incoming mails for service desk and tickets after upgrading from v15.2.2 to v15.3.1
What is the expected correct behavior?
Gitlab processes the incoming emails
Relevant logs and/or screenshots
I checked the mailroom.log and it says:
"action":"Disconnected. Resetting...","error":"Error in IMAP command received by server."
We also found this log entry in mailroom's log:
2022-08-23_15:42:42.81953 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/jwt.rb:30:in `read': No such file or directory @ rb_sysopen - .gitlab_incoming_email_secret (Errno::ENOENT)
2022-08-23_15:42:42.81953 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/jwt.rb:30:in `token'
2022-08-23_15:42:42.81953 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:100:in `config_request_jwt_auth'
2022-08-23_15:42:42.81954 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:82:in `block in deliver'
2022-08-23_15:42:42.81954 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:442:in `block in run_request'
2022-08-23_15:42:42.81955 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:459:in `block in build_request'
2022-08-23_15:42:42.81956 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:40:in `block in create'
2022-08-23_15:42:42.81956 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:39:in `tap'
2022-08-23_15:42:42.81957 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:39:in `create'
2022-08-23_15:42:42.81957 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:455:in `build_request'
2022-08-23_15:42:42.81958 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:437:in `run_request'
2022-08-23_15:42:42.81958 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:281:in `post'
2022-08-23_15:42:42.81958 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:78:in `deliver'
2022-08-23_15:42:42.81959 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox.rb:118:in `deliver'
2022-08-23_15:42:42.81959 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:32:in `block in run'
2022-08-23_15:42:42.81960 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:138:in `map'
2022-08-23_15:42:42.81961 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:138:in `process_mailbox'
2022-08-23_15:42:42.81961 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:49:in `wait'
2022-08-23_15:42:42.81962 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:37:in `block in run'
2022-08-23_15:42:42.81973 /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/jwt.rb:30:in `read': No such file or directory @ rb_sysopen - .gitlab_incoming_email_secret (Errno::ENOENT)
2022-08-23_15:42:42.81975 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/jwt.rb:30:in `token'
2022-08-23_15:42:42.81975 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:100:in `config_request_jwt_auth'
2022-08-23_15:42:42.81976 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:82:in `block in deliver'
2022-08-23_15:42:42.81977 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:442:in `block in run_request'
2022-08-23_15:42:42.81977 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:459:in `block in build_request'
2022-08-23_15:42:42.81977 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:40:in `block in create'
2022-08-23_15:42:42.81978 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:39:in `tap'
2022-08-23_15:42:42.81979 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/request.rb:39:in `create'
2022-08-23_15:42:42.81979 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:455:in `build_request'
2022-08-23_15:42:42.81980 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:437:in `run_request'
2022-08-23_15:42:42.81980 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/faraday-2.5.2/lib/faraday/connection.rb:281:in `post'
2022-08-23_15:42:42.81981 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/delivery/postback.rb:78:in `deliver'
2022-08-23_15:42:42.81981 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox.rb:118:in `deliver'
2022-08-23_15:42:42.81981 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:32:in `block in run'
2022-08-23_15:42:42.81982 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:138:in `map'
2022-08-23_15:42:42.81982 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:138:in `process_mailbox'
2022-08-23_15:42:42.81983 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/imap/connection.rb:49:in `wait'
2022-08-23_15:42:42.81984 from /opt/gitlab/embedded/lib/ruby/gems/2.7.0/gems/gitlab-mail_room-0.0.20/lib/mail_room/mailbox_watcher.rb:37:in `block in run'
2022-08-23_15:42:42.82013 #<struct MailRoom::Delivery::Postback::Options url="https://[REDACTED]/api/v4/internal/mail_room/incoming_email", token=nil, username=nil, password=nil, logger=#<MailRoom::Logger::Structured:0x00007f37646b51a0 @level=0, @progname=nil, @default_formatter=#<Logger::Formatter:0x00007f37646b5178 @datetime_format=nil>, @formatter=nil, @logdev=#<Logger::LogDevice:0x00007f37646b5128 @shift_period_suffix="%Y%m%d", @shift_size=1048576, @shift_age=0, @filename="/var/log/gitlab/mailroom/mail_room_json.log", @dev=#<File:/var/log/gitlab/mailroom/mail_room_json.log>, @binmode=false, @mon_data=#<Monitor:0x00007f37646b50d8>, @mon_data_owner_object_id=1460>>, content_type="text/plain", jwt=#<MailRoom::JWT:0x00007f37643d9670 @header="Gitlab-Mailroom-Api-Request", @secret_path=".gitlab_incoming_email_secret", @issuer="gitlab-mailroom", @algorithm="HS256">>Output of checks
docker exec -it gitlab gitlab-rake gitlab:incoming_email:check’s output:
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking email1@example.com
Checking email2@example.com
yes
Mailroom enabled? ... skipped
MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finishedopenssl s_client -connect example.myhost.com:993 works fine
telnet smtp.myhost.com 587 works
telnet smtp.myhost.com 25 works
telnet smtp.myhost.com 993 I see the sent and received packages on docker host with tcpdump, but I could not get a prompt for imap. If I create an ubuntu:22.04 container on the same docker network it works perfectly fine. If I downgrade to v15.2.2 I also can not get prompt but the mailing working for newly received mails.
Results of GitLab environment info
It is the downgraded working v15.2.2's check because I can not break the production right now. Maybe it helps somehow. (I can post the v15.3.1 output later if needed.)
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.7.5p203
Gem Version: 3.1.4
Bundler Version:2.3.15
Rake Version: 13.0.6
Redis Version: 6.2.7
Sidekiq Version:6.4.0
Go Version: unknown
GitLab information
Version: 15.2.2
Revision: 4ecb014a935
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.10
URL: https://gitlab.myhost.com
HTTP Clone URL: https://gitlab.myhost.com/some-group/some-project.git
SSH Clone URL: git@gitlab.myhost.com:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 14.9.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shellResults of GitLab application Check
It is the current working v15.2.2's check because I can not break the production right now. If needed I will post the v15.3.1's check. The mailroom resulst are the same in both checks.
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.9.0 ? ... OK (14.9.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking tickets@myhost.com
Checking support@myhost.com
yes
Mailroom enabled? ... skipped
MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units)
Projects have namespace: ...
13/2 ... yes
13/3 ... yes
13/5 ... yes
13/6 ... yes
13/8 ... yes
13/9 ... yes
13/14 ... yes
13/15 ... yes
13/16 ... yes
13/17 ... yes
13/18 ... yes
13/20 ... yes
13/21 ... yes
13/22 ... yes
13/24 ... yes
13/25 ... yes
13/26 ... yes
13/27 ... yes
13/28 ... yes
13/29 ... yes
13/30 ... yes
13/31 ... yes
Redis version >= 6.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git user has default SSH configuration? ... yes
Active users: ... 17
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
From @cablett:
As part of omnibus-gitlab#6991 (closed) this was fixed in %15.4 in Omnibus. For %15.4 and above, you can do the following:
- switch service desk back to webhook on Omnibus
- call the
gitlab-ctl reconfigure, which will autogenerate a secret with the appropriately random content
From original author:
The workaround I did:
- Stop the Gitlab container
- Make a backup from the whole directory
- Tag the previous image with
docker tag xxxxxxxxxxxx gitlab/gitlab-ce:last-working - Replace the image in
docker-compose.yml to image: gitlab/gitlab-ce:last-working - Start the Gitlab container
After this it works with the new incoming emails but it will not process the mails arrived to inbox while gitlab was v15.3.1.
It is this .gitlab_incoming_email_secret file that gitlab expects. I don’t know what needs to be in there though. If you just touch /var/opt/gitlab/gitlab-rails/working/.gitlab_incoming_email_secret to create the file, then the exception does not happen. But the mails are just retrieved and deleted from the inbox.
Status
(From @cablett) See #371716 (comment 1190383834)
- Raise an MR to add to documentation
👉 !105958 (merged) - Raise an MR to raise the relevant error if the config is missing JWT fields for webhook.
- Mailroom. Check for required JWT fields. Be very noisy if these are missing (assuming the upstream maintainer is OK with that approach)
👉 https://github.com/tpitale/mail_room/issues/146 - Omnibus (or some other relevant place) to noisily fail if the config is missing the relevant JWT fields (I think most of them are automatically added but we can check for
secret_fileat the very least). (edit: this is fixed in Omnibus %15.4 onwards🏅 see👉 #371716 (comment 1191962462)) - In the UI under Service Desk itself - it is contextually relevant ("why is service desk broken?" for those without access to logs) but at the wrong "level" - the project owner may not have admin access, but we could say something like "Secret file is missing, check with your administrator" or "Service Desk Email is misconfigured. Please ask the admin to check".
🤔 👉 #384530
- Mailroom. Check for required JWT fields. Be very noisy if these are missing (assuming the upstream maintainer is OK with that approach)
Edited by charlie ablett