Email errors: SMTP "invalid argument"

Summary

I installed gitlab on an internal server, pointing it to our own mail server (kerio). It worked perfectly for almost a year, then suddenly stopped. I thought this was due to a forced password change, but when I updated the password it still would not mail.

From the console when I run Notify.test_email() I get an error:

Errno::EINVAL (Invalid argument - connect(2) for "calendar.ivpress.com" port 587)

Interestingly, if I use a definitely wrong password in the configuration, it still errors in the same way; it doesn't make it to the authentication error.

I have tried:

  • sending an email in gitlab using a personal gmail account (WORKS)
  • sending an email in outside program using the same credentials/settings (WORKS FROM POWERSHELL, FROM NEXTCLOUD)
  • changing ports, changing settings, trying pretty much all permutations (DOES NOT WORK)

Business context: We have been doing some light testing within the IT department over the past year, hoping to deploy GitLab more broadly to replace JIRA. I was just adding some outside beta testers when this email issue popped up. So I can't invite them, nor can they test major functionality. Any help would be appreciated.

Steps to reproduce

Send SMTP email from gitlab via a kerio mail server.

What is the current bug behavior?

Error is raised, no email sent

What is the expected correct behavior?

Emails should send

Relevant logs

Relevant logs 
==> /var/log/gitlab/sidekiq/current <==
2019-03-11_15:16:54.66985 2019-03-11T15:16:54.669Z 25264 TID-oud4syhs8 ActionMailer::DeliveryJob JID-6e47b7d1427cc451e0a7bce8 INFO: start
2019-03-11_15:16:54.73282 2019-03-11T15:16:54.732Z 25264 TID-oud4syhs8 ActionMailer::DeliveryJob JID-6e47b7d1427cc451e0a7bce8 INFO: fail: 0.063 sec
2019-03-11_15:16:54.73304 2019-03-11T15:16:54.732Z 25264 TID-oud4syhs8 WARN: {"context":"Job raised exception","job":{"class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::DeliveryJob","queue":"mailers","args":[{"job_class":"ActionMailer::DeliveryJob","job_id":"4b8d2e34-26fc-4c2a-84dd-c2258821a913","provider_job_id":null,"queue_name":"mailers","priority":null,"arguments":["Notify","send_admin_notification","deliver_now",2,"test","hi"],"locale":"en"}],"retry":3,"jid":"6e47b7d1427cc451e0a7bce8","created_at":1552317381.6325784,"correlation_id":"tI5i8OY9eC6","enqueued_at":1552317414.6683447,"error_message":"Invalid argument - connect(2) for \"calendar.ivpress.com\" port 587","error_class":"Errno::EINVAL","failed_at":1552317381.8407252,"retry_count":0},"jobstr":"{\"class\":\"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper\",\"wrapped\":\"ActionMailer::DeliveryJob\",\"queue\":\"mailers\",\"args\":[{\"job_class\":\"ActionMailer::DeliveryJob\",\"job_id\":\"4b8d2e34-26fc-4c2a-84dd-c2258821a913\",\"provider_job_id\":null,\"queue_name\":\"mailers\",\"priority\":null,\"arguments\":[\"Notify\",\"send_admin_notification\",\"deliver_now\",2,\"test\",\"hi\"],\"locale\":\"en\"}],\"retry\":3,\"jid\":\"6e47b7d1427cc451e0a7bce8\",\"created_at\":1552317381.6325784,\"correlation_id\":\"tI5i8OY9eC6\",\"enqueued_at\":1552317414.6683447,\"error_message\":\"Invalid argument - connect(2) for \\\"calendar.ivpress.com\\\" port 587\",\"error_class\":\"Errno::EINVAL\",\"failed_at\":1552317381.8407252,\"retry_count\":0}"}
2019-03-11_15:16:54.73312 2019-03-11T15:16:54.732Z 25264 TID-oud4syhs8 WARN: Errno::EINVAL: Invalid argument - connect(2) for "calendar.ivpress.com" port 587
2019-03-11_15:16:54.73323 2019-03-11T15:16:54.733Z 25264 TID-oud4syhs8 WARN: /opt/gitlab/embedded/lib/ruby/2.5.0/net/smtp.rb:539:in `initialize'

Also, from the console:

ActionMailer::Base.smtp_settings

{:authentication=>:plain, :address=>"calendar.ivpress.com", :port=>587, :user_name=>"USER", :password=>"PASSWORD", :domain=>"calendar.ivpress.com", :ca_file=>"/opt/gitlab/embedded/ssl/certs/cacert.pem"}

There are no certs in the :ca_file location. Is this a problem? Why does it include this by default (when there are no certs specified in the configuration file)?

Details of package version

Provide the package version installation details 
ii  gitlab-ee                       11.8.1-ee.0          amd64                GitLab Enterprise Edition (including NGINX, Postgres, Redis)

Environment details

  • Operating System: Debian 9
  • Installation Target, remove incorrect values:
    • Bare Metal Machine
  • Installation Type, remove incorrect values:
    • New Installation (last year)
  • Is there any other software running on the machine: Nextcloud

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb`

Currently:

gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "calendar.ivpress.com"
gitlab_rails['smtp_port'] = 587
gitlab_rails['smtp_user_name'] = "USERNAME"
gitlab_rails['smtp_password'] = "PASSWORD"
gitlab_rails['smtp_domain'] = "calendar.ivpress.com"
gitlab_rails['smtp_authentication'] = "plain"
gitlab_rails['gitlab_email_from'] = 'USERNAME'
gitlab_rails['gitlab_email_reply_to'] = 'USERNAME'

But I have tested with many other permutations from the SMTP docs example page.

I have tried

  • port 465
  • different smtp_authentication values
  • with and without the gitlab_email_from values
  • different smtp_openssl_verify_mode values

Are there other logs I should pursue that would tell me more? Is this an issue with the ruby mailer? Is this an issue with the empty ca_