SMTP Mail not working

I try to configure gitlab to send mail via smtp using exhchange server config part: gitlab_rails['smtp_enable'] = true

 gitlab_rails['smtp_address'] = "mail.domain.com"
 gitlab_rails['smtp_port'] = 587
 #gitlab_rails['smtp_user_name'] = "gitlab_mail_user"
 #gitlab_rails['smtp_password'] = "secret_password"
 gitlab_rails['smtp_domain'] = "domain.com"
 gitlab_rails['smtp_authentication'] = "login"
 gitlab_rails['smtp_enable_starttls_auto'] = true

When i connect to console and try to send test mail i get error:

Traceback (most recent call last):
        2: from (irb):8
        1: from (irb):8:in `rescue in irb_binding'
OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: wrong version number)

another thing to mention is that i get error 500 on user profile setting by clicking sent confirmation mail.

iam using omnibus version of gitlab:

Components 
GitLab 12.1.0 (295480f4553)
GitLab Shell 9.3.0
GitLab Workhorse v8.7.0
GitLab API v4
Ruby 2.6.3p62
Rails 5.2.3
PostgreSQL 10.7
Gitaly Servers