Validate that SMTP settings do not enable both TLS and STARTTLS

What does this MR do?

GitLab versions 15.10.4 and up shipped with gitlab-org/gitlab!116925 (merged) in order to fix a Ruby 3 upgrade issue that made it impossible to disable STARTTLS if the SMTP server advertised support for it.

However, this change enforced the constraint that both SMTP TLS and STARTTLS cannot be enabled simultaneously. This follows the behavior of the net-smtp gem, which will raise an exception if a user attempts to enable both.

This constraint was added because as described in https://github.com/mikel/mail/pull/1536, the logic for enabling/disabling TLS and/or STARTTLS is a bit tricky to get right and missed some important edge cases.

This commit adds a validation step that will throw an error if both global.smtp.tls and global.smtp.enable_starttls_auto are both set to true.

Previously if SMTP TLS were enabled, STARTTLS was disabled outright.

Related issues

Relates to gitlab-org/gitlab#409835 (closed)

Omnibus MR: gitlab-org/omnibus-gitlab!6863 (merged)

Testing

With both true:

  --set global.smtp.tls=true \
  --set global.smtp.enable_starttls_auto=true

✅ Not allowed

With only one true:

  --set global.smtp.tls=false \
  --set global.smtp.enable_starttls_auto=true

or

  --set global.smtp.tls=true \
  --set global.smtp.enable_starttls_auto=false

✅

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion.

Required

• Merge Request Title and Description are up to date, accurate, and descriptive
• MR targeting the appropriate branch
• MR has a green pipeline on GitLab.com
• When ready for review, MR is labeled "~workflow::ready for review" per the Distribution MR workflow

Expected (please provide an explanation if not completing)

• Test plan indicating conditions for success has been posted and passes
• Documentation created/updated
• Tests added
• Integration tests added to GitLab QA
• Equivalent MR/issue for omnibus-gitlab opened
• Validate potential values for new configuration settings. Formats such as integer 10, duration 10s, URI scheme://user:passwd@host:port may require quotation or other special handling when rendered in a template and written to a configuration file.