Adds custom email verification email ingestion (!123681) · Merge requests · GitLab.org / GitLab

Marc Saleiko requested to merge ms-sd-custom-email-verification-ingestion into master Jun 14, 2023

Feature context

Click to expand 👇

Right now it is not possible to customize the Service Desk email address (intake and sending) in its entirety. On self-hosted instances you have more control over the used addresses, but you will still have a rather cryptic target email address for a specific service desk in a project. For .com users it's currently not possible to customize the Service Desk email at all.

There is a proposal and a further exploration around this issue. A summary of the solution path is the following: Users set up their custom email to forward all emails to the cryptic Service Desk email and provide SMTP credentials so we can send emails on their behalf. This way customers seeking support will only see the custom email address in their communication.

There is further discussion about improving and changing the general infrastructure, but this approach is a MVC to solve the issue for our customers.

🗺 How does it contribute to the whole feature?

This MR is part of a series of MRs that will follow in order to complete this feature. See #329990 (comment 1227384943) for a detailed breakdown. Here's a summary:

✅ Using SMTP credentials. Foundation work. Add Service Desk custom email foundation (!108017 - merged)
🎯 Verify email ownership, correct function and setup Part 9: Add custom email verification email ingestion
Ingest replies from custom email
Add settings and validation to Settings page
Add documentation

What does this MR do and why?

🎏 This feature is hidden behind a feature flag and not used on production yet.

In Adds Service Desk custom email verification ser... (!120758 - merged) we added the services that actually handle the verification. The missing bit is, that we intercept the email ingestion and instead of adding a new issue, we push the email to the ServiceDesk::CustomEmailVerifications::UpdateService to handle the second part of custom email verification for Service Desk.

As both incoming_email and service_desk_email in the end (although using different Receiver classes) use Gitlab::Email::Handler::ServiceDeskHandler, I added this here.

Screenshots or screen recordings

🚫 backend only

How to set up and validate locally

You can also directly ingest emails if you want to. To keep things simple we mimic the ingestion and instead call the right email receiver class that then calls ServiceDeskHandler and that should process the verification email (like in the test cases).

Find a project ID in your installation that you have not used for any Service Desk (including CustomEmailCredentials and CustomEmailVerification testing) setup and testing. Why? It makes these steps a lot easier 🙂
Open the rails console bin/rails c in gitlab folder

Find the project by id

project = Project.find(5) # Where 5 is your project id

Get the root user (as it's the owner of each project)
```
user = User.first
```

Now add all of the records (ServiceDeskSetting, credentials and verification) so that we are ready to go

settings = ServiceDeskSetting.create!(
  project: project,
  custom_email: 'user@example.com'
)
credential = ServiceDesk::CustomEmailCredential.create!(
  project: project,
  smtp_address: 'smtp.gmail.com',
  smtp_port: 587,
  smtp_username: 'user@example.com',
  smtp_password: 'supersecret'
 )
 verification = ServiceDesk::CustomEmailVerification.create!(
   project: project,
   state: 'started',
   token: 'ZROT4ZZXA-Y6',
   triggerer: user,
   triggered_at: Time.current 
 )
 project.reset

We use the raw email text for the receiver, so let's stub a verification email

verification_email_text = <<~EMAIL
   Delivered-To: support+project_slug-project_id-issue-@example.com
   Received: by 2002:a05:7022:aa3:b0:5d:66:2e64 with SMTP id dd35csp3394266dlb; Mon, 23 Jan 2023 08:50:49 -0800 (PST)
   X-Received: by 2002:a19:a40e:0:b0:4c8:d65:da81 with SMTP id q14-20020a19a40e000000b004c80d65da81mr9022372lfc.60.1674492649184; Mon, 23 Jan 2023 08:50:49 -0800 (PST)
   Received: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41]) by mx.google.com with SMTPS id t20-20020a195f14000000b00499004f4b1asor10121263lfb.188.2023.01.23.08.50.48 for <support+project_slug-project_id-issue-@example.com> (Google Transport Security); Mon, 23 Jan 2023 08:50:49 -0800 (PST)
   X-Received: by 2002:a05:6512:224c:b0:4cc:7937:fa04 with SMTP id i12-20020a056512224c00b004cc7937fa04mr1421048lfu.378.1674492648772; Mon, 23 Jan 2023 08:50:48 -0800 (PST)
   X-Forwarded-To: support+project_slug-project_id-issue-@example.com
   X-Forwarded-For: custom-support-email@example.com support+project_slug-project_id-issue-@example.com
   Return-Path: <custom-support-email@example.com>
   Received: from gmail.com ([94.31.107.53]) by smtp.gmail.com with ESMTPSA id t13-20020a1c770d000000b003db0ee277b2sm11097876wmi.5.2023.01.23.08.50.47 for <fatjuiceofficial+verify@gmail.com> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 08:50:47 -0800 (PST)
   From: Flight Support <custom-support-email@example.com>
   X-Google-Original-From: Flight Support <example@example.com>
   Date: Mon, 23 Jan 2023 17:50:46 +0100
   Reply-To: GitLab <noreply@example.com>
   To: custom-support-email+verify@example.com
   Message-ID: <63d927a0e407c_5f8f3ac0267d@mail.gmail.com>
   Subject: Verify custom email address custom-support-email@example.com for Flight
   Mime-Version: 1.0
   Content-Type: text/plain; charset=UTF-8
   Content-Transfer-Encoding: 7bit
   Auto-Submitted: no
   X-Auto-Response-Suppress: All


   This email will verify the ownership of the enered custom email address and correct functionality of the email forwarder.

   Verification token: ZROT4ZZXA-Y6
   -- 

   You're receiving this email because of your account on 127.0.0.1.
   EMAIL
 # Add our custom email here
 verification_email_text.gsub!('custom-support-email@example.com', settings.custom_email)
 verification_email_text.gsub!('custom-support-email+verify@example.com', settings.custom_email_address_for_verification)

Now change To address to incoming_email and kick off receiver

incoming_to_address = Gitlab::Email::IncomingEmail.reply_address("#{project.full_path_slug}-#{project.id}-issue-")
verification_email_text.gsub!('support+project_slug-project_id-issue-@example.com', incoming_to_address)
Gitlab::Email::Receiver.new(verification_email_text).execute

This should create an issue as the feature flag has not been enabled yet

Enable the feature flag and ingest the email text again. The verification should now be marked as finished

Feature.enable(:service_desk_custom_email, project)
Gitlab::Email::Receiver.new(verification_email_text).execute
verification.reload

Now let's do the same for service_desk_email. We reset the verification, exchange the To address in the email text and ingest the email text again.

verification.mark_as_started!(user)
verification.update!(token: 'ZROT4ZZXA-Y6')
service_desk_to_address = Gitlab::Email::ServiceDeskEmail.address_for_key("#{project.full_path_slug}-#{project.id}-issue-")
verification_email_text.gsub!(incoming_to_address, service_desk_to_address)
Gitlab::Email::ServiceDeskReceiver.new(verification_email_text).execute
verification.reload

If you want to test different cases, you can now use a different token, move the triggered_at time or change the custom_email address. This will produce errors on the verification record.

(Optional) clean up the mess and remove entries

settings.destroy
credential.destroy
verification.destroy

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

I have evaluated the MR acceptance checklist for this MR.

Edited Jun 15, 2023 by Marc Saleiko

Adds custom email verification email ingestion