Adds Service Desk custom email verification services (!120758) · Merge requests · GitLab.org / GitLab

Marc Saleiko requested to merge ms-sd-custom-email-verification-services into master May 15, 2023

Feature context

Click to expand 👇

Right now it is not possible to customize the Service Desk email address (intake and sending) in its entirety. On self-hosted instances you have more control over the used addresses, but you will still have a rather cryptic target email address for a specific service desk in a project. For .com users it's currently not possible to customize the Service Desk email at all.

There is a proposal and a further exploration around this issue. A summary of the solution path is the following: Users set up their custom email to forward all emails to the cryptic Service Desk email and provide SMTP credentials so we can send emails on their behalf. This way customers seeking support will only see the custom email address in their communication.

There is further discussion about improving and changing the general infrastructure, but this approach is a MVC to solve the issue for our customers.

🗺 How does it contribute to the whole feature?

This MR is the second part in a series of MRs that will follow in order to complete this feature. See #329990 (comment 1227384943) for a detailed breakdown. Here's a summary:

✅ Using SMTP credentials. Foundation work. Add Service Desk custom email foundation (!108017 - merged)
🎯 Verify email ownership, correct function and setup Part 8: Add verification services
Ingest replies from custom email
Add settings and validation to Settings page
Add documentation

What does this MR do and why?

🎏 This feature is hidden behind a feature flag and not used on production yet.

This MR adds the logic for the verification part of Customizable email addresses for Service Desk. The ServiceDesk::CustomEmailVerifications::CreateService starts the verification process and tries to send the verification email and ServiceDesk::CustomEmailVerifications::UpdateService checks the incoming verification email and moves the verification entry to finished or failed.

We can only call the services from the console right now. The email ingestion (***EmailHandler is not connected in this MR yet).

Flow for `CreateService`

CreateService starts the verification process, creates ServiceDesk::CustomEmailVerification in started state and tries to send a verification email (directly) with the provided settings in ServiceDesk::CustomEmailCredential.
We notify project owners and the user who triggered the verification.
If an error occurred while sending the email, we'll end the verification process, mark it as failed. We notify project owners and the user about the failure.
If the email was sent, we are done here.

flowchart TD
    A[Start] -->B{CreateService}
    B -->|success| C[Sent verification email]
    B -->|failure| D[End process, send failure email]

Flow for `UpdateService`

UpdateService takes an email as additional parameter.
If the verification email was ingested successfully, we have the email object as a parameter and check for a few things
1. Was the email received within the 30 minutes timeframe?
2. Does the From header match the original sender (this is us. Some service providers do not preserve the original from. We need that for correct author attribution)
3. Is the verification token correct?
4. Only if all the above was successful, we mark the verification as finished and move it to the finished state. If not we move it to failed.
We also notify project owners and the user who triggered the verification.
If we call the UpdateService and we have already verified it, we exit early.
If we call the UpdateService and it has already failed, we exit early.

flowchart TD
    A(Ingested verification email) --> B
    D(Without email) --> B{UpdateService}
    B -->|mark as finished| E(verified)
    B -->|mark as failed| F(verification failed)

Screenshots or screen recordings

🚫 None. backend only

How to set up and validate locally

⚠ As this forcibly tries to send the email via the provided SMTP credentials you can only test this by providing correct SMTP credentials and the service provider must support subaddressing. See the guide on how to set up service desk in GDK step no. 5 that describes how you can get SMTP credentials for a new GMail account (enable 2FA and create an app password). We **don't need email ingestion here via mail_room, instead we use prepared email eml files.

Find a project ID in your installation that you have not used for any Service Desk (including CustomEmailCredentials and CustomEmailVerification testing) setup and testing. Why? It makes these steps a lot easier 🙂
Open the rails console bin/rails c in gitlab folder

Find the project by id

project = Project.find(5) # Where 5 is your project id

Get the root user (as it's the owner of each project)
```
user = User.first
```

Create ServiceDeskSetting entry (which holds Service Desk related meta info and custom email)

settings = ServiceDeskSetting.create!(
  project: project,
  custom_email: 'user@gmail.com' # Use your "real" test email address
)

Create ServiceDesk::CustomEmailCredential

# Use your "real" test email credentials here. If you use a GMail address
# you only need to change the smtp_username and smtp_password
credential = ServiceDesk::CustomEmailCredential.create!(
   project: project,
   smtp_address: 'smtp.gmail.com',
   smtp_port: 587,
   smtp_username: 'user@gmail.com',
   smtp_password: 'supersecret'
 )

Reset associations on project, because we access these objects from ServiceDeskSetting internally via project
```
project.reset
```
Call the CreateService. It should return a ServiceResponse with success. You should see a notification email in Letter Opener and in your custom email mailbox you should find the verification email.
```
ServiceDesk::CustomEmailVerifications::CreateService.new(project: project, current_user: user).execute
```
You can try out error cases, if you change your credentials. In those cases you should not see the verification email in your custom email mailbox, but see two notifications in Letter Opener (1. verification has been triggered and 2. result of the verification --> error)
- Change the smtp_host to something like wrong.gmail.com (error smtp_host_issue)
- Change your password to something wrong etc.

Now we move to UpdateService: For this to work, we need a Mail object with a verification email. Let's take the example from the specs and modify it, so that it contains our defined custom email address:

verification_email_text = <<~EMAIL
   Delivered-To: support+project_slug-project_id-issue-@example.com
   Received: by 2002:a05:7022:aa3:b0:5d:66:2e64 with SMTP id dd35csp3394266dlb; Mon, 23 Jan 2023 08:50:49 -0800 (PST)
   X-Received: by 2002:a19:a40e:0:b0:4c8:d65:da81 with SMTP id q14-20020a19a40e000000b004c80d65da81mr9022372lfc.60.1674492649184; Mon, 23 Jan 2023 08:50:49 -0800 (PST)
   Received: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41]) by mx.google.com with SMTPS id t20-20020a195f14000000b00499004f4b1asor10121263lfb.188.2023.01.23.08.50.48 for <support+project_slug-project_id-issue-@example.com> (Google Transport Security); Mon, 23 Jan 2023 08:50:49 -0800 (PST)
   X-Received: by 2002:a05:6512:224c:b0:4cc:7937:fa04 with SMTP id i12-20020a056512224c00b004cc7937fa04mr1421048lfu.378.1674492648772; Mon, 23 Jan 2023 08:50:48 -0800 (PST)
   X-Forwarded-To: support+project_slug-project_id-issue-@example.com
   X-Forwarded-For: custom-support-email@example.com support+project_slug-project_id-issue-@example.com
   Return-Path: <custom-support-email@example.com>
   Received: from gmail.com ([94.31.107.53]) by smtp.gmail.com with ESMTPSA id t13-20020a1c770d000000b003db0ee277b2sm11097876wmi.5.2023.01.23.08.50.47 for <fatjuiceofficial+verify@gmail.com> (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Jan 2023 08:50:47 -0800 (PST)
   From: Flight Support <custom-support-email@example.com>
   X-Google-Original-From: Flight Support <example@example.com>
   Date: Mon, 23 Jan 2023 17:50:46 +0100
   Reply-To: GitLab <noreply@example.com>
   To: custom-support-email+verify@example.com
   Message-ID: <63d927a0e407c_5f8f3ac0267d@mail.gmail.com>
   Subject: Verify custom email address custom-support-email@example.com for Flight
   Mime-Version: 1.0
   Content-Type: text/plain; charset=UTF-8
   Content-Transfer-Encoding: 7bit
   Auto-Submitted: no
   X-Auto-Response-Suppress: All


   This email will verify the ownership of the enered custom email address and correct functionality of the email forwarder.

   Verification token: ZROT4ZZXA-Y6
   -- 

   You're receiving this email because of your account on 127.0.0.1.
   EMAIL
 # Add our custom email here
 verification_email_text.gsub!('custom-support-email@example.com', settings.custom_email)
 verification_email_text.gsub!('custom-support-email+verify@example.com', settings.custom_email_address_for_verification)
 # Make a Mail object from the text
 verification_mail = Mail.new(verification_email_text)

Now let's update the token of the verification object and use the token from the email:

verification = project.service_desk_custom_email_verification
verification.update!(token: 'ZROT4ZZXA-Y6')
project.reset

Run the UpdateService. and provide the mail object as param. You should see a notification email that says that the email address has been verified in Letter Opener. Additionally the service should return a ServicResponse with success
```
ServiceDesk::CustomEmailVerifications::UpdateService.new(project: project, params: { mail: verification_mail }).execute
```
Now you can play a bit with this to force different error scenarios:
1. For a fresh start you can always reset the verification to started with
```
verification.mark_as_started!(user)
project.reset
```
2. Provide no email would lead to mail_not_received_within_timeframe error with error ServiceResponse
3. Provide a ramp_up_error e.g. like params: { ramp_up_error: 'smtp_host_issue' }, which would lead to smtp_host_issue error with error ServiceResponse
4. Move triggered_at back in time and call the service like in the success example. This would lead to mail_not_received_within_timeframe error with error ServiceResponse
```
verification.mark_as_started!(user)
verification.update!(triggered_at: 40.minutes.ago)
project.reset
```
5. Provide an invalid token or modify the From header to something else to see the corresponding errors.
6. Call the service twice to see that it exits early at the second time without writing to the database

(Optional) clean up the mess and remove entries

settings.destroy
credential.destroy
verification.destroy

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

I have evaluated the MR acceptance checklist for this MR.

Edited May 23, 2023 by Marc Saleiko

Adds Service Desk custom email verification services