Add webhook delivery method to mailroom
What does this MR do?
For gitlab-com/gl-infra/scalability#1458 (closed)
Mailroom is a component that constantly listens to configured mailboxes to handle incoming emails and service desk emails. When an email arrives, the component delivers the email content (*.eml file) to the Rails applications for processing. At the moment, mailroom is pushing the email content directly into Redis instance of Sidekiq without going through the middleware layer. This flow creates multiple known issues. In gitlab-com/gl-infra&596 (closed), we are trying to migrate to use webhook instead. With the new mechanism, mailroom processes trigger a HTTP request to a Rails webservice internal endpoint. This endpoint does some preparation stuff and then pushes the email content to Sidekiq in a normal way.
The HTTP requests are authenticated with JWT. The mailroom process generates a JWT token based on a secret fetched from a secret file. Rails internal endpoints uses the same secret to decode the JWT tokens of incoming requests. This practice was applied in Shell, Kas, and Pages before.
During the migration, we need to support both methods. This MR adds deliveryMethod (either sidekiq or webhook) to both incomingEmail and serviceDeskEmail app configuration. The default method sidekiq. When switching to new webhook method, the following things happen:
-
shared-secretjob generates the auth tokens for both mailboxes if the secret is not specified - Those secrets are mounted to mailroom deployment and
- Mailroom configuration is switched to use new postback with JWT. The following settings are compatible with the internal HTTP endpoint.
:delivery_method: postback
:delivery_options:
:delivery_url: '{{ template "gitlab.workhorse.url" $ }}/api/v4/internal/mail_room/incoming_email'
:jwt_auth_header: "Gitlab-Mailroom-Api-Request"
:jwt_issuer: "gitlab-mailroom"
:jwt_algorithm: "HS256"
:jwt_secret_path: "/etc/gitlab/mailroom/incoming_email_webhook_secret"- The secrets are mounted to webservice deployment.
- The Rails webservice's
gitlab.ymlis modified to include the path of the secret.
Testing plan
- Setup IncomingEmail and ServiceDesk:
diff --git a/values.yaml b/values.yaml
index d87c2c9d3..db8beb9a9 100644
--- a/values.yaml
+++ b/values.yaml
@@ -349,15 +349,15 @@ global:
## https://docs.gitlab.com/charts/installation/command-line-options.html#incoming-email-configuration
## https://docs.gitlab.com/charts/charts/gitlab/mailroom/index.html#incoming-email
incomingEmail:
- enabled: false
- address: ""
+ enabled: true
+ address: "my-personal-email+%{key}@gmail.com"
host: "imap.gmail.com"
port: 993
ssl: true
startTls: false
- user: ""
+ user: "my-personal-email@gmail.com"
password:
- secret: ""
+ secret: "incoming-email-password"
key: password
expungeDeleted: false
logger:
@@ -368,22 +368,22 @@ global:
clientSecret:
key: secret
pollInterval: 60
- deliveryMethod: sidekiq
+ deliveryMethod: webhook
authToken: {}
# secret:
# key:
## https://docs.gitlab.com/charts/charts/gitlab/mailroom/index.html#service-desk-email
serviceDeskEmail:
- enabled: false
- address: ""
+ enabled: true
+ address: "my-personal-email+%{key}@gmail.com"
host: "imap.gmail.com"
port: 993
ssl: true
startTls: false
- user: ""
+ user: "my-personal-email@gmail.com"
password:
- secret: ""
+ secret: "incoming-email-password"
key: password
expungeDeleted: false
logger:
@@ -394,7 +394,7 @@ global:
clientSecret:
key: secret
pollInterval: 60
- deliveryMethod: sidekiq
+ deliveryMethod: webhook
authToken: {}
# secret:
# key:
@@ -704,10 +704,10 @@ upgradeCheck:
cpu: 50m- Turn on Service Desk Email in a project's setting (Project Settings > General)
- Compose an email using Gmail client to the above service desk email.
- Wait for a while, the service desk email should create a corresponding issue:
- Filter webservice logs for internal endpoints, we should find one line about
api/v4/internal/mail_room/incoming_email, indicating that the email is handled by the webhook.
gitlab-local-webservice-default-f747674d6-zwgt5 gitlab-workhorse 2022-01-24T12:07:14.932067000+07:00 {"content_type":"application/json","correlation_id":"01FT57PAT5BN3H7WMY83RBQV1W","duration_ms":45,"host":"gitlab-local-webservice-default.default.svc:8181","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"172.17.0.1:18638","remote_ip":"172.17.0.1","route":"^/api/","status":200,"system":"http","time":"2022-01-24T05:07:14Z","ttfb_ms":45,"uri":"/api/v4/internal/mail_room/incoming_email","user_agent":"Faraday v1.9.3","written_bytes":16}
gitlab-local-webservice-default-f747674d6-zwgt5 webservice 2022-01-24T12:07:15.721857000+07:00 Started POST "/api/v4/internal/mail_room/incoming_email" for 172.17.0.1 at 2022-01-24 05:07:14 +0000
gitlab-local-webservice-default-f747674d6-zwgt5 webservice 2022-01-24T12:07:15.721971600+07:00 {"time":"2022-01-24T05:07:14.929Z","severity":"INFO","duration_s":0.01702,"db_duration_s":0.0,"view_duration_s":0.01702,"status":200,"method":"POST","path":"/api/v4/internal/mail_room/incoming_email","params":[{"key":"Delivered-To: darkwingbt root-test-service-desk-3-issue-@gmail.com\r\nReceived: by 2002:a4a:8546:0:0:0:0:0 with SMTP id l6csp1603329ooh;\r\n Sun, 23 Jan 2022 21:07:09 -0800 (PST)\r\nX-Received: by 2002:a17:90a:f3c6:: with SMTP id ha6mr334805pjb.28.1643000828927;\r\n Sun, 23 Jan 2022 21:07:08 -0800 (PST)\r\nARC-Seal: i","value":"1; a=rsa-sha256; t=1643000828; cv=none;\r\n d=google.com; s=arc-20160816;\r\n b=tA9wwpWkVd8lLq64sANvO38jhpIdVN5cqN15pPLJ0ZjSmEbz2HyHN1AiML2yOlG81h\r\n 3 4Wt/PJSrA2LQWlyM2afjhjPXYtUMcYanPmdgQmT7m9cXxV4SQI0Yo9yRN9bUhT2npI\r\n iiS0iqhBQ72aV9hLEWo5ch2/5cOgEgCki62zhMvlYzu3Rq0WV8bG2W4 l48CDcOnp0PS\r\n Vp4EWeOV0b5EpAqBUnQ80D4LjTq63kRHoCrv0bb8FW03Y2L3sCuQe5q8yKitgS5ljVzq\r\n 9J5pD 03IYPkErgPaiPb900g8cSxbdBrtNPhqu8aJBSYdmXZ4bdVVb7SrCKUSDQ6LJv7\r\n 3M/w==\r\nARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;\r\n h=to:subject:message-id:date:from:mime-version:dkim-signature;\r\n bh=75imEutLjiLJNfGxt B6ORQVwEqONPBMfuD5EPvYmHA=;\r\n b=jtufR3hKtNjbEU9iKaKYNtK5XiEcGGcpZ6mYx2iRArq5q1e/XuO67iO9rMPkVhVl9m\r\n TlWb//12OX/t9ELmj3OT0VIpKnM1RW0rE4sUb4gFFydrg9/TS BKpLMJ2UVYf76ooFic\r\n KWK9d//HJ RuMjbcAEyInB1W9EDOwiidhVVCAJ/Vkp63RUegPBjEBMUfgAmWjGsmylFG\r\n PdeQRj9ntk1YjakFhF0t8yi/c2X3ICkqEfngqOCj7h4DzJbsptD8NvMb5ANaTS46fSwl\r\n 5q1jn75yj9FO627GuiP9IVNthSGqQsN crHMpM2KvUl5juZYOAxBwt5OArSKu/kj5Tbe\r\n cdEg==\r\nARC-Authentication-Results: i=1; mx.google.com;\r\n dkim=pass header.i=@gitlab.com header.s=google header.b=LUSvOy2D;\r\n spf=pass (google.com: domain of qmnguyen@gitlab.com designates 209.85.220.41 as permitted sender) smtp.mailfrom=qmnguyen@gitlab.com;\r\n dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=gitlab.com\r\nReturn-Path: <qmnguyen@gitlab.com>\r\nReceived: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41])\r\n by mx.google.com with SMTPS id x21sor5319583pln.87.2022.01.23.21.07.08\r\n for <darkwingbt root-test-service-desk-3-issue-@gmail.com>\r\n (Google Transport Security);\r\n Sun, 23 Jan 2022 21:07:08 -0800 (PST)\r\nReceived-SPF: pass (google.com: domain of qmnguyen@gitlab.com designates 209.85.220.41 as permitted sender) client-ip=209.85.220.41;\r\nAuthentication-Results: mx.google.com;\r\n dkim=pass header.i=@gitlab.com header.s=google header.b=LUSvOy2D;\r\n spf=pass (google.com: domain of qmnguyen@gitlab.com designates 209.85.220.41 as permitted sender) smtp.mailfrom=qmnguyen@gitlab.com;\r\n dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=gitlab.com\r\nDKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;\r\n d=gitlab.com; s=google;\r\n h=mime-version:from:date:message-id:subject:to;\r\n bh=75imEutLjiLJNfGxt B6ORQVwEqONPBMfuD5EPvYmHA=;\r\n b=LUSvOy2DP5q0UjdjtBc6A fz/9vl9Jnj HbK1ZuUa6P2NyeD1LSWcrU1B0su moZFd\r\n WixIBVxTgSDo4lBMQGcr609vux6igqXMV1QADQf9lyanAiZHDArkTG0FGdpnC/7AB0Ag\r\n J8/WhH Vu6 4pVILRDPNwgM6anWcnDwCzbUM=\r\nX-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;\r\n d=1e100.net; s=20210112;\r\n h=x-gm-message-state:mime-version:from:date:message-id:subject:to;\r\n bh=75imEutLjiLJNfGxt B6ORQVwEqONPBMfuD5EPvYmHA=;\r\n b=n56uTVJ7XlJ6s6FO9IR/dH30jxeSn8U5EHGMHMmDVcWIpRxvbxXxz7jhQ28K7GnCsb\r\n vpAYQofuW gtrHBN/5ju7KydSWnPxCzQyQ3q W4bqWzptNilKR7rx2kL WkI3xKcqzAp\r\n ZpkLa0rEWaO5tPKZa5MNKSDqXCks6mqf/Gw78tIr81j8A/ztijBGWJKR2ix7t GJn6x5\r\n v7EHMCZ7uOjNDNSYMAoGiJwViD3uuYyxaid//khR3nVq9qs3YahtftH8d c Qc5m9/vv\r\n DVRCGHtRjxbjDfSw/sX5iYCWExHJRSlAYBPQ6RhzcjHvguzcK1rQadw25uzY3t2V4uGb\r\n OPmQ==\r\nX-Gm-Message-State: AOAM531SlGRzl6cjhiP6DW 29nMP51ONn2p7X9I0CZ2/ yV75El6h2/7\r\n\tN 1hHgKTtxg0PX2uPFVyiYvCfXm08ImzoihnzMeawoGZgXU=\r\nX-Google-Smtp-Source: ABdhPJwz8JpAskkvdpxXKVZVVjrlDh6ytipKNrSvFl5cFz5uqvClw0gof7BQOnlzKxTzidWQNjGo2kgawm/7wgzwhNg=\r\nX-Received: by 2002:a17:902:e84e:b0:14b:3550:e892 with SMTP id\r\n t14-20020a170902e84e00b0014b3550e892mr7151736plg.75.1643000828241; Sun, 23\r\n Jan 2022 21:07:08 -0800 (PST)\r\nMIME-Version: 1.0\r\nFrom: Quang-Minh Nguyen <qmnguyen@gitlab.com>\r\nDate: Mon, 24 Jan 2022 12:06:57 0700\r\nMessage-ID: <CAH=ftaV63_25LtRTvw1mwtYaTft3L fEochuhqDE-kyu52=yAg@mail.gmail.com>\r\nSubject: Test service desk email\r\nTo: darkwingbt root-test-service-desk-3-issue-@gmail.com\r\nContent-Type: multipart/alternative; boundary=\"000000000000bc5a2c05d64cf0d3\"\r\n\r\n--000000000000bc5a2c05d64cf0d3\r\nContent-Type: text/plain; charset=\"UTF-8\"\r\n\r\nTest service desk email content.\r\n\r\n--000000000000bc5a2c05d64cf0d3\r\nContent-Type: text/html; charset=\"UTF-8\"\r\n\r\n<div dir=\"ltr\">Test service desk email content.</div>\r\n\r\n--000000000000bc5a2c05d64cf0d3--\r\n"}],"host":"gitlab-local-webservice-default.default.svc","remote_ip":"172.17.0.1","ua":"Faraday v1.9.3","route":"/api/:version/internal/mail_room/*mailbox_type","queue_duration_s":0.025529,"redis_calls":1,"redis_duration_s":0.00063,"redis_read_bytes":44,"redis_write_bytes":5894,"redis_queues_calls":1,"redis_queues_duration_s":0.00063,"redis_queues_read_bytes":44,"redis_queues_write_bytes":5894,"db_count":1,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":1,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.004,"cpu_s":0.033472,"mem_objects":14934,"mem_bytes":1892543,"mem_mallocs":4015,"mem_total_bytes":2489903,"pid":32,"correlation_id":"01FT57PAT5BN3H7WMY83RBQV1W","meta.caller_id":"POST /api/:version/internal/mail_room/*mailbox_type","meta.remote_ip":"172.17.0.1","meta.feature_category":"service_desk","meta.client_id":"ip/172.17.0.1","content_length":"4706","request_urgency":"default","target_duration_s":1}Related issues
- Add JWT support to mail_room gem's postback strategy: https://github.com/tpitale/mail_room/pull/134
- Implement an internal endpoint to handle webhook: gitlab-org/gitlab!76724 (merged)
- CNG's mailroom version bumps:
- v0.0.18 that includes the JWT stuff: gitlab-org/build/CNG!854 (merged)
- v0.0.19 that fixes a small big of mail_room gem: gitlab-org/build/CNG!867 (merged)
Checklist
See Definition of done.
For anything in this list which will not be completed, please provide a reason in the MR discussion.
Required
-
Merge Request Title and Description are up to date, accurate, and descriptive -
MR targeting the appropriate branch -
MR has a green pipeline on GitLab.com
Expected (please provide an explanation if not completing)
-
Test plan indicating conditions for success has been posted and passes -
Documentation created/updated -
Tests added -
Integration tests added to GitLab QA -
Equivalent MR/issue for omnibus-gitlab opened
Edited by Mitchell Nielsen