GitLab.org / GitLab Community Edition

Added bug label

/cc @marin

Thanks for diving into this Ben!

@marin Can you look into this?

@bbodenmiller when this was developed few versions ago I had it working correctly on the dev environment, actions showed up in my test inbox and all the checks passed, so I don't know if something changed in the meantime. If you check the rake task, it generates the email with dummy data that gets sent for whitelisting.

My suspicion was that the dummy email was the issue as I got asked if the email was sent from the production server. This was the case but the data was not real to prevent user data exposure.

BTW, setting up environment to try this out is a very big hassle so you will need some patience to get it up and running.

I think we should try again with a real email (no dummy data). That might help or we might just get lucky.

@marin can you remove all dummy date from GitLab completely (to prevent anyone submitting dummy data again or having the suspicion that we used dummy data) and generate an email based on production data? I think we should try again.

@sytses If I remove dummy data I can remove the whole rake task as it is built around the dummy data. Updating the task will mean rewriting it completely. It can be done when attempting this task again but keep in mind that setting up environment for this is extremely time consuming(unless something changed in the meantime ofcourse). Which one do you prefer?

@marin Can we just enable structured text emails in 7.14 for everyone and remove the rake task?

Can we just enable structured text emails in 7.14 for everyone

If this means removing the plain text emails and only have structured text emails then a new issue might be a better place for this discussion.

and remove the rake task

Rake task can be removed even now in my opinion as it doesn't bring anything useful and might even create issues for users.

If Google approved us, would we remove plan text emails?

@sytses No but I was confused with your question as we already have the google required stuff in our emails.

@marin OK, can we just send one of these emails to Google then?

And OK to remove the rake task.

@sytses Some work still needs to be done as it seems that something broke the rendering of the required data like Ben mentioned at the description of this issue.

I can add this to my TODO list if you think we should try applying again during 7.14 milestone.

Edited 2015-07-17 14:17:05 UTC

@marin Yes, I think we should try that. This feature would be awesome to have.

Reassigned to @marin

Milestone changed to 7.14

At the moment I am trying to understand why some emails get rendered properly and some emails have the html escaped which will cause equal to be escaped, eg.<div class=3D'content'> .

After some investigation I am suspecting the issue to be at Content-Transfer-Encoding which gets set to quoted-printable by default. github emails are forced to 7-bit.

On a test machine I have setup, forcing this on all emails made a change but on a production machine that doesn't seem to be the case.

At the moment I am just trying to see how to go around it because 7-bit doesn't seem like something we could use due to our Emails on push service and the limitation 7-bit brings in the form of maximum number of characters(1000).

Some nice info about content transfer encoding I found on this SO article.

Edited 2015-07-21 13:06:05 UTC

Well this is/was interesting. I found a character in layout/notify.html that I suspect caused the breakage for the mails rendering. This line in notify.html is not a backslash dash but it is \�~@~T .

After I've replaced that line the email gets rendered without escaping and email markup passes.

@marin wow, interesting indeed

It is easy to replace the broken character with something, bit more difficult to make it look good but ultimately the question now is what happens if the email contains a comment with non-ascii characters.

Curious because forcing content transfer encoding might cause issues.

@marin I expect issue with Chinese text on Dev.gitlab.org now :)

@sytses :)

That already happened and the results are (sadly) expected. The Content-Transfer-Encoding got set to quoted-printable and the output of the email looks broken:

----==_mimepart_55ae4f796e99_1a8f5842b0876c8
Content-Type: text/plain;
 charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Marin pushed to branch master at Marin / GitLab


Commits:
e043265a by Marin Jankovski at 2015-07-21T13:56:07Z
ee?
- - - - -


1 changed file:

- README.md


Changes:

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
README.md
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

Now I need to go back to (more) reading.

Edited 2015-07-21 13:59:51 UTC

@marin OK

After some more investigation it looks like this is a situation which cannot be easily fixed(if it can be fixed at all).

However, I do not think it is even required to fix it for cases where there are non-ascii characters. Looking at some other emails I received it is expected to receive an email with 7bit content-transfer-encoding with ascii chars and quoted-printable when there are some non-ascii chars.

What I will do now is:

• Change the char that caused the failures on all emails => In !1024
• remove the rake task and the doc pointing to the rake task => !1024
• Go through the required steps from the google documentation and confirm all is in order
• prepare a new request and apply
• reconsider how to radiate this information for other GitLab users(maybe only document the process)

Edited 2015-07-22 10:03:00 UTC

mentioned in merge request !1024

I wanted to avoid setting up the environment in which to test this by using the tip from registering doc:

All schemas you send to yourself (from x@gmail.com to x@gmail.com) will be displayed in Google products.

This is plain not true. Email sent this way won't be signed by DKIM and the action won't be shown.

Sadly, I will have to go through setting up the whole environment for this once again :/

@sytses I would need admin access for email account used in dev.gitlab.org and possibly to the DNS settings of dev.gitlab.org so not sure how easier that would be considering I might interrupt work during release for everyone

Since I did this before this time round it seems to go faster apart from setting up dkim verification which is not working as google is not generating the record yet. Possibly have to wait for the DNS records to propagate since everything was recently setup.

@marin OK, glad to hear it is going faster this time

The environment is up and running, the emails get sent with the expected headers and pass the spf and dkim check. However, The emails also pass the email markup check but at the moment the emails I receive on the account are not showing the button.

At the moment I am debugging why this is the case.

After some fiddling around(with the dev env) I got this to work:

so I think I can proceed further

@marin Yay!

mentioned in merge request !1024

I've asked for the MR to be included in the next patch release so we can have this sooner on GitLab.com so we can apply as soon as possible.

mentioned in commit b7be08c5

Still no luck on GitLab.com emails:

@bbodenmiller We haven't applied yet. I will update this issue once I do that.

Milestone changed to 8.0

agreed on 8.0 with @marin

@marin Can you look into this one again? Is it listed for any next patch release?

@sytses I need to handle bureaucracy behind it and apply with google. Hope to have this ready to send this week.

Due to our internal infrastructure work that is blocking the task I was working at I can move this to the top of my todo list.

The Google Registering procedure is laid out in google developer docs.

Edited 2015-09-08 08:19:10 UTC

Registration Guidelines:

• Emails must be authenticated via DKIM or SPF

From current email:

Authentication-Results: mx.google.com;
       spf=pass (google.com: domain of bounce+cd3a2c.947b4-marin=gitlab.com@mg.gitlab.com designates 198.61.254.240 as permitted sender) smtp.mailfrom=bounce+cd3a2c.947b4-marin=gitlab.com@mg.gitlab.com;
       dkim=pass header.i=@mg.gitlab.com

Edited 2015-09-08 08:29:47 UTC

• The top-level domain (TLD) of the SPF check or DKIM signature must match the TLD of your From: email address.

e.g. if you use From: foo@bar.com the DKIM or SPF must be for bar.com or sub.bar.com

We send email from gitlab AT gitlab.com and our dkim and spf are from mg.gitlab.com.

Edited 2015-09-08 08:34:34 UTC

• Gmail only checks the SPF on the domain of your Return-Path: email. If you only rely on SPF and not DKIM your Return-Path: email must match the TLD of your From: email.

This is fulfilled too, our emails contain:

mailed-by:  mg.gitlab.com
signed-by:  mg.gitlab.com

Edited 2015-09-08 08:39:04 UTC

• Emails must come from a static email address, e.g. foo@bar.com

We have a static email address.

Edited 2015-09-08 08:40:07 UTC

• Emails must follow the Gmail Bulk Sender Guidelines

We are not sending emails in bulk directly, we send it one by one to mailgun. After that it is handled by mailgun.

Edited 2015-09-08 09:41:55 UTC

• Consistent history of sending a high volume of mail from your domain (order of hundred emails a day minimum to Gmail) for a few weeks at least.

Need to check if I can find this out. => Confirmed in this comment

Edited 2015-09-09 08:54:50 UTC

• A very very low rate of spam complaints from users.

Don't know how to check this, possibly when I get access to mailgun. Also, this doesn't say much, is 1k low on 100k sent emails ??

=>

Confirmed in this comment

Edited 2015-09-09 08:55:22 UTC

• The highest-fidelity action available should be used. For example, if an interaction can be achieved by an In-App Action (One-Click, RSVP, Review), that must be used. For more complex interactions, Go-To Actions can be used.

Examples of actions are shown in this doc.

• One-Click Action => Not applicable to us as our actions are not to approve, confirm and acknowledge something
• RSVP Action => Not applicable
• Review action => Not applicable
• Go To action => We want this because we need to take the user to the comment user received for Commit, Merge request or Issue.

Edited 2015-09-08 10:00:07 UTC

• Actions should be used for transactional mail where a high interaction rate is expected. They should not be used on promotional bulk mail.

This is why we are trying to add this :)

Edited 2015-09-08 11:51:04 UTC

• Go-To Actions:

Must deep link into the specific page on which the action can be performed.

Url will look like https://gitlab.com/gitlab-org/gitlab-ce/issues/2446#note_2064826

Label of button needs to reflect clear action to be taken and must be true to page the user is going to. Label of action should not contain punctuation or all caps. Must be short and concise.

The label will show View Issue, View Merge request or View Commit.

We are currently only approving Go-To actions for very specific high-value usecase with high interaction rate (e.g. Flight Check-in, Shipment tracking links).

I hope they approve us.

Low failure rate and fast response for services handling Action Requests.

We try to keep GitLab.com online.

Edited 2015-09-09 08:56:18 UTC

https://www.google.com/webmasters/markup-tester/u/0/ shows no errors when an email from GitLab.com is inserted in the tester.

After I confirm that the 2 remaining checkboxes are fine, I need to submit a request using this Google Form.

I need to submit the form 3 times, because we are applying for Issue, Merge Request and Commit.

Listing what I will input in certain fields:

• What does your company do? What is its main goal or product? *

GitLab.com is a free git repository hosting service. GitLab.com is a SaaS that runs GitLab Enterprise Edition which is a superset of open-source GitLab Community Edition. GitLab.com offers unlimited private and public repositories for an unlimited number of public or private collaborators, completely free of charge.

• Describe the email to which you plan to add schema. What is its purpose?

For Issues:

Project can have multiple collaborators. 
Each collaborator can participate in a discussion in an Issue, Merge request or Commit.
Collaborator can decide to follow or not follow discussions by setting preference in their user profile. 
If they chose to follow discussions, they can participate in a discussion by commenting or by being mentioned in the discussion.
Collaborartor can also choose to receive notifications in an Issue where they are not mentioned by subscribing to receive email notifications.
Similarly, they can opt out of receiving notification for an Issue where they are a collaborator.
Depending on the user-set preference as explained above, collaborator will receive an email notification for discussions.
Email notification sent to the collaborator will contain the latest comment that triggered the notification. 
With the Gmail Action, collaborator should receive the email with Go-To action with label "View Issue". "View Issue" has a link that takes the collaborator directly to the comment that triggered the notification.

For Merge requests:

Project can have multiple collaborators. 
Each collaborator can participate in a discussion in an Issue, Merge request or Commit.
Collaborator can decide to follow or not follow discussions by setting preference in their user profile. 
If they chose to follow discussions, they can participate in a discussion by commenting or by being mentioned in the discussion.
Collaborator can also choose to receive notifications in a Merge request where they are not mentioned by subscribing to receive email notifications.
Similarly, they can opt out of receiving notification for a Merge request where they are a collaborator.
Depending on the user-set preference as explained above, collaborator will receive an email notification for discussions.
Email notification sent to the collaborator will contain the latest comment that triggered the notification. 
With the Gmail Action, collaborator should receive the email with Go-To action with label "View Merge Request". "View Merge Request" has a link that takes the collaborator directly to the comment that triggered the notification.

For Commit:

Project can have multiple collaborators. 
Each collaborator can participate in a discussion in an Issue, Merge request or Commit.
Collaborator can decide to follow or not follow discussions by setting preference in their user profile. 
If they chose to follow discussions, they can participate in a discussion by commenting or by being mentioned in the discussion.
Depending on the user-set preference as explained above, collaborator will receive an email notification for discussions.
Email notification sent to the collaborator will contain the latest comment that triggered the notification. 
With the Gmail Action, collaborator should receive the email with Go-To action with label "View Commit". "View Commit" has a link that takes the collaborator directly to the comment that triggered the notification.

Edited 2015-09-08 15:07:27 UTC

send a real-life email coming from your production servers

I will use this issue for "View Issue" action.

I will use MR !1024 for "View Merge request" action.

I will use aee579ea (comment 2069770) for "View Commit"

Edited 2015-09-08 15:06:51 UTC

@marin Is all this code live on GitLab.com? (I got that idea from gitlab-org/gitlab-ee#9 (comment 2066925))

Edited 2015-09-09 01:41:51 UTC

@sytses Yes, looking at the registering guidelines all needed changes are already live.

Ok, I have access to mailgun so I am revisiting:

• A very very low rate of spam complaints from users.

We had 3 complaints for the past 30 days for several hundred thousand emails we've sent. Over the period of 6 months where we've sent several hundred thousand emails over one million we had 35 spam complaints. I would call this a very, very low rate of spam complaints. We also have an ongoing effort to make this even lower by adding user report option.

Edited 2015-09-09 08:45:55 UTC

• Consistent history of sending a high volume of mail from your domain (order of hundred emails a day minimum to Gmail) for a few weeks at least.

Daily we send anything between several thousand to a 13 thousand emails. Weekends being lower than workdays ofcourse. This has been the trend for the past 3 months at least. I would say we are fulfilling this easily too.

Edited 2015-09-09 08:51:55 UTC

According to all questions and answers given above I am confident that we are ready to apply to Google for this. I will start filling in the registration form and send the request.

All 3 requests have been submitted. With all 3 requests, I've sent a real live email using rails console on GitLab.com.

Edited 2015-09-09 09:37:16 UTC

Status changed to closed

Great news!

Marin,

Great news, your email (gitlab at gitlab.com) has been whitelisted and your actions (view issue, view merge, view commit) are now live. 

Please feel free to get in touch if you have any questions. 

@marin Great news! Where are the docs on how to enable this for on-premises again? So we can reference it in the 8.0 blog post.

Status changed to reopened

@sytses There is some info in the docs but I should update it with more information according to this issue.

@marin thanks!

Nice! Don't forget to close http://feedback.gitlab.com/forums/176466-general/suggestions/6534681-add-view-pull-request-buttons-for-gmail

mentioned in merge request !1327

I'll close the feedback item.

Status changed to closed by commit 26544eab

mentioned in commit 26544eab

Looks to be dead again:

There was a Gmail action for me on the notification about the previous comment.

Odd. Looks like even GitHub Gmail actions aren't working for me.

And today I'm getting actions on emails that should not have actions.

That looks like a bug @bbodenmiller, can you raise a separate issue?

mentioned in issue #3014

mentioned in issue #3142

mentioned in issue #14624