Create automated email for SaaS trial survey for trials that do not upgrade

Problem

We previously ran a survey where we collected feedback from users on why they did not upgrade after their SaaS trials ended, we previously manually ran this survey so we're not consistently getting this feedback from users.

Solution

We should create a product email sent via MailGun that asks users for their feedback on their trial experience. This email should send 40 days after their trial start date if the namespace that started the trial is on the Free plan (plan_id = 34) and the email should only be sent to the user that started the trial.

Technical Details

Run a cron job every day which checks every namespace which started a trial at least 40 days ago, which is currently on the Free plan, and which has not yet received this specific email (query in batches of 1,000; we might need to create a new index for this query). From there, we’ll need to get the user who started the trial (I don’t think we have this information) and queue up this survey email for them (in a background job with 3 retries). Once the email has been successfully delivered, mark that we’ve sent the survey email for this namespace in the database.

Breakdown

To break it down, here’s how we’re currently thinking about the implementation:

  1. We create a script which we will run daily, at the same time each day
    • This is scheduled on a system level and can be scheduled daily, hourly, minutely, weekly, monthly, etc.; for this experiment we’ll stick with daily.
  2. That script will get all of the namespaces from the onboarding_progresses table in the database which match the following criteria:
    • They have a trial_started_at of exactly 40 days ago
    • They do not have a subscription_created_at value (meaning they have not yet upgraded to a paid account)
    • Example query: 
      SELECT namespace_id FROM onboarding_progresses WHERE date_trunc('day', trial_started_at) = CURRENT_DATE - 40 AND subscription_created_at IS NULL;
  3. It will then use those namespaces to get a list of current owner email addresses (@namespace.owners.pluck(:email))
  4. It will then queue up an email per owner to be delivered to them using our internal job queue
    • That internal job queue will try to send the email to our mail delivery system (MailGun), retrying up to 3 times if needed (if the MailGun service is unresponsive, for example)
    • If it fails, then the namespaces for that day will not receive the email
  5. MailGun will then do its thing to deliver the email to each user
    • The MailGun service has its own retry mechanism for temporary bounces, and after a period of time or number of retries it will consider the email address to be a hard bounce and won’t try again
    • If MailGun classifies it as a hard bounce and fails to deliver the email, we won’t have any record of that in our database

The Email

Email subject

We'd love your feedback on your GitLab trial

Email body

How can we make GitLab better?

We'd love your feedback - it only takes one minute.

You recently started a trial of GitLab, but didn't upgrade your account. We'd love to know how we can make GitLab better for you. This anonymous three-question survey should take less than a minute of your time.

CTA

Answer 3 questions

Survey Links

Email HTML and design

Internal reference points

Engineering Research

Open Questions & Assumptions

  • Question: I’m not sure we can easily identify the person who started the trial. I think we’d have to make an API call to CustomersDot to do so, and we’d have to make that API call per trialed namespace each time we run the cron job. Is there an alternative Member record we can use instead (e.g. the admin/owner member with the earliest created_at timestamp)?
    • Possible Answer: Just use the list of current owners: @namespace.owners
    • Ongoing discussions: Thread 1, Thread 2
  • Question: What time should we schedule the cron job to run?
    • Potential Answer: The Activation team recently updated the UTC time that their emails are sent to increase their open/click rates we should use the same time that they're using for their sequence
      • Investigate this work and see if it will work for our needs here
  • Assumption: We don’t care what day or time the user receives the email.
    • Somewhat correct, see previous point.
    • Possible Follow-up Issue: We could probably create a follow-up issue for scheduling the delivery according to the user’s timezone so that it arrives at an optimal time of day (we may also wish to control which day it is delivered on by adjusting the cron job schedule).
  • Question: Should we notify ourselves in some way if the background job fails?
  • Question: Does the background job queue system log job failures?
    • Look into if current job failures get logged somewhere

Resolved Questions & Assumptions

  • Question: Should the email be sent to the same user for different namespaces/trials?
    • Answer: Yes, if it’s easy enough to do. If we can only keep track at the user level, once per user is acceptable.
  • Question: Can we reuse the InProductMarketingEmail stuff that @nicolasdular created?
    • Answer: It doesn’t make sense to do so.
  • Question: Where should this email live (Mailer class & template directory)?
    • Answer: It should be easy enough to follow existing examples.
  • Assumption: We will use a daily cron job to check for eligible namespaces
    • Correct.
  • Question: What should we do if the background job fails even after multiple retries?
    • Q: Should we just wait a day and try again?
      • A: No, that won’t be possible with the currently spec’d MVC.
  • Spec: We will not need to record which namespaces have received the email as we’re looking back at trial_started_at dates which are exactly 40 days ago each day
    • So we need to be careful to only run the script once a day!
  • Spec: We do not need access to any data (user, group, gitlab_subscription, etc.) at the time the email content is generated.
  • Spec: We do need to access data when the script runs, to determine who should receive the email.
  • Spec: This will not be run as an experiment, it will be a baseline improvement for everybody.

To-do:

  • Chat with @nicolasdular to better understand how reusable the InProductMarketingEmails stuff might be for this.
    • See result of this conversation here.
Edited by 🤖 GitLab Bot 🤖