Engineering: Implement trial reminder as trial gets close to expiring

Experiment summary

We believe that we can increase the trial conversion rate by reminding admins at key moments during the trial. To test this, we will remind admins of active trials once when they have 14–7 days remaining and once when they have 3 or fewer days remaining in their trial. We will remember that we’ve shown them these reminders and will not display that reminder again but each admin will receive one reminder when 14–7 days are remaining and one when 3 or fewer days are remaining in the trial.

Hypothesis

We believe we can increase the trial upgrade rate by more acutely reminding admins how many days are remaining in their trial.

Business problem

It can be difficult to know how much time remains in your trial, and even with the new trial status in the sidebar menu, the repetition can lead to UX blindness. We need a way to remind users who can take billing-related action that their trial is about to expire.

Supporting data

In our initial experiment, we saw increase trial conversion rates from users enrolled in the experiment where we displayed the days remaining in the trial in a simple sidebar widget. We saw increased trial conversions in users enrolled in the experiment and saw a 10% open to click through rate for users that viewed the amount of time remaining in their trial. By displaying the days remaining at key moments – when 14–7 days remain in the trial and when 3 or fewer days remain – we believe we can further increase the upgrade rate.

Expected outcome

We expect to see a higher trial-to-paid conversion rate in the experiment vs the control within 40 days of the start of their respective trials.

Experiment design & implementation

In the control, users have easy access to the time remaining in their trial by means of the sidebar widget. On hover of the widget, a popover is revealed which has more information and 2 CTAs; they can click to upgrade or explore plans.

In the experiment, we will forcibly “pop” the modal out for group admins when the group has between 14-7 days remaining in their trial and again a second time when they have 3–0 days left in their trial. Once we’ve shown them the modal, we will remember that action and not forcibly pop it out again until the second date range is reached.

For example, we will display a modal starting at 14 days remaining, if the user sees it on day 14 we won’t display it again during that modal time period (14–7 days). Once the admin makes it to 3 or fewer days remaining we will forcibly display the modal one more time, again remembering that we’ve done so and not displaying it again.

ICE score

Impact Confidence Ease Score
8 7 value 3 Average(1:3)

Known assumptions

We want to respect users by not inundating them with automatic modals while balancing the importance of reminding them of how much time is left in their trial. We will monitor for user feedback on how users feel about this reminder on log-in and around the cadence of receiving this modal once when 14-7 days are remaining and again when fewer than 3 days are remaining in the trial.

Results, lessons learned, next steps

TBD

Checklist

  • Fill in the experiment summary and write more about the details of the experiment in the rest of the issue description. Some of these may be filled in through time (the "Result, learnings, next steps" section for example) but at least the experiment summary should be filled in right from the start.
  • Add the label of the group:: that will work on this experiment (if known).
  • Mention the Product Manager, Engineering Manager, and at least one Product Designer from the group that owns the part of the product that the experiment will affect.
  • Fill in the values in the ICE score table ping other team members for the values you aren’t confident about (i.e. engineering should almost always fill out the ease section). Add the ~"ICE Score Needed" label to indicate that the score is incomplete.
  • Replace the ~"ICE Score Needed" with an ICE low/medium/high score label once all values in the ICE table have been added.
  • Mention the [at]gitlab-core-team team and ask for their feedback.
Edited by Dallas Reedy