Experiment Tracking: MVC Add contact sales option in app

This is the experiment tracking issue for the following issue.

Experiment key: :contact_sales_btn_in_app

Feature flag name: :contact_sales_btn_in_app_experiment_percentage

Tracking the experiment through to cleanup and removal of the feature flag (Cleanup issue template: https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Experiment%20Successful%20Cleanup)

Experiment tracking log

  • 2020-08-18 22:42 UTC – Experiment went live to 50% of users
  • 2020-10-02 13:37:03 UTC – Experiment went live to 100% of users
  • 2020-12-02 informed this was not live in production
  • 2020-12-02 - re-enabled at 100%

See the due date for next check-in.

Tracking information

  • CTAs will be located on the /billings page
  • CTA links will be as follows
    • Bronze: https://about.gitlab.com/sales/index.html?test=inappcontactsalesbronze
    • Silver: https://about.gitlab.com/sales/index.html?test=inappcontactsalessilver
    • Gold: https://about.gitlab.com/sales/index.html?test=inappcontactsalesgold

Tracking Dashboard

Tracked Events

When Sent from Action Category Label Property Value
User lands on Profile billing page backend 'page_view:billing_plans:profile' <CATEGORY> <SUBJ_ID> <SEGMENT> N/A
User lands on Group billing page backend 'page_view:billing_plans:group' <CATEGORY> <SUBJ_ID> <SEGMENT> N/A
User clicks “Contact Sales” button frontend 'click_button' 'profiles:billings:index' or 'groups:billings:index' 'contact_sales' <CATEGORY>:<SEGMENT> N/A
User clicks “Upgrade” button frontend 'click_button' 'profiles:billings:index' or 'groups:billings:index' 'upgrade' <CATEGORY>:<SEGMENT> N/A

Where:

  • <CATEGORY> is 'Growth::Conversion::Experiment::ContactSalesInApp'
  • <SUBJ_ID> is like 349e4f12-1cf8-4429-9406-64aeb1f63cb3
  • <SEGMENT> is either 'experimental_group' or 'control_group'

A Caveat

I (@dreedy) realized far too late that I had actually made a bit of a mistake in how I set up the Ruby code for creating the data- attributes for tracking the frontend experiment events. So, they’re actually not getting sent at all, I don’t think.

Instead of creating data-track-event, data-track-label, & data-track-property attributes, the code I wrote actually creates a single data-track attribute with all of the other stuff attached to it as a JSON object, essentially:

<a data-track='{ "event": "click_button", "label": "contact_sales", "property": "Growth::Conversion::Experiment::ContactSalesInApp:experimental_group" }'>Contact sales</a>

Luckily, we didn’t seem to need those events in order to determine the success of this experiment, and it was considered a success as of 2020-09-17.

Edited by Phil Calder