Frontend: Experiment Implementation: "Performance Insights" Button

Experiment Summary

This tests if adding a button which opens a (modal/sidebar) with a few stats on it about the pipeline timing results in users looking at the data.

From this idea in the team test discussion issue.

Experiment design/measures of success

• This will run for 2 months/milestones for all users
• We expect at least 5% of active monthly users who visit the page to click the button
• We expect 25% of users who click the button for details to click further into a job
• We expect to collect 10 or more comments from users the feedback issue over the 2 months.
  • We will utilize social media and the Field Team to make users aware who have previously requested pipeline timing data.
  • Feedback will be collected in a feedback issue

Design

1. Add a button with title "Performance Statistics" to the pipeline detail page that opens a (modal/sidebar). (see below)
2. Details should show some/all/what we can get added within the implementation time of the following ranked in order of preference.
   1. Top 5 slowest jobs in the pipeline (using available job duration) -
   2. Longest wait for the job - shows how long last job took before it was executed (e.g. waiting for jobs in previous stages to finish) - started_at - created_at
   3. A tooltip/explanation of how longest wait and longest queueing time is calculated
   4. A link to feedback issue - TBD
   5. Slowest job per stage (using job duration)
   6. Longest queueing time - job that spent the longest in pending state waiting to be picked by a Runner (using available job queued_duration)

Rollout strategy

This is not an A/B test and can be rolled out to all users using our normal Feature Flag rolloutstrategy.

Tracking Details

• Track how often the modal is opened.
• Track how often a detail is clicked

Open Questions

• Which is faster implementation modal or sidebar for the data?
  • Design is modal which Dev agreed was doable.
• Can we link to job details?
  • Yes the modal links to job details for listed jobs
• How are we tracking this / how will we report it?
  • Snowplow tracking (click_insights_button action, performance_insights label)
• What is the criteria for success/we should prioritize making this an MVC and more performant?
  • Noted in the experiment design.