Cohort data is inconsistent depending on date a user views it

Summary

Based on GitLab.com, monthly returners by cohort for past months seems to increase over time.

  • On 2018-03-13, I pulled cohort retention from admin/cohorts from the admin area for GitLab.com. Month 1 retention for January's cohort was found to be X.
  • On 2018-04-16, I pulled the same cohort report from admin/cohorts. Looking back at January Month 1 retention, this number is now X + 4%.

Considering January (and February, the month 1 retention month) were both completely in the past for both pulls, I'd expect this number to be unchanged.

Please see the "Cohort Data" tab of https://docs.google.com/spreadsheets/d/1l5F9SNfZ8ztQwTnpZ6dXmYk_kZ1AaEl1rwL0hj2jQmE/edit#gid=170500415 for specific values.

What is the current bug behavior?

Retention numbers seem to increase over time.

What is the expected correct behavior?

Cohort retention for completed months should not change.

Output of checks

This bug happens on GitLab.com.

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)

Edited Apr 16, 2018 by Jeremy Watson (ex-GitLab)
Assignee Loading
Time tracking Loading