Cohort data is inconsistent depending on date a user views it
Summary
Based on GitLab.com, monthly returners by cohort for past months seems to increase over time.
- On 2018-03-13, I pulled cohort retention from
admin/cohortsfrom the admin area for GitLab.com. Month 1 retention for January's cohort was found to be X. - On 2018-04-16, I pulled the same cohort report from
admin/cohorts. Looking back at January Month 1 retention, this number is now X + 4%.
Considering January (and February, the month 1 retention month) were both completely in the past for both pulls, I'd expect this number to be unchanged.
Please see the "Cohort Data" tab of https://docs.google.com/spreadsheets/d/1l5F9SNfZ8ztQwTnpZ6dXmYk_kZ1AaEl1rwL0hj2jQmE/edit#gid=170500415 for specific values.
What is the current bug behavior?
Retention numbers seem to increase over time.
What is the expected correct behavior?
Cohort retention for completed months should not change.
Output of checks
This bug happens on GitLab.com.
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)
Edited by Jeremy Watson (ex-GitLab)