WIP: Use a new table for user contribution stats
What does this MR do?
It defers the user contribution stats to a new table, instead of the events table.
Are there points in the code the reviewer needs to double check?
Why was this MR needed?
This will allow down sizing the events table.
Screenshots (if relevant)
Does this MR meet the acceptance criteria?
-
Changelog entry added -
Documentation created/updated -
API support added - Tests
-
Added for this feature/bug -
All builds are passing
-
-
Conform by the merge request performance guides -
Conform by the style guides -
Branch has no merge conflicts with master
(if it does - rebase it please) -
Squashed related commits together
What are the relevant issue numbers?
Merge request reports
Activity
@yorickpeterse Hi, what do you think of the migration?
- Resolved by ido leibovich
- Resolved by ido leibovich
- Resolved by Yorick Peterse
@leibo I posted some comments on the migration above.
@yorickpeterse Hi, I fixed according to your comments, WDYT?
- Resolved by ido leibovich
- Resolved by ido leibovich
Reassigned to @leibo
@yorickpeterse fixed, moving on I think
@leibo Looks good so far
Mentioned in issue #24344 (closed)
@yorickpeterse Added the model and Sidekiq worker, take a look.
- Resolved by ido leibovich
- Resolved by ido leibovich
Added 661 commits:
-
31c2b3b1...75c8faf7 - 660 commits from branch
gitlab-org:master
- ecb46de4 - Use a new table for user conribution stats
-
31c2b3b1...75c8faf7 - 660 commits from branch
@yorickpeterse Ok, comments fixed, how do we move on? What else is needed?
@leibo Two things remain:
- The contribution calendar should use this new table
- Tests
@yorickpeterse I'm wondering about two more things:
- How will the user_contribution table be updated with all the data from the past?
- How do we tell the worker to run every day?
How will the user_contribution table be updated with all the data from the past?
We can run a migration that populates the table with data from the past year. This migration may take quite a while to run so perhaps we need to split it into chunks and run those in parallel. Let's start with a simple migration and we can measure how long this takes on GitLab's staging environment.
How do we tell the worker to run every day?
In
config/initializers/1_settings.rb
we define various cronjobs using this pattern:Settings.cron_jobs['trending_projects_worker'] ||= Settingslogic.new({}) Settings.cron_jobs['trending_projects_worker']['cron'] = '0 1 * * *' Settings.cron_jobs['trending_projects_worker']['job_class'] = 'TrendingProjectsWorker'
We can probably run the worker around 01:00.
@yorickpeterse Does gitlab have a notion of 'data migrations', or should I just use a normal migration?
@leibo We use regular migrations for it.
Added 617 commits:
-
defba3a6...448c19aa - 616 commits from branch
gitlab-org:master
- e5786eef - Use a new table for user conribution stats
-
defba3a6...448c19aa - 616 commits from branch
@yorickpeterse I've added the new migration and the schedule. Once you'll go over them I'll continue to the tests and contribution calendar.
- Resolved by ido leibovich
- Resolved by ido leibovich
@yorickpeterse Hi, just an update: I've made some of the changes, but experiencing trouble in my environment: for some reason when I try to push to my code to the remote branch, I get rejected. I've posted questions on the gitlab forum and IRC channel, hope I get can resolve this soon.
@yorickpeterse Hi again, I can't get this to work at the moment - very weird. Can you help in any way or point to someone who can help?
@leibo What's the link of the support forum question?
@leibo You could try re-creating the project, perhaps that solves the problem. Alternatively, make sure that your SSH keys and all that are still set up properly.
@yorickpeterse Ok, I finally got some help in the IRC channel, take a look at the new migration :)
- Resolved by ido leibovich
@yorickpeterse Ok, fixed the migration as discussed. WDYT? I'm not sure how to test it - do sidekiq workers go up by default when running gdk? How can I see when if the job was scheduled? Or if it failed/succeeded?
@leibo All Sidekiq workers are executed by default, given the queues are in
config/sidekiq_queues.yml
. Testing is best done in two steps:- Writing an RSpec test
- Scheduling a job from a Rails console (mostly to see if you might have missed anything)
- Resolved by ido leibovich
- Resolved by ido leibovich
- Resolved by ido leibovich
changed milestone to %8.16
mentioned in issue #22623 (moved)
@yorickpeterse @smcgivern Any conclusions? how should we progress with this?