Skip to content
Snippets Groups Projects

WIP: Use a new table for user contribution stats

Closed ido leibovich requested to merge leibo/gitlab-ce:user-contribution-table into master

What does this MR do?

It defers the user contribution stats to a new table, instead of the events table.

Are there points in the code the reviewer needs to double check?

Why was this MR needed?

This will allow down sizing the events table.

Screenshots (if relevant)

Does this MR meet the acceptance criteria?

What are the relevant issue numbers?

#22623 (moved)

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • @leibo I posted some comments on the migration above.

  • ido leibovich Added 1 commit:

    Added 1 commit:

    • 0cd5c1c1 - Use a new table for user conribution stats

    Compare with previous version

  • Author Contributor

    @yorickpeterse Hi, I fixed according to your comments, WDYT?

  • Reassigned to @leibo

  • ido leibovich Added 1 commit:

    Added 1 commit:

    • 76160d06 - Use a new table for user conribution stats

    Compare with previous version

  • ido leibovich Resolved all discussions

    Resolved all discussions

  • Author Contributor

    @yorickpeterse fixed, moving on I think

  • @leibo Looks good so far :thumbsup:

  • Mentioned in issue #24344 (closed)

  • Clement Ho Added ~164274 label

    Added ~164274 label

  • ido leibovich Added 1 commit:

    Added 1 commit:

    • 31c2b3b1 - Use a new table for user conribution stats

    Compare with previous version

  • Author Contributor

    @yorickpeterse Added the model and Sidekiq worker, take a look.

  • ido leibovich Resolved all discussions

    Resolved all discussions

  • ido leibovich Added 661 commits:

    Added 661 commits:

    Compare with previous version

  • Author Contributor

    @yorickpeterse Ok, comments fixed, how do we move on? What else is needed?

  • ido leibovich Added 1 commit:

    Added 1 commit:

    • defba3a6 - Use a new table for user conribution stats

    Compare with previous version

  • @leibo Two things remain:

    1. The contribution calendar should use this new table
    2. Tests
  • Author Contributor

    @yorickpeterse I'm wondering about two more things:

    1. How will the user_contribution table be updated with all the data from the past?
    2. How do we tell the worker to run every day?
  • @leibo

    How will the user_contribution table be updated with all the data from the past?

    We can run a migration that populates the table with data from the past year. This migration may take quite a while to run so perhaps we need to split it into chunks and run those in parallel. Let's start with a simple migration and we can measure how long this takes on GitLab's staging environment.

    How do we tell the worker to run every day?

    In config/initializers/1_settings.rb we define various cronjobs using this pattern:

    Settings.cron_jobs['trending_projects_worker'] ||= Settingslogic.new({})
    Settings.cron_jobs['trending_projects_worker']['cron'] = '0 1 * * *'
    Settings.cron_jobs['trending_projects_worker']['job_class'] = 'TrendingProjectsWorker'

    We can probably run the worker around 01:00.

  • Author Contributor

    @yorickpeterse Does gitlab have a notion of 'data migrations', or should I just use a normal migration?

  • @leibo We use regular migrations for it.

  • ido leibovich Added 617 commits:

    Added 617 commits:

    Compare with previous version

  • Author Contributor

    @yorickpeterse I've added the new migration and the schedule. Once you'll go over them I'll continue to the tests and contribution calendar.

  • Author Contributor

    @yorickpeterse Hi, just an update: I've made some of the changes, but experiencing trouble in my environment: for some reason when I try to push to my code to the remote branch, I get rejected. I've posted questions on the gitlab forum and IRC channel, hope I get can resolve this soon.

  • Author Contributor

    @yorickpeterse Hi again, I can't get this to work at the moment - very weird. Can you help in any way or point to someone who can help?

  • @leibo What's the link of the support forum question?

  • @leibo You could try re-creating the project, perhaps that solves the problem. Alternatively, make sure that your SSH keys and all that are still set up properly.

  • ido leibovich added 1 commit

    added 1 commit

    • 3973daa0 - Use a new table for user conribution stats

    Compare with previous version

  • Author Contributor

    @yorickpeterse Ok, I finally got some help in the IRC channel, take a look at the new migration :)

  • ido leibovich added 1 commit

    added 1 commit

    • e0cf9afb - Use a new table for user conribution stats

    Compare with previous version

  • Author Contributor

    @yorickpeterse Ok, fixed the migration as discussed. WDYT? I'm not sure how to test it - do sidekiq workers go up by default when running gdk? How can I see when if the job was scheduled? Or if it failed/succeeded?

  • @leibo All Sidekiq workers are executed by default, given the queues are in config/sidekiq_queues.yml. Testing is best done in two steps:

    1. Writing an RSpec test
    2. Scheduling a job from a Rails console (mostly to see if you might have missed anything)
  • Yorick Peterse changed milestone to %8.16

    changed milestone to %8.16

  • mentioned in issue #22623 (moved)

  • Author Contributor

    @yorickpeterse @smcgivern Any conclusions? how should we progress with this?

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading