Skip to content
Snippets Groups Projects

Get rid of user activites table and replace it with redis

Merged James Lopez requested to merge fix/user-activities-performance into master
  • Gets rid of the user activities table
  • Switch from postgreSQL to Redis

Fixes https://gitlab.com/gitlab-org/gitlab-ee/issues/1158

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Reassigned to @jameslopez

  • James Lopez Marked this merge request as a Work In Progress

    Marked this merge request as a Work In Progress

  • Mentioned in issue #1288 (closed)

  • James Lopez Added 1 commit:

    Added 1 commit:

    • 3da15069 - Fixed spec, removed duplicate push count, also updated code based on feedback

    Compare with previous version

  • James Lopez Added 1 commit:

    Added 1 commit:

    • 3caba170 - use post deployment migration

    Compare with previous version

  • James Lopez Unmarked this merge request as a Work In Progress

    Unmarked this merge request as a Work In Progress

  • James Lopez Added 1 commit:

    Added 1 commit:

    • a6226b6a - add extra spec for testing the actual time change in an update

    Compare with previous version

  • Author Maintainer

    @yorickpeterse updated the code. Can you review again? Also added a spec to double-check we are updating users and not creating new entries.

    Edited by James Lopez
  • James Lopez Added 1 commit:

    Added 1 commit:

    Compare with previous version

  • Author Maintainer

    @yorickpeterse quick question. Do you know when does the Gitlab::Redis cache expire (if ever?) and is there any task or background job that runs that could clean/remove the user/activities set ?

  • Yorick Peterse
  • Yorick Peterse
  • Reassigned to @jameslopez

  • @jameslopez I believe it uses the same expiration time as the other data, so 2 weeks. Not entirely sure though, perhaps the automatic TTL is a Rails.cache only thing.

    Edited by Yorick Peterse
  • Author Maintainer

    @yorickpeterse thanks again! left a question https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/915#note_19149221 and a couple of responses to your comments :)

  • Author Maintainer

    @yorickpeterse ahhh.. We don't want that. That's probably not very useful for the customer as we want to know the users with less activity over time... Is there anything I can do to change that? A new namespace ?

  • @jameslopez It seems Gitlab::Redis does not automatically set a TTL:

    2.3.2p217 > Gitlab::Redis.with { |r| r.set('foobar', 10) }
    => "OK"
    2.3.2p217 > Gitlab::Redis.with { |r| r.ttl('foobar') }
    => -1
  • Author Maintainer

    @yorickpeterse I see, so what will make an entry to expire? I checked the omnibus redis conf but couldn't find anything...

  • @jameslopez When using Gitlab::Redis the entry should not be expired unless one runs FLUSHALL or Redis somehow loses data.

  • James Lopez Resolved all discussions

    Resolved all discussions

  • Author Maintainer

    @yorickpeterse perfect! Thanks for your help :) I'll now assign this to an endboss unless you have further questions :thumbsup:

  • @jameslopez Perfect, go ahead. :smile:

  • Reassigned to @rspeicher

  • Author Maintainer

    @rspeicher can you review this one? Thanks!

  • Valery Sizov
  • Love this MR :)

  • Robert Speicher Milestone changed to %8.15

    Milestone changed to %8.15

  • Author Maintainer

    @rspeicher any reason to move it to 8.15? As this was affecting performance I thought it would be more useful to have it in a patch release...

  • Author Maintainer

    Looking into why the build is failing (migrations were fine locally) Mysql2::Error: Unknown table 'gitlabhq_test.user_activities': DROP TABLE user_activities

  • @jameslopez The attached issue didn't have a milestone and this wasn't ~"Pick into Stable" so I figured it was a mistake.

  • Author Maintainer

    ohh right! I'll update... thanks @rspeicher !

  • Reassigned to @jameslopez

  • James Lopez Added 1 commit:

    Added 1 commit:

    • 54e147b2 - fix schema and updated spec

    Compare with previous version

  • James Lopez Added ~149424 label

    Added ~149424 label

  • James Lopez Milestone changed to %8.14

    Milestone changed to %8.14

  • James Lopez Resolved all discussions

    Resolved all discussions

  • James Lopez Added 1 commit:

    Added 1 commit:

    Compare with previous version

  • James Lopez Added 1 commit:

    Added 1 commit:

    • 423cb005 - fix other spec failures and refactor common stuff into module

    Compare with previous version

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading