[Improved] Periodically save last activity date data from Redis to the database
What does this MR do?
- It changes the way we store users last activity in Redis, from a Sorted Set to a Hash
- It adds a new
users.last_activity_oncolumn- Use it in the
/user/activitiesAPI instead of getting the data from Redis
- Use it in the
- It periodically (each day) schedules
UpdateUserActivityWorkerwhich takes a batch of user activities from the Redis Hash (i.e.{ "USER_ID" => "TIMESTAMP" }, usingScheduleUpdateUserActivityWorker - It updates users'
last_activity_onusingUpdateUserActivityWorkerwith a single SQL query and removes the keys from the Redis Hash - It adds a post-deploy migration to migrate the data from the Sorted Set to the new
users.last_activity_oncolumn-
Test the migration with 400K records in the legacy
user/activitiesSorted set
-
[1] pry(main)> Gitlab::Redis.with do |redis|
[1] pry(main)* redis.zcount('user/activities', 6.months.ago.to_i, Time.now.to_i)
[1] pry(main)* end
=> 455000
[2] pry(main)> User.where.not(last_activity_on: nil).count
(73.3ms) SELECT COUNT() FROM "users" WHERE ("users"."last_activity_on" IS NOT NULL)
=> 0
[3] pry(main)> User.where(last_activity_on: nil).count
(60.1ms) SELECT COUNT() FROM "users" WHERE "users"."last_activity_on" IS NULL
=> 457500
```
```bin/rake db:migrate == 20170324160416 MigrateUserActivitiesToUsersLastActivityOn: migrating ======= == 20170324160416 MigrateUserActivitiesToUsersLastActivityOn: migrated (57.9876s) ```
- Test the migration on MySQL
- It removes the
lib/gitlab/user_activities/code
Are there points in the code the reviewer needs to double check?
Does this MR meet the acceptance criteria?
- Changelog entry added, if necessary
- Documentation created/updated
- API support added
- Tests
- Added for this feature/bug
- All builds are passing
- Conform by the merge request performance guides
- Conform by the style guides
- Branch has no merge conflicts with
master(if it does - rebase it please) - Squashed related commits together
What are the relevant issue numbers?
Closes #27790 (closed)