[Improved] Periodically save last activity date data from Redis to the database

What does this MR do?

  • It changes the way we store users last activity in Redis, from a Sorted Set to a Hash
  • It adds a new users.last_activity_on column
    • Use it in the /user/activities API instead of getting the data from Redis
  • It periodically (each day) schedules UpdateUserActivityWorker which takes a batch of user activities from the Redis Hash (i.e. { "USER_ID" => "TIMESTAMP" }, using ScheduleUpdateUserActivityWorker
  • It updates users' last_activity_on using UpdateUserActivityWorker with a single SQL query and removes the keys from the Redis Hash
  • It adds a post-deploy migration to migrate the data from the Sorted Set to the new users.last_activity_on column
    • Test the migration with 400K records in the legacy user/activities Sorted set

[1] pry(main)> Gitlab::Redis.with do |redis| [1] pry(main)* redis.zcount('user/activities', 6.months.ago.to_i, Time.now.to_i)
[1] pry(main)* end
=> 455000 [2] pry(main)> User.where.not(last_activity_on: nil).count (73.3ms) SELECT COUNT() FROM "users" WHERE ("users"."last_activity_on" IS NOT NULL) => 0 [3] pry(main)> User.where(last_activity_on: nil).count (60.1ms) SELECT COUNT() FROM "users" WHERE "users"."last_activity_on" IS NULL => 457500 ```


bin/rake db:migrate == 20170324160416 MigrateUserActivitiesToUsersLastActivityOn: migrating ======= == 20170324160416 MigrateUserActivitiesToUsersLastActivityOn: migrated (57.9876s) ```

  • Test the migration on MySQL
  • It removes the lib/gitlab/user_activities/ code

Are there points in the code the reviewer needs to double check?

Does this MR meet the acceptance criteria?

What are the relevant issue numbers?

Closes #27790