Dormant user deactivation does not work
Summary
We have a strange issue where automatic dormant user deactivation does not work, it appears that all of the conditions are met in order for the user to get deactivated, however, the DeactivateDormantUsersWorker
does not pick up this user.
Installation: GitLab helm. Version: 16.5.1
Below output from the rails console meets all of the conditions for the user to be deactivated automatically:
user = User.find() # you can replace that ID with any user's ID you think should be getting deactivated
user.can_be_deactivated?
user.last_activity_on
user.last_sign_in_at
user.current_sign_in_at
Also User.dormant
returns the user in the list:
User.dormant
The application settings are set correctly as well:
Gitlab::CurrentSettings.deactivate_dormant_users
Gitlab::CurrentSettings.deactivate_dormant_users_period
Sidekiq worker Users::DeactivateDormantUsersWorker
runs without issue also tried running it manually from the console Users::DeactivateDormantUsersWorker.perform
user still does not get deactivated.
However, running the service itself DOES deactivate the user Users::DeactivateService.new(admin_bot).execute(user)
As a last resort we ran to clear the redis cache sudo gitlab-rake cache:clear
nothing changed.
Looking at the sidekiq log it appears that sidekiq does not have any args when running:
| 1697604124132 | {"log":"{\"severity\":\"INFO\",\"time\":\"2023-10-18T04:42:04.078Z\",\"retry\":0,\"queue\":\"default\",\"version\":0,\"queue_namespace\":\"cronjob\",\"args\":[],\"class\":\"Users::DeactivateDormantUsersWorker\",\"jid\":\"7e6132718f7a5563609a36e1\",\"created_at\":\"2023-10-18T04:42:03.819Z\",\"meta.caller_id\":\"Cronjob\",\"correlation_id\":\"d84bb56771cbe7c443efa01b696f208a\",\"meta.root_caller_id\":\"Cronjob\",\"meta.feature_category\":\"seat_cost_management\",\"worker_data_consistency\":\"always\",\"idempotency_key\":\"resque:gitlab:duplicate:default:4dcffa7fe1c8f101f359dcdc169989fcba5c9fb7843a52fc30d1b9caae12380b\",\"size_limiter\":\"validated\",\"enqueued_at\":\"2023-10-18T04:42:03.825Z\",\"job_size_bytes\":2,\"pid\":24,\"message\":\"Users::DeactivateDormantUsersWorker JID-7e6132718f7a5563609a36e1: done: 0.244422 sec\",\"job_status\":\"done\",\"scheduling_latency_s\":0.008352,\"redis_calls\":4,\"redis_duration_s\":0.019184,\"redis_read_bytes\":4,\"redis_write_bytes\":570,\"redis_feature_flag_calls\":2,\"redis_feature_flag_duration_s\":0.010419,\"redis_feature_flag_read_bytes\":2,\"redis_feature_flag_write_bytes\":384,\"redis_queues_calls\":2,\"redis_queues_duration_s\":0.008765,\"redis_queues_read_bytes\":2,\"redis_queues_write_bytes\":186,\"db_count\":6,\"db_write_count\":0,\"db_cached_count\":0,\"db_replica_count\":0,\"db_primary_count\":6,\"db_main_count\":6,\"db_ci_count\":0,\"db_main_replica_count\":0,\"db_ci_replica_count\":0,\"db_replica_cached_count\":0,\"db_primary_cached_count\":0,\"db_main_cached_count\":0,\"db_ci_cached_count\":0,\"db_main_replica_cached_count\":0,\"db_ci_replica_cached_count\":0,\"db_replica_wal_count\":0,\"db_primary_wal_count\":0,\"db_main_wal_count\":0,\"db_ci_wal_count\":0,\"db_main_replica_wal_count\":0,\"db_ci_replica_wal_count\":0,\"db_replica_wal_cached_count\":0,\"db_primary_wal_cached_count\":0,\"db_main_wal_cached_count\":0,\"db_ci_wal_cached_count\":0,\"db_main_replica_wal_cached_count\":0,\"db_ci_replica_wal_cached_count\":0,\"db_replica_duration_s\":0.0,\"db_primary_duration_s\":0.191,\"db_main_duration_s\":0.191,\"db_ci_duration_s\":0.0,\"db_main_replica_duration_s\":0.0,\"db_ci_replica_duration_s\":0.0,\"cpu_s\":0.015506,\"mem_objects\":7871,\"mem_bytes\":755784,\"mem_mallocs\":3635,\"mem_total_bytes\":1070624,\"worker_id\":\"sidekiq_0\",\"rate_limiting_gates\":[],\"duration_s\":0.244422,\"completed_at\":\"2023-10-18T04:42:04.078Z\",\"load_balancing_strategy\":\"primary\",\"db_duration_s\":0.1911,\"urgency\":\"low\",\"target_duration_s\":300,\"target_scheduling_latency_s\":60}\n","stream":"stdout","docker":{"container_id":""},"kubernetes":{"container_name":"sidekiq","namespace_name":"default","pod_name":"","container_image":"registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ee:v16.4.1","container_image_id":"docker-pullable://registry.gitlab.com","pod_id":"","pod_ip":"","host":"","labels":{"app":"sidekiq","chart":"sidekiq-7.4.1","heritage":"Helm","pod-template-hash":"67fdd6c9cd","queue-pod-name":"all-in-1","release":""},"master_url":"https://ip/api","namespace_id":"","namespace_labels":{"kubernetes.io/metadata.name":"default"}}} |
Steps to reproduce
- Be running at least GitLab 16.3.
- Enable Admin Mode
- Create a dormant user
- Manually execute the dormant workers cron from the Admin interface
To force a user to qualify for deactivation:
user = <id> # pick a user to make dormant and ready for deactivation
last_active = Gitlab::CurrentSettings.deactivate_dormant_users_period.days.ago.to_date - 1
user.last_activity_on = last_active
user.last_sign_in_at = last_active
user.current_sign_in_at = last_active
Possible fixes
N/A
Gitlab internal ZD