Users are not receiving upcoming token expiration email notifications
Summary
As per the documentation and this MR, users should be sent emails advising of upcoming group/project token expiration. Users are reporting that they are not receiving emails when expected.
What is the current bug behavior?
Users are not receiving emails for upcoming expiration of project and group tokens on gitlab.com
What is the expected correct behavior?
Users should receive email notification for upcoming expiring tokens as per the documentation.
All project maintainers receive an email when project access tokens are 7 days or less from expiration.
All group owners receive an email when group access tokens are 7 days or less from expiration.
Relevant logs and/or screenshots
The following database timeout can be seen in Kibana for PersonalAccessTokens::ExpiringWorker
, every day at 01:00
(except Nov 20) - which may or may not be relevant?
- Kibana link - internal
- Sentry link - internal
PG::QueryCanceled: ERROR: canceling statement due to statement timeout
[lib/gitlab/database/load_balancing/connection_proxy.rb:107:in `public_send', lib/gitlab/database/load_balancing/connection_proxy.rb:107:in `block in read_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write', lib/gitlab/database/load_balancing/load_balancer.rb:235:in `retry_with_backoff', lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write', lib/gitlab/database/load_balancing/connection_proxy.rb:106:in `read_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:48:in `select_all', lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send', lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write', lib/gitlab/database/load_balancing/load_balancer.rb:235:in `retry_with_backoff', lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write', lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing', lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send', lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer', lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write', lib/gitlab/database/load_balancing/load_balancer.rb:235:in `retry_with_backoff', lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write', lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer', lib/gitlab/database/load_balancing/connection_proxy.rb:96:in `method_missing', app/workers/personal_access_tokens/expiring_worker.rb:19:in `perform', lib/gitlab/sidekiq_middleware/skip_jobs.rb:49:in `call', lib/gitlab/database/load_balancing/sidekiq_server_middleware.rb:29:in `call', lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executing.rb:16:in `perform', lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:45:in `perform', lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call', lib/gitlab/sidekiq_middleware/pause_control/strategies/base.rb:31:in `perform', lib/gitlab/sidekiq_middleware/pause_control/strategy_handler.rb:22:in `perform', lib/gitlab/sidekiq_middleware/pause_control/server.rb:8:in `call', lib/gitlab/application_context.rb:130:in `block in use', lib/gitlab/application_context.rb:130:in `use', lib/gitlab/sidekiq_middleware/worker_context.rb:11:in `wrap_in_optional_context', lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call', lib/gitlab/application_context.rb:130:in `block in use', lib/gitlab/application_context.rb:130:in `use', lib/gitlab/application_context.rb:64:in `with_context', lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call', lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call', lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call', lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call', lib/gitlab/database/query_analyzer.rb:37:in `within', lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call', lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call', lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call', lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call', lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call', lib/gitlab/sidekiq_middleware/request_store_middleware.rb:8:in `block in call', lib/gitlab/sidekiq_middleware/request_store_middleware.rb:7:in `call', lib/gitlab/sidekiq_middleware/server_metrics.rb:105:in `block in call', lib/gitlab/sidekiq_middleware/server_metrics.rb:133:in `block in instrument', lib/gitlab/metrics/background_transaction.rb:33:in `run', lib/gitlab/sidekiq_middleware/server_metrics.rb:133:in `instrument', lib/gitlab/sidekiq_middleware/server_metrics.rb:104:in `call', lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call', lib/gitlab/sidekiq_daemon/monitor.rb:46:in `within_job', lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call', lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call', lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call']
Edited by Chris Stone