## Problem Currently, notification jobs are scattered across multiple queues with varying priorities: **Existing notification queues**: - `monthly_notification` (weight 2): Monthly seat usage digests - `mailers` (weight 2): General email delivery - Various queues contain notification jobs mixed with other operations **Issues with current setup**: 1. Time-sensitive notifications (trial expiration, payment failures) compete with scheduled digests 2. No clear distinction between urgent vs. routine notifications 3. Difficult to monitor and optimize notification delivery separately **Examples of time-sensitive notifications**: - Trial expiration warnings (should be immediate) - Payment failure alerts (revenue-impacting) - License delivery notifications (customer-blocking) - Reconciliation alerts (time-bound) ## Proposal Create a dedicated high-priority queue for urgent, time-sensitive notifications. ### New Queue Structure **`notifications_urgent` (weight 7-8)** Purpose: Time-sensitive, customer-impacting notifications that should be delivered immediately Jobs to move here: - License delivery notifications (from `gitlab` queue) - Trial expiration warnings (from `expiration` queue) - Payment failure alerts - Reconciliation deadline notifications - Any notification where delay impacts customer experience **Keep `monthly_notification` (weight 3)** Purpose: Scheduled digest emails that can tolerate some delay Jobs that stay here: - `SeatsUsageMonthlyNotificationJob` - Monthly usage reports - Periodic summary emails **Keep `mailers` (weight 3)** Purpose: General transactional emails Jobs that stay here: - Order confirmations - Account updates - General customer communications ### Benefits 1. **Better customer experience**: Urgent notifications delivered promptly 2. **Clear prioritization**: Explicit distinction between urgent and routine 3. **Improved monitoring**: Can track urgent notification delivery separately 4. **Reduced risk**: Payment and expiration alerts won't be delayed by digest emails ## Implementation Steps 1. **Identify all notification jobs**: - Audit codebase for mailer jobs - Categorize by urgency and customer impact - Document current queue assignments 2. **Create new queue**: ```ruby # app/jobs/notifications/urgent_base_job.rb module Notifications class UrgentBaseJob < ApplicationJob queue_as :notifications_urgent # Common configuration for urgent notifications end end ``` 3. **Update job classes**: ```ruby # Example: License notification class LicenseNotificationJob < Notifications::UrgentBaseJob def perform(subscription_id) # Send license email end end ``` 4. **Update `config/sidekiq.yml`**: ```yaml :queues: # ... other queues ... - [notifications_urgent, 8] # ... other queues ... - [monthly_notification, 3] - [mailers, 3] ``` 5. **Add monitoring**: - Alert on queue depth for `notifications_urgent` - Track delivery latency for urgent notifications - Monitor failure rates 6. **Document guidelines**: - When to use `notifications_urgent` vs. `mailers` - SLA expectations for each notification queue - Examples of urgent vs. routine notifications ## Examples of Queue Assignment **Use `notifications_urgent`**: - ✅ Trial expires in 24 hours - ✅ Payment method failed - ✅ License ready for download - ✅ Reconciliation due in 7 days - ✅ Subscription cancelled (immediate impact) **Use `monthly_notification`**: - ✅ Monthly seat usage digest - ✅ Quarterly usage summary - ✅ Monthly invoice available **Use `mailers`**: - ✅ Order confirmation - ✅ Account settings changed - ✅ Password reset - ✅ General announcements ## Success Criteria - All time-sensitive notifications identified and moved to new queue - Urgent notifications delivered within 1 minute of enqueue (P95) - No customer complaints about delayed critical notifications - Clear documentation for future notification jobs ## Related - Parent epic: gitlab-org&19587 - Related: #14270 (user-facing operations)