Handle notifications for background processing

What does this MR do and why?

Adds background operation tracking and email notifications for bulk attribute updates in Security Inventory. This is controlled by the security_bulk_operations_notifications feature flag.

When enabled, bulk operations track progress in Redis and send email notifications when failures occur.

Changelog: added
EE: true

References

• Backend: Handle errors and notifications for ba... (#577580) • Nicolae Rotaru • 18.9

Screenshots

Architecture

Flow

BulkUpdateService → BulkUpdateSchedulerWorker → BackgroundOperationBulkUpdateWorker → Email (if failures)
                            ↓                              ↓
                    Create Redis Operation          Update Counters

1. BulkUpdateSchedulerWorker - Collects project IDs, creates Redis operation state, schedules batch workers (50 projects/batch)
2. BackgroundOperationBulkUpdateWorker - Processes projects, tracks success/failure in Redis
3. Finalization - Last batch sends failure email if needed, cleans up Redis

Feature Flag

The security_bulk_operations_notifications feature flag controls which worker is used:

• Disabled: Uses existing BulkUpdateWorker (no tracking/notifications)
• Enabled: Uses new BackgroundOperationBulkUpdateWorker (with tracking/notifications)

Components

File	Purpose
ee/app/workers/security/attributes/bulk_update_scheduler_worker.rb	Orchestrates batch scheduling, creates Redis operation
ee/app/workers/security/attributes/background_operation_bulk_update_worker.rb	Processes projects with tracking
ee/lib/gitlab/background_operations/redis_store.rb	Redis state management
ee/app/mailers/security/background_operation_mailer.rb	Failure notification emails

Redis Keys

• background_operation:{id} - Operation state (hash)
• background_operation:{id}:failed_items - Failed project details (list)
• TTL: 24 hours

How to set up and validate locally

Setup

# In Rails console (bin/rails console)

# Get user and group with projects
user = User.first
group = Group.first
project = group.projects.first

# Ensure user has proper access
group.add_owner(user) unless group.member?(user)
project.add_maintainer(user) unless project.member?(user)

# Enable feature flags
Feature.enable(:security_categories_and_attributes)
Feature.enable(:security_bulk_operations_notifications)

Test 1: Verify operation tracking with successful completion

# Run the scheduler worker manually
Security::Attributes::BulkUpdateSchedulerWorker.new.perform(
  [],              # group_ids
  [project.id],    # project_ids
  [],              # attribute_ids
  'ADD',           # mode
  user.id          # user_id
)

# Note: In development, jobs run inline, so the operation will be processed
# and cleaned up immediately. To see the Redis state, you can check logs
# or add breakpoints in the worker.

Test 2: Operation with failures (email notification)

# Create a mock scenario where service fails
# Option 1: Use invalid attribute IDs
Security::Attributes::BulkUpdateSchedulerWorker.new.perform(
  [],
  [project.id],
  [-1, -2],  # Invalid attribute IDs will cause service errors
  'ADD',
  user.id
)

# Check email was sent (in development, check letter_opener at http://localhost:3000/rails/letter_opener)
# The email subject should be: "Bulk operation failed: attribute_update"

Test 3: Manual step-by-step testing

# Step 1: Create operation manually
operation_id = Gitlab::BackgroundOperations::RedisStore.create_operation(
  operation_type: 'attribute_update',
  user_id: user.id,
  total_items: 1,
  parameters: { attribute_ids: [], mode: 'ADD' }
)
puts "Operation ID: #{operation_id}"

# Step 2: Check operation exists
operation = Gitlab::BackgroundOperations::RedisStore.get_operation(operation_id)
puts "Total: #{operation.total_items}, Successful: #{operation.successful_items}, Failed: #{operation.failed_items}"

# Step 3: Run batch worker
Security::Attributes::BackgroundOperationBulkUpdateWorker.new.perform(
  [project.id],
  [],
  'ADD',
  user.id,
  operation_id
)

# Step 4: Operation should be cleaned up after completion
Gitlab::BackgroundOperations::RedisStore.get_operation(operation_id)
# => nil (cleaned up)

Test 4: Verify feature flag behavior

# With flag disabled - uses legacy worker (no tracking)
Feature.disable(:security_bulk_operations_notifications)

Security::Attributes::BulkUpdateSchedulerWorker.new.perform(
  [], [project.id], [], 'ADD', user.id
)
# Uses BulkUpdateWorker, no Redis operation created

# With flag enabled - uses new worker (with tracking)
Feature.enable(:security_bulk_operations_notifications)

Security::Attributes::BulkUpdateSchedulerWorker.new.perform(
  [], [project.id], [], 'ADD', user.id
)
# Uses BackgroundOperationBulkUpdateWorker, creates Redis operation

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.