Add partner token verification worker for Secret Detection

What does this MR do and why?

This MR adds the background worker component for partner token verification, building on the service layer introduced in !197819 (merged). The worker enables asynchronous processing of token verification requests with intelligent retry logic.

Parent MR: !197819 (merged)

Key features:

1. Asynchronous Processing

   • Moves token verification out of the request cycle
   • Prevents UI timeouts for slow partner API responses
   • Improves overall system responsiveness

2. Smart Retry Logic

   • Exponential backoff for network errors (4, 16, 64 seconds)
   • Immediate failure for permanent errors (auth, config issues)
   • Prevents retry storms and unnecessary API calls

3. Comprehensive Error Handling

   • Classifies errors as retryable vs non-retryable
   • Network/timeout errors get retried automatically
   • Configuration/business errors fail fast

4. Monitoring & Observability

   • Detailed job metadata for tracking
   • Request ID correlation for debugging
   • Clear logging of retry decisions

Architecture Flow:

UpdateTokenStatusWorker (existing)
    ↓ (enqueues partner tokens)
PartnerTokenVerificationWorker (this MR)
    ↓ (calls service)
PartnerTokenVerificationService (previous MR)
    ↓ (sends to SDRS)
SDRS (external service)

References

• Follows up: !197819 (merged)(Add partner token verification service)
• Issue: #551358 (closed) - Implement partner token verification for Secret Detection

How to set up and validate locally

Prerequisites:

Ensure you have completed the setup from the previous MR.

Testing the worker:

1. Test successful async processing:

   # Create a test finding
   finding = create(:vulnerabilities_finding,
     project: project,
     report_type: 'secret_detection',
     raw_metadata: {
       'token_type' => 'github_pat',
       'raw_source_code_extract' => 'ghp_test123'
     }.to_json
   )
   
   # Enqueue the job
   Security::SecretDetection::PartnerTokenVerificationWorker.perform_async(finding.id, user.id)
   
   # Process the job
   Security::SecretDetection::PartnerTokenVerificationWorker.drain
   
   # Check the status
   puts finding.reload.finding_token_status.status # => "pending"

2. Test retry behavior for network errors:

   # Simulate network timeout, use WebMock
   stub_request(:post, /api\/v1\/token\/verify/)
     .to_timeout
     .times(2)
     .then.to_return(status: 202)
   
   # Enqueue and process
   job_id = Security::SecretDetection::PartnerTokenVerificationWorker.perform_async(finding.id, user.id)
   
   # Check retry count
   retry_set = Sidekiq::RetrySet.new
   job = retry_set.find { |j| j.jid == job_id }
   puts job['retry_count'] # Will increment on each retry

3. Test non-retryable errors:

   # Disable SDRS to trigger config error
   ApplicationSetting.current.update!(sdrs_enabled: false)
   
   # This should fail immediately without retries
   Security::SecretDetection::PartnerTokenVerificationWorker.new.perform(finding.id, user.id)
   
   # Check Sidekiq - no retries queued
   Sidekiq::RetrySet.new.size # => 0

Monitoring in development:

1. Watch Sidekiq processing:

   # In one terminal
   bundle exec sidekiq
   
   # In another terminal
   tail -f log/sidekiq.log | grep PartnerTokenVerificationWorker

2. Check job metadata:

   # After job completes, check logged metadata
   job = Sidekiq::Queue.new.find { |j| j.klass == 'Security::SecretDetection::PartnerTokenVerificationWorker' }
   puts job.args # [finding_id, user_id]

3. Monitor retry behavior:

   # Check retry queue
   Sidekiq::RetrySet.new.each do |job|
     puts "#{job.klass}: retry #{job['retry_count']}/#{job['retry']}"
     puts "Next retry at: #{Time.at(job.at)}"
   end

MR acceptance checklist

• Code follows GitLab development guidelines
• Tests cover success, failure, and retry scenarios
• Worker is idempotent and handles interruptions
• Exponential backoff prevents retry storms
• Logging provides debugging information
• Documentation updated
• Performance impact assessed (async processing)

Related to #551358 (closed)