Remove old_set_statuses check after migration period in EventDeduplicationService

Summary

After the migration period (24-48 hours) from !14584, remove the backward compatibility check for old_set_statuses in the filter_batch_for_new_ids method.

Background

The consumption event deduplication system was migrated from a set-based approach to individual Redis keys for atomic in-progress and processed tracking. During the migration period, we maintain dual-write to the old set for backward compatibility.

Tasks

• Verify all workers are using the new key format (in-progress and processed keys)
• Confirm no double-billing issues detected during migration period
• Remove old_set_statuses check from filter_batch_for_new_ids method in app/services/billing/usage/event_deduplication_service.rb
• Remove mark_in_old_set method and its call in mark_as_processed
• Remove REDIS_KEY_PREFIX constant and redis_key method (if no longer needed)
• Update tests to remove old set-related test cases
• Update Redis call count comment (currently says "3 Redis calls per batch")

Code Location

File: app/services/billing/usage/event_deduplication_service.rb

Method: filter_batch_for_new_ids

# TODO: Remove old_set_statuses check after migration period (see mark_in_old_set).
def filter_batch_for_new_ids(batch)
  processed_keys = batch.map { |id| processed_key(id) }
  in_progress_keys = batch.map { |id| in_progress_key(id) }

  old_set_statuses, processed_statuses, in_progress_statuses = redis_client.pipelined do |pipeline|
    pipeline.smismember(redis_key, *batch)  # <- Remove this line
    pipeline.mget(*processed_keys)
    pipeline.mget(*in_progress_keys)
  end

  batch.each_with_index.filter_map do |id, i|
    next if id_exists_in_redis?(old_set_statuses[i])  # <- Remove this line
    next if processed_statuses[i] == MARKER_VALUE
    next if in_progress_statuses[i] == MARKER_VALUE

    id
  end
end

Also remove the mark_in_old_set method:

# TODO: Remove after migration period (~24-48 hours) when all workers use new key format.
def mark_in_old_set(event_ids_to_mark)
  redis_client.call('EVAL', SCRIPT_MARK, 1, redis_key, TTL.to_i, *event_ids_to_mark)
end

Related

• Parent MR: !14584
• Original issue: #15689