Implement new checksum approach for Ci::JobDefinition

What does this MR do and why?

Implement new checksum approach for Ci::JobDefinition

• The new approach applies normalized defaults (like interruptible) before checksum generation
• This ensures that job definitions with implicit defaults are properly deduplicated.
• The old approach applied defaults after checksum calculation, leading to potential duplicate job definitions.
• Self-managed instances use the new approach immediately
• Add feature flag ci_job_definitions_new_checksum to derisk gitlab.com
• Add NEW_CHECKSUM_PARTITION_THRESHOLD constant (partition 109) to determine when to use the new approach on GitLab.com

References

Resolves Include normalized defaults in checksum calcula... (#577902)

Screenshots or screen recordings

N/A

How to set up and validate locally

In Rails console:

# Create a project and basic config
project = FactoryBot.create(:project, path: "#{Time.now.to_i}-577902", namespace: Namespace.first)
config = { options: { script: ['echo test'] } }

ENV['GITLAB_SIMULATE_SAAS'] = "1" # gitlab.com simulate
Feature.enable(:ci_job_definitions_new_checksum, :instance)
# Test old approach (partition below threshold)
old_partition = Ci::JobDefinition::NEW_CHECKSUM_PARTITION_THRESHOLD - 1
old_job_def = Ci::JobDefinition.fabricate(
  config: config,
  project_id: project.id,
  partition_id: old_partition
)

# Test new approach (partition at/above threshold)
new_partition = Ci::JobDefinition::NEW_CHECKSUM_PARTITION_THRESHOLD
new_job_def = Ci::JobDefinition.fabricate(
  config: config,
  project_id: project.id,
  partition_id: new_partition
)

# Verify different checksums
puts "Old checksum: #{old_job_def.checksum}"
puts "New checksum: #{new_job_def.checksum}"
puts "Checksums differ: #{old_job_def.checksum != new_job_def.checksum}"

# Verify both have the default interruptible value
puts "Old interruptible: #{old_job_def.interruptible}"
puts "New interruptible: #{new_job_def.interruptible}"

# Test with feature flag disabled (should use old approach for both)
Feature.disable(:ci_job_definitions_new_checksum, :instance)

new_job_def_ff_off = Ci::JobDefinition.fabricate(
  config: config,
  project_id: project.id,
  partition_id: new_partition
)

puts "With FF disabled, checksums match old approach: #{old_job_def.checksum == new_job_def_ff_off.checksum}"

# cleanup
project.destroy!
ENV['GITLAB_SIMULATE_SAAS'] = "0"

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.