Prevent race condition on db load balancer

What does this MR do and why?

Use a lua script to prevent race conditions on the db load balancer. See issue below for important details.

References

https://gitlab.com/gitlab-org/gitlab/-/issues/573726

Screenshots or screen recordings

Before	After

How to set up and validate the Lua script locally

1. Load rails console
2. Copy and paste the following:

# Get Redis connection
redis = Gitlab::Redis::DbLoadBalancing.with { |conn| conn }

# Load Lua script
script_content = Gitlab::Database::LoadBalancing::Sticking::LUA_SET_LSN_IF_GREATER
script_sha = redis.script(:load, script_content)
test_key = 'test:manual:lsn'

puts "\n" + "="*80
puts "Testing Lua Script: set_lsn.lua"
puts "="*80
puts "Script SHA: #{script_sha}"
puts "="*80 + "\n\n"

# Test 1: Set initial value (no current value exists)
puts "Test 1: Set initial value (no current value exists)"
puts "-" * 80
redis.del(test_key)
result = redis.evalsha(script_sha, keys: [test_key], argv: ['0/16B3DC0', 30])
value = redis.get(test_key)
ttl = redis.ttl(test_key)
puts "  Result: #{result} (expected: 1)"
puts "  Value:  #{value} (expected: 0/16B3DC0)"
puts "  TTL:    #{ttl} seconds (expected: ~30)"
puts "  Status: #{result == 1 && value == '0/16B3DC0' ? '✓ PASS' : '✗ FAIL'}"
puts "\n"

# Test 2: Update with greater LSN (higher offset)
puts "Test 2: Update with greater LSN (higher offset)"
puts "-" * 80
result = redis.evalsha(script_sha, keys: [test_key], argv: ['0/16B3DD0', 30])
value = redis.get(test_key)
puts "  Result: #{result} (expected: 1)"
puts "  Value:  #{value} (expected: 0/16B3DD0)"
puts "  Status: #{result == 1 && value == '0/16B3DD0' ? '✓ PASS' : '✗ FAIL'}"
puts "\n"

# Test 3: Reject lower LSN (lower offset)
puts "Test 3: Reject lower LSN (lower offset)"
puts "-" * 80
result = redis.evalsha(script_sha, keys: [test_key], argv: ['0/16B3DC0', 30])
value = redis.get(test_key)
ttl = redis.ttl(test_key)
puts "  Result: #{result} (expected: 0)"
puts "  Value:  #{value} (expected: 0/16B3DD0 - unchanged)"
puts "  TTL:    #{ttl} seconds (expected: ~30 - refreshed)"
puts "  Status: #{result == 0 && value == '0/16B3DD0' ? '✓ PASS' : '✗ FAIL'}"
puts "\n"

# Test 4: Reject equal LSN
puts "Test 4: Reject equal LSN"
puts "-" * 80
result = redis.evalsha(script_sha, keys: [test_key], argv: ['0/16B3DD0', 30])
value = redis.get(test_key)
puts "  Result: #{result} (expected: 0)"
puts "  Value:  #{value} (expected: 0/16B3DD0 - unchanged)"
puts "  Status: #{result == 0 && value == '0/16B3DD0' ? '✓ PASS' : '✗ FAIL'}"
puts "\n"

# Test 5: File boundary comparison (1/0 > 0/FFFFFFFF)
puts "Test 5: File boundary comparison (1/0 > 0/FFFFFFFF)"
puts "-" * 80
redis.del(test_key)
redis.evalsha(script_sha, keys: [test_key], argv: ['0/FFFFFFFF', 30])
puts "  Set current: 0/FFFFFFFF"
result = redis.evalsha(script_sha, keys: [test_key], argv: ['1/0', 30])
value = redis.get(test_key)
puts "  Result: #{result} (expected: 1)"
puts "  Value:  #{value} (expected: 1/0)"
puts "  Status: #{result == 1 && value == '1/0' ? '✓ PASS' : '✗ FAIL'}"
puts "\n"

# Test 6: Hex file number comparison (10/0 > 2/0, not string comparison)
puts "Test 6: Hex file number comparison (0x10 = 16 > 2)"
puts "-" * 80
redis.del(test_key)
redis.evalsha(script_sha, keys: [test_key], argv: ['2/0', 30])
puts "  Set current: 2/0"
result = redis.evalsha(script_sha, keys: [test_key], argv: ['10/0', 30])
value = redis.get(test_key)
puts "  Result: #{result} (expected: 1)"
puts "  Value:  #{value} (expected: 10/0)"
puts "  Note:   0x10 (16) > 0x2 (2) - numeric comparison, not string"
puts "  Status: #{result == 1 && value == '10/0' ? '✓ PASS' : '✗ FAIL'}"
puts "\n"

# Test 7: Offset comparison within same file
puts "Test 7: Offset comparison within same file"
puts "-" * 80
redis.del(test_key)
redis.evalsha(script_sha, keys: [test_key], argv: ['5/1000', 30])
puts "  Set current: 5/1000"
result = redis.evalsha(script_sha, keys: [test_key], argv: ['5/2000', 30])
value = redis.get(test_key)
puts "  Test 5/2000 > 5/1000"
puts "  Result: #{result} (expected: 1)"
puts "  Value:  #{value} (expected: 5/2000)"
puts "  Status: #{result == 1 && value == '5/2000' ? '✓ PASS' : '✗ FAIL'}"
result2 = redis.evalsha(script_sha, keys: [test_key], argv: ['5/500', 30])
value2 = redis.get(test_key)
puts "  Test 5/500 < 5/2000"
puts "  Result: #{result2} (expected: 0)"
puts "  Value:  #{value2} (expected: 5/2000 - unchanged)"
puts "  Status: #{result2 == 0 && value2 == '5/2000' ? '✓ PASS' : '✗ FAIL'}"
puts "\n"

# Test 8: Invalid LSN handling (malformed current)
puts "Test 8: Invalid LSN handling (malformed current LSN)"
puts "-" * 80
redis.del(test_key)
redis.set(test_key, 'invalid-lsn', ex: 30)
puts "  Set current: invalid-lsn"
result = redis.evalsha(script_sha, keys: [test_key], argv: ['0/16B3DC0', 30])
value = redis.get(test_key)
puts "  Result: #{result} (expected: 1)"
puts "  Value:  #{value} (expected: 0/16B3DC0 - replaced invalid)"
puts "  Status: #{result == 1 && value == '0/16B3DC0' ? '✓ PASS' : '✗ FAIL'}"
puts "\n"

# Test 9: Invalid LSN handling (malformed new)
puts "Test 9: Invalid LSN handling (malformed new LSN)"
puts "-" * 80
redis.del(test_key)
redis.evalsha(script_sha, keys: [test_key], argv: ['0/16B3DC0', 30])
puts "  Set current: 0/16B3DC0"
result = redis.evalsha(script_sha, keys: [test_key], argv: ['invalid-lsn', 30])
value = redis.get(test_key)
puts "  Result: #{result} (expected: 0)"
puts "  Value:  #{value} (expected: 0/16B3DC0 - rejected invalid)"
puts "  Status: #{result == 0 && value == '0/16B3DC0' ? '✓ PASS' : '✗ FAIL'}"
puts "\n"

# Test 10: TTL refresh on non-update
puts "Test 10: TTL refresh when value not updated"
puts "-" * 80
redis.del(test_key)
redis.evalsha(script_sha, keys: [test_key], argv: ['0/16B3DC0', 10])
puts "  Set current: 0/16B3DC0 with TTL=10"
sleep 3
ttl_before = redis.ttl(test_key)
puts "  TTL after 3s: #{ttl_before} (expected: ~7)"
result = redis.evalsha(script_sha, keys: [test_key], argv: ['0/16B3DB0', 60])
ttl_after = redis.ttl(test_key)
value = redis.get(test_key)
puts "  Tried lower LSN with TTL=60"
puts "  Result: #{result} (expected: 0 - not updated)"
puts "  Value:  #{value} (expected: 0/16B3DC0 - unchanged)"
puts "  TTL:    #{ttl_after} (expected: ~60 - refreshed)"
puts "  Status: #{result == 0 && value == '0/16B3DC0' && ttl_after > ttl_before ? '✓ PASS' : '✗ FAIL'}"
puts "\n"

# Test 11: TTL set on update
puts "Test 11: TTL set when value updated"
puts "-" * 80
redis.del(test_key)
redis.evalsha(script_sha, keys: [test_key], argv: ['0/16B3DC0', 10])
result = redis.evalsha(script_sha, keys: [test_key], argv: ['0/16B3DD0', 60])
ttl = redis.ttl(test_key)
value = redis.get(test_key)
puts "  Result: #{result} (expected: 1)"
puts "  Value:  #{value} (expected: 0/16B3DD0)"
puts "  TTL:    #{ttl} (expected: ~60)"
puts "  Status: #{result == 1 && value == '0/16B3DD0' && ttl > 55 ? '✓ PASS' : '✗ FAIL'}"
puts "\n"

# Cleanup
redis.del(test_key)

puts "="*80
puts "All Tests Complete - Test key cleaned up"
puts "="*80
puts "\nQuick commands for manual testing:"
puts "  redis = Gitlab::Redis::DbLoadBalancing.with { |conn| conn }"
puts "  script_sha = redis.script(:load, Gitlab::Database::LoadBalancing::Sticking::LUA_SET_LSN_IF_GREATER)"
puts "  redis.evalsha(script_sha, keys: ['test:key'], argv: ['0/16B3DC0', 30])"
puts "  redis.get('test:key')"
puts "  redis.ttl('test:key')"
puts "  redis.del('test:key')"
puts "\n"

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.