Update NamespaceMonthlyUsage#increase_usage to batch updates in Redis

What does this MR do and why?

This is part 1 of #490968.

In part 1 we increment usage in redis by namespace. This is to reduce contention at the database level as many updates may hit the same record in some environments. Redis can handle this better. We've had to disable some workers in non-gitlab.com environments due to high contention.

In part 2 we flush that data to the databse at a regular interval via a cron job.

The full outline is here: !194219

TODO: Lease needs to wrap transaction otherwise it can keep open transation

Steps to Verify

  1. Enable the feature flag Feature.enable(:fix_minute_contention)
  2. Run ci jobs
  3. Check view usage data to ensure data goes to the right table.
  4. Check redis to ensure that the data exists for the namespace in the rails console.
# Get the namespace ID you want to check
namespace_id = Namespace.find_by(name: 'your-namespace-name').id

# Create a RedisBatchUsage instance for that namespace
redis_usage = Ci::Minutes::RedisBatchUsage.new(namespace_id: namespace_id)

# Check if data exists
redis_usage.key_exists?  # Should return true if data exists

# Get specific field values
redis_usage.fetch_field('amount_used')
redis_usage.fetch_field('shared_runners_duration')

# Get all fields
redis_usage.fetch_all_fields  # 
  1. Check that the total usage reflects the data in the db added to the redis data.
# Get the namespace
namespace = Namespace.find_by(name: 'your-namespace-name')

# Get the Ci::Minutes::Usage instance for the namespace
usage = Ci::Minutes::Usage.new(namespace)

# This will include both DB and Redis data because it ultimately calls
# NamespaceMonthlyUsage#amount_used which has been modified to include Redis data
total_minutes = usage.total_minutes_used

# To verify this is correct, you can compare with the individual components:
monthly_usage = Ci::Minutes::NamespaceMonthlyUsage.find_or_create_current(namespace_id: namespace.id)
db_amount = monthly_usage.read_attribute(:amount_used)
redis_usage = Ci::Minutes::RedisBatchUsage.new(namespace_id: namespace.id)
redis_amount = redis_usage.fetch_field('amount_used')

puts "Database amount: #{db_amount}"
puts "Redis amount: #{redis_amount}"
puts "Combined total from components: #{db_amount + redis_amount}"
puts "Total from Ci::Minutes::Usage: #{total_minutes}"
Edited by Allison Browne

Merge request reports

Loading