Skip to content

Move approximate counters to batch counting

What does this MR do?

Use batch counters instead of approximate counters in usage_data

Queries

labels: count(Label)
merge_requests: count(MergeRequest)
notes: count(Note)
# todos: count(Todo),  # that was already calculated by batch counting and overridden by the approximate

Timing

@a_akgun Thank you for the times

[ gprd ] production> start=Time.now; [Gitlab::UsageData.count(::Label), Time.now - start]
[ gprd ] production> start=Time.now; [Gitlab::UsageData.count(::Label), Time.now - start]
=> [10865127, 5.407331995]
[ gprd ] production> start=Time.now; [Gitlab::UsageData.count(::MergeRequest), Time.now - start]
=> [42346375, 30.651281439]
[ gprd ] production> start=Time.now; [Gitlab::UsageData.count(::Todo), Time.now - start]
=> [43487615, 31.160777776]
[ gprd ] production> start=Time.now; [Gitlab::UsageData.count(::Note), Time.now - start]
=> [263697806, 542.07525909]
[ gprd ] production> 
[4] pry(main)> Gitlab::UsageData.count(Label)
   (0.3ms)  SELECT "features"."key" FROM "features"
  Feature::FlipperGate Load (0.5ms)  SELECT "feature_gates".* FROM "feature_gates" WHERE "feature_gates"."feature_key" = 'usage_ping_batch_counter'
   (1.9ms)  SELECT MIN("labels"."id") FROM "labels"
   (0.4ms)  SELECT MAX("labels"."id") FROM "labels"
   (0.3ms)  SELECT COUNT("labels"."id") FROM "labels" WHERE "labels"."id" BETWEEN 1 AND 100000
=> 108
[5] pry(main)> Gitlab::UsageData.count(MergeRequest)
   (3.8ms)  SELECT MIN("merge_requests"."id") FROM "merge_requests"
   (0.4ms)  SELECT MAX("merge_requests"."id") FROM "merge_requests"
   (0.4ms)  SELECT COUNT("merge_requests"."id") FROM "merge_requests" WHERE "merge_requests"."id" BETWEEN 1 AND 100000
=> 32
[6] pry(main)> Gitlab::UsageData.count(Note)
   (3.0ms)  SELECT MIN("notes"."id") FROM "notes"
   (0.6ms)  SELECT MAX("notes"."id") FROM "notes"
   (0.6ms)  SELECT COUNT("notes"."id") FROM "notes" WHERE "notes"."id" BETWEEN 1 AND 100000
=> 1491
[7] pry(main)> Gitlab::UsageData.count(Todo)
   (2.4ms)  SELECT MIN("todos"."id") FROM "todos"
   (0.6ms)  SELECT MAX("todos"."id") FROM "todos"
   (1.0ms)  SELECT COUNT("todos"."id") FROM "todos" WHERE "todos"."id" BETWEEN 1 AND 100000
=> 449

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team

Related to #210332 (closed)

Edited by Mayra Cabrera

Merge request reports