Skip to content

Update BatchCounter to stop counting at finish value [RUN AS-IF-FOSS]

What does this MR do?

Given 7181 records (with id between 1 and 7181) in the issues table:

[4] pry(main)> Issue.count
   (2.2ms)  SELECT COUNT(*) FROM "issues"
=> 7181
[5] pry(main)> Issue.first.id
  Issue Load (1.0ms)  SELECT "issues".* FROM "issues" ORDER BY "issues"."id" ASC LIMIT 1
=> 1
[6] pry(main)> Issue.last.id
  Issue Load (0.7ms)  SELECT "issues".* FROM "issues" ORDER BY "issues"."id" DESC LIMIT 1
=> 7181

Before

When we ask for a count of records with id between 1 and 7000, we get a larger result (7181). This is because BatchCounter counts the entire final batch (1333 records up to 7999), which goes past the finish value.

[1] pry(main)> Gitlab::Database::BatchCount.batch_count(Issue, batch_size: 1333, start: 1, finish: 7000)
   (2.8ms)  SELECT COUNT("issues"."id") FROM "issues" WHERE "issues"."id" >= 1 AND "issues"."id" < 1334
   (2.2ms)  SELECT COUNT("issues"."id") FROM "issues" WHERE "issues"."id" >= 1334 AND "issues"."id" < 2667
   (1.7ms)  SELECT COUNT("issues"."id") FROM "issues" WHERE "issues"."id" >= 2667 AND "issues"."id" < 4000
   (1.4ms)  SELECT COUNT("issues"."id") FROM "issues" WHERE "issues"."id" >= 4000 AND "issues"."id" < 5333
   (1.4ms)  SELECT COUNT("issues"."id") FROM "issues" WHERE "issues"."id" >= 5333 AND "issues"."id" < 6666
   (1.3ms)  SELECT COUNT("issues"."id") FROM "issues" WHERE "issues"."id" >= 6666 AND "issues"."id" < 7999
=> 7181

After

With the adjustments in this MR, a count of records between 1 and 7000 gives us the correct result (7000). The final batch is adjusted so that it does not query past the finish value.

[3] pry(main)> Gitlab::Database::BatchCount.batch_count(Issue, batch_size: 1333, start: 1, finish: 7000)
   (0.8ms)  SELECT COUNT("issues"."id") FROM "issues" WHERE "issues"."id" >= 1 AND "issues"."id" < 1334
   (0.8ms)  SELECT COUNT("issues"."id") FROM "issues" WHERE "issues"."id" >= 1334 AND "issues"."id" < 2667
   (1.4ms)  SELECT COUNT("issues"."id") FROM "issues" WHERE "issues"."id" >= 2667 AND "issues"."id" < 4000
   (1.0ms)  SELECT COUNT("issues"."id") FROM "issues" WHERE "issues"."id" >= 4000 AND "issues"."id" < 5333
   (0.8ms)  SELECT COUNT("issues"."id") FROM "issues" WHERE "issues"."id" >= 5333 AND "issues"."id" < 6666
   (0.6ms)  SELECT COUNT("issues"."id") FROM "issues" WHERE "issues"."id" >= 6666 AND "issues"."id" < 7001
=> 7000

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Alishan Ladhani

Merge request reports