Skip to content

Improve `InternalId.track_greatest` for imported projects

Problem to solve

During import we perform track_greatest of internal_id. This is to ensure that internal_id table holds the greatest ever used ID by the entries in database.

The Internal ID is being used to have a sequential numbers in context of the project, so issues for project start with 1, instead of 1_000_000.

It seems that there are multiple things to consider:

  1. We always run .track_greatest for each iid recorded during import,
  2. We seem to restore objects in reverse order (from the biggest),
  3. We seem to run Project#after_import => InternalId.flush_records!(project: self) that seems to resolve all internal_ids entries,

This seems that we can optimise that easily, by either:

  1. If the .flush_records is desired, we should ignore all internal id tracking code during import, as this is fully redundant,
  2. If the .flush_records is not desired, we should cache in memory the tracket greatest value instead of checking that against database.

The check against database looks as this:

D, [2019-12-09T17:09:39.322184 #64] DEBUG -- :   InternalId Create (0.4ms)  INSERT INTO "internal_ids" ("project_id", "usage", "last_value") VALUES ($1, $2, $3) RETURNING "id"  [["project_id", 43], ["usage", 1], ["last_value", 0]]
D, [2019-12-09T17:09:39.322253 #64] DEBUG -- :   ↳ app/models/internal_id.rb:200
D, [2019-12-09T17:09:39.322548 #64] DEBUG -- :    (0.1ms)  RELEASE SAVEPOINT active_record_1
D, [2019-12-09T17:09:39.322628 #64] DEBUG -- :   ↳ lib/gitlab/database.rb:301
D, [2019-12-09T17:09:39.323256 #64] DEBUG -- :   InternalId Load (0.3ms)  SELECT  "internal_ids".* FROM "internal_ids" WHERE "internal_ids"."id" = $1 LIMIT $2 FOR UPDATE  [["id", 97], ["LIMIT", 1]]
D, [2019-12-09T17:09:39.323323 #64] DEBUG -- :   ↳ app/models/internal_id.rb:50
D, [2019-12-09T17:09:39.324285 #64] DEBUG -- :   InternalId Update (0.4ms)  UPDATE "internal_ids" SET "last_value" = $1 WHERE "internal_ids"."id" = $2  [["last_value", 1], ["id", 97]]
D, [2019-12-09T17:09:39.324342 #64] DEBUG -- :   ↳ config/initializers/config_initializers_active_record_locking.rb:13

Intended users