Use direct SQL to generate internal ids

What does this MR do?

Related issue: #325861 (closed)

Reimplement changes from !64240 (closed) that removes database round-trips and explicit locking when updating internal_ids. We make the assumption on generating a new value that most operations will modify existing records, so we prefer to attempt to update the row first. After some discussion, we decided to put consider putting the changes behind a feature flag, so this MR was created to do that (rather than the direct cutover in the previous MR).

Example of creation of object with internal_id, where no existing record exists (create_record!):

  Update InternalId (0.7ms)  UPDATE "internal_ids" SET "last_value" = ("internal_ids"."last_value" + 1) WHERE "internal_ids"."project_id" = 7 AND "internal_ids"."usage" = 0 RETURNING "last_value" /*application:console,line:/app/models/internal_id.rb:91:in `update_record!'*/
  TRANSACTION (0.1ms)  SAVEPOINT active_record_1 /*application:console,line:/app/models/concerns/atomic_internal_id.rb:211:in `block in project_init'*/
   (2.1ms)  SELECT MAX("issues"."iid") FROM "issues" WHERE "issues"."project_id" = 7 /*application:console,line:/app/models/concerns/atomic_internal_id.rb:211:in `block in project_init'*/
  InternalId Create (0.7ms)  INSERT INTO "internal_ids" ("project_id", "usage", "last_value") VALUES (7, 0, 1) RETURNING "id" /*application:console,line:/app/models/internal_id.rb:102:in `block in create_record!'*/

First attempts to update the existing record, and because none is found, it creates a new record. If the new record were to conflict with a concurrent insert, it retries the update.

Example of update of object with internal_id, where a record exists (update_record!):

 Update InternalId (0.3ms)  UPDATE "internal_ids" SET "last_value" = ("internal_ids"."last_value" + 1) WHERE "internal_ids"."project_id" = 7 AND "internal_ids"."usage" = 0 RETURNING "last_value" /*application:console,line:/app/models/internal_id.rb:91:in `update_record!'*/

It updates the record only.

Screenshots or Screencasts (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

• I have included changelog trailers, or none are needed. (Does this MR need a changelog?)
• I have added/updated documentation, or it's not needed. (Is documentation required?)
• I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?)
• I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?)
• I have self-reviewed this MR per code review guidelines.
• This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines)
• I have followed the style guides.
• This change is backwards compatible across updates, or this does not apply.

Availability and Testing

• I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.)
• I have tested this MR in all supported browsers, or it's not needed.
• I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

• Label as security and @ mention @gitlab-com/gl-security/appsec
• The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
• Security reports checked/validated by a reviewer from the AppSec team

app/models/concerns/atomic_internal_id.rb

@@ -159,9 +159,8 @@ def define_instance_internal_id_methods(scope, column, init)
     # Defines class methods:
     #
     # - with_{scope}_{column}_supply
-    #   This method can be used to allocate a block of IID values during
-    #   bulk operations (importing/copying, etc). This can be more efficient
-    #   than creating instances one-by-one.
+    #   This method can be used to allocate a stream of IID values during
+    #   bulk operations (importing/copying, etc).
     #
     #   Pass in a block that receives a `Supply` instance. To allocate a new
     #   IID value, call `Supply#next_value`.
@@ -181,14 +180,8 @@ def define_singleton_internal_id_methods(scope, column, init)
         scope_attrs = ::AtomicInternalId.scope_attrs(scope_value)
         usage = ::AtomicInternalId.scope_usage(self)
 
-        generator = InternalId::InternalIdGenerator.new(subject, scope_attrs, usage, init)
-
-        generator.with_lock do
-          supply = Supply.new(generator.record.last_value)
-          block.call(supply)
-        ensure
-          generator.track_greatest(supply.current_value) if supply
-        end
+        supply = Supply.new(-> { InternalId.generate_next(subject, scope_attrs, usage, init) })
+        block.call(supply)
       end
     end
   end
@@ -236,14 +229,14 @@ def internal_id_read_scope(scope)
   end
 
   class Supply
-    attr_reader :current_value
+    attr_reader :generator
 
-    def initialize(start_value)
-      @current_value = start_value
+    def initialize(generator)
+      @generator = generator
     end
 
     def next_value
-      @current_value += 1
+      @generator.call
     end
   end
 end