Include option to add additional sequence_range cells from TS

Why?

TopologyService:SequenceService provides 1 billion IDs to staging cells, 100 billion to production cells and 1 trillion IDs to the legacy cell.

Cells sequence data are monitored regularly, and TS should provide an additional set of ID range (from currentMaxId) to the cell, if their consumption is crossing the hard SLO of 90% (or may be a bit more?).

What?

Currently each cell has a singular sequence_range array. This should be updated to sequence_ranges to hold multiple ranges.

env = "production"

[[cells]]
id = 1
address = "legacy.gitlab.com"
sequence_ranges = [[1, 999999999999]] # 1 trillion

[[cells]]
id = 2
address = "cell-2-example.gitlab.com"
session_prefix = "cell-2"
sequence_ranges = [[1000000000000, 1099999999999]] # 100 billion

[[cells]]
id = 3
address = "cells-3-test.gitlab.com"
session_prefix = "cell-3"
sequence_ranges = [[1100000000000, 1199999999999]] # 100 billion

And a rake gitlab:db:increase_sequence_range to be introduced in gitlab-org/gitlab repo, which will take sequence_names as input. It will fetch sequence_ranges array from Topology service and alters the sequence(s) limit with the range immediately larger than the current sequence value.

Example scenario:

  1. Let's consider cell-2 for this example and security_findings_id_seq and web_hook_logs_id_seq are at 90% usage of it's sequence range.
  2. We must have been alerted about this from pg_id_sequences saturation monitoring.
  3. As the next step, we should add the next 100 billion IDs (or the range based on the type of the cell) to cells.sequence_ranges for cell-2 in the topology-service config.toml.
    • And the new cells-2.sequence_ranges will be [[1000000000000, 1099999999999], [1200000000000, 1299999999999]]. New sequence_range = [currentMaxId + 1, 10 ** 11 - 1].

    1. Open a CR to run gitlab:db:increase_sequence_range['security_findings_id_seq', 'web_hook_logs_id_seq'] on cell-2 instance.

          desc 'Bumps up the sequence range for given sequence_names'
    task increase_sequence_range: :environment do |_, args|
      sequence_ranges = Gitlab::TopologyServiceClient::CellService.new.cell_sequence_ranges

      Gitlab::Database::EachDatabase.each_connection do |connection, _database_name|
        args.each do |sequence_name|
          with_lock_retries do
            last_sequence = connection.exec_query("SELECT last_value FROM #{sequence_name}")
            minval, maxval = sequence_ranges.find { |min, _max| min > last_sequence }

            sql = <<~SQL
              ALTER SEQUENCE #{sequence_name}
              START #{minval} RESTART #{minval} MINVALUE #{minval} MAXVALUE #{maxval}
            SQL

            connection.execute(sql)

            # Also updates the `maxval` (sequence_ranges.last[1]) for all new IDs in alter_new_sequences_range function (https://gitlab.com/gitlab-org/gitlab/blob/e51a48ba87ecbc70d2c65976e320773f78445045/lib/gitlab/database/alter_cell_sequences_range.rb#L52).
          end
        end
      end
    end
  4. Monitoring should start having the new consumption percentage for both of the sequences.

Pros:

  • Since sequence range are altered only for the saturating sequences, others will continue to use their available sequences from the initially given range.
  • IDs requiring additional range will consume the next possible range from sequence_ranges.
  • Newly created IDs will have the first range minval: sequence_ranges[0][0] and maxval: sequence_ranges[0][1], and continue the same process if it saturates.

Cons:

  • Even though only a few sequences need to get their sequences bumped, we will end up adding an entire 100 billion ID to the cell.
Edited by Prabakaran Murugesan