Skip to content
  • Pedro Pombeiro @pedropombeiro ·
    Maintainer

    @stanhu it would probably be safer if we also patch the Ci::Runner records, in addition to Ci::PendingBuild. Would you mind adding that for users that might come across this script?

  • Author Owner

    @pedropombeiro Which field in Ci::Runner records needs patching? Is that handled by the Tagging updates?

  • Pedro Pombeiro @pedropombeiro ·
    Maintainer

    @stanhu oh, you're right! It's been a while since I've looked at tags, and I assumed that Ci::PendingBuild and Ci::Runner worked the same way, should have checked it 👍

  • Author Owner

    @gerardo reported that fetching the tags might result in a statement timeout. We might want to disable statement timeouts there.

  • A came across this script linked from here: https://docs.gitlab.com/runner/faq/#no-unique-index-found-for-name

    After successfully running it, should I manually (or should the ruby script) clean up the duplicate tag ID's in the 'tags' table once there are no references to them in the 'taggings' table?

  • Author Owner

    Yes, that documentation appears to be missing the following SQL commands:

    DELETE FROM tags
    WHERE id NOT IN (
        SELECT MIN(id)
        FROM tags
        GROUP BY name
    );
    REINDEX INDEX CONCURRENTLY index_tags_on_name;
    REINDEX INDEX CONCURRENTLY index_tags_on_name_trigram;
  • Got the following error:

    /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/activerecord-7.0.8.4/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `exec': PG::UndefinedTable: ERROR:  relation "dast_scanner_profiles_tags" does not exist (ActiveRecord::StatementInvalid)
    LINE 9:  WHERE a.attrelid = '"dast_scanner_profiles_tags"'::regclass

    Existing dast tables:

     public | dast_pre_scan_verification_steps                          | table             | gitlab
     public | dast_pre_scan_verifications                               | table             | gitlab
     public | dast_profile_schedules                                    | table             | gitlab
     public | dast_profiles                                             | table             | gitlab
     public | dast_profiles_pipelines                                   | table             | gitlab
     public | dast_profiles_tags                                        | table             | gitlab
     public | dast_scanner_profiles                                     | table             | gitlab
     public | dast_scanner_profiles_builds                              | table             | gitlab
     public | dast_site_profile_secret_variables                        | table             | gitlab
     public | dast_site_profiles                                        | table             | gitlab
     public | dast_site_profiles_builds                                 | table             | gitlab
     public | dast_site_tokens                                          | table             | gitlab
     public | dast_site_validations                                     | table             | gitlab
     public | dast_sites                                                | table             | gitlab

    Seems that tables have been dropped: #442744 (comment 1909999726)

    And I'm confused between:

    • FAQ
    DROP INDEX IF EXISTS index_tags_on_name; -- Drop a potentially invalid index
    CREATE UNIQUE INDEX index_tags_on_name ON tags USING btree (name);
    • and the previous comment:
    REINDEX INDEX CONCURRENTLY index_tags_on_name;
    REINDEX INDEX CONCURRENTLY index_tags_on_name_trigram;
    Edited by Marc Finet (Rtone)
  • I'm was seeing the same error noted by @marcfinet_rtone above: ERROR: relation "dast_scanner_profiles_tags" does not exist. As also noted by @stanhu it doesn't actually resolve the duplicate tag values on it's own when it's running without error -- requiring users to also go into the pg console and run arbitrary SQL commands.

    Below is my attempt at an updated version of @stanhu 's original script that removes the breaking reference to the dropped table, adds on deletion of the duplicate records from the tags table, and also adds dropping and re-creating the tag name index. (In other words, it tries to be an single fix.). You can create the file locally and then run it in the gitlab-rails context with:

    sudo gitlab-rails runner /full/path/to/dedudpe.rb

    Also created as it's own snippet at $4780138.

    # enable query logging
    ActiveRecord::Base.logger = Logger.new(STDOUT)
    
    # query for the distinct set of "good" tags and any duplicates 
    good_tag_ids = ActiveRecord::Base.connection.select_all('SELECT name, MIN(id) AS id FROM tags GROUP BY name').rows.to_h
    bad_tag_map = ActiveRecord::Base.connection.select_all('SELECT id, name FROM tags WHERE id NOT IN (SELECT MIN(id) FROM tags GROUP BY name)').rows.to_h
    bad_tags = bad_tag_map.keys
    tag_remap = bad_tag_map.map { |id, name| [id, good_tag_ids[name]] }.to_h
    
    # taggings table interface
    class TempTaggings < ActiveRecord::Base
      include EachBatch
    
      self.table_name = "taggings"
    end
    
    # tags table interface
    class TempTags < ActiveRecord::Base
      include EachBatch
    
      self.table_name = "tags"
    end
    
    # for pending builds with the duplicate tag id values, update them with the selected good tag id
    Ci::PendingBuild.all.each_batch do |batch|
      batch.each do |build|
        tag_ids = build.tag_ids
    
        next unless tag_ids & bad_tags
    
        build.tag_ids = tag_ids.map { |x| tag_remap.fetch(x, x) }
        build.save!
      end
    end
    
    # for taggings entries with the duplicate tag id values, update them with the selected good tag id
    TempTaggings.all.where(tag_id: bad_tags).each_batch(of: 10000) do |batch|
      batch.each do |row|
        new_tag_id = tag_remap.fetch(row.tag_id)
        row.update!(tag_id: new_tag_id)
      end
    end
    
    # for additional gitlab ee specific tables, update any duplicate tag ids ith the selected good tag id
    if Gitlab.ee?
      models = [Dast::ProfileTag]
    
      models.each do |model|
        model.all.where(tag_id: bad_tags).each do |row|
          new_tag_id = tag_remap.fetch(row.tag_id)
          row.update!(tag_id: new_tag_id)
        end
      end
    end
    
    # references to the duplicate tags should now be replaced, so we can remove the duplicate tag entries to allow for unique index
    TempTags.all.where(id: bad_tag_map.keys).each_batch(of: 10000) do |batch|
      batch.destroy_all
    end
    
    # drop the potentially bad index and re-create it
    TempTags.connection.exec_query("DROP INDEX IF EXISTS index_tags_on_name;")
    TempTags.connection.exec_query("CREATE UNIQUE INDEX index_tags_on_name ON tags USING btree (name);")
    
  • We're seeing a similar issue (and were sent here by GitLab support) when upgrading from 17.5.3 to 17.5.4, but unfortunately neither the original script, nor Kevin's fixed version (for the missing dast_scanner_profile_tags table issue) work in our case.

    Just in case anybody else is in a similar situation and doesn't find anything with this specific issue, here is the error message:

    PG::ForeignKeyViolation: ERROR:  insert or update on table "ci_build_tags_100" violates foreign key constraint "fk_rails_8284d35c66"
    DETAIL:  Key (tag_id)=(95) is not present in table "tags".
  • Pedro Pombeiro @pedropombeiro ·
    Maintainer

    @jangrewe Seems like the script would need to handle that table as well, maybe the following would work?

    ActiveRecord::Base.logger = Logger.new(STDOUT)
    
    good_tag_ids = ActiveRecord::Base.connection.select_all('SELECT name, MIN(id) AS id FROM tags GROUP BY name').rows.to_h
    bad_tag_map = ActiveRecord::Base.connection.select_all('SELECT id, name FROM tags WHERE id NOT IN (SELECT MIN(id) FROM tags GROUP BY name)').rows.to_h
    bad_tags = bad_tag_map.keys
    tag_remap = bad_tag_map.map { |id, name| [id, good_tag_ids[name]] }.to_h
    
    class Tagging < ActiveRecord::Base
      include EachBatch
    
      self.table_name = "taggings"
    end
    
    class DastScannerProfilesTags < ActiveRecord::Base
      self.table_name = "dast_scanner_profiles_tags"
    end
    
    Ci::BuildTag.include EachBatch;
    
    Ci::BuildTag.all.each_batch do |batch|
      batch.each do |build_tag|
        tag_id = build_tag.tag_id
    
        next unless bad_tags.include?(tag_id)
    
        build_tag.update!(tag_id: tag_remap.fetch(tag_id, tag_id))
      end
    end
    
    Ci::PendingBuild.all.each_batch do |batch|
      batch.each do |build|
        tag_ids = build.tag_ids
    
        next unless tag_ids & bad_tags
    
        build.tag_ids = tag_ids.map { |x| tag_remap.fetch(x, x) }
        build.save!
      end
    end
    
    Tagging.all.where(tag_id: bad_tags).each_batch(of: 10000) do |batch|
      batch.each do |row|
        new_tag_id = tag_remap.fetch(row.tag_id)
        row.update!(tag_id: new_tag_id)
      end
    end
    
    if Gitlab.ee?
      models = [Dast::ProfileTag, DastScannerProfilesTags]
    
      models.each do |model|
        model.all.where(tag_id: bad_tags).each do |row|
          new_tag_id = tag_remap.fetch(row.tag_id)
          row.update!(tag_id: new_tag_id)
        end
      end
    end
    Edited by Pedro Pombeiro
  • Thanks @pedropombeiro, as this is the non-working script (we don't have dast_scanner_profile_tags anymore), i assume the relevant lines are this?

    Ci::BuildTag.include EachBatch;
    
    Ci::BuildTag.all.each_batch do |batch|
      batch.each do |build_tag|
        tag_id = build_tag.tag_id
    
        next unless bad_tags.include?(tag_id)
    
        build_tag.update!(tag_id: tag_remap.fetch(tag_id, tag_id))
      end
    end

    Then i'll try to create a mashup of this and Kevin's script, to also handle the SQL. 😃

    Edit: Would this work?

    ActiveRecord::Base.logger = Logger.new(STDOUT)
    
    good_tag_ids = ActiveRecord::Base.connection.select_all('SELECT name, MIN(id) AS id FROM tags GROUP BY name').rows.to_h
    bad_tag_map = ActiveRecord::Base.connection.select_all('SELECT id, name FROM tags WHERE id NOT IN (SELECT MIN(id) FROM tags GROUP BY name)').rows.to_h
    bad_tags = bad_tag_map.keys
    tag_remap = bad_tag_map.map { |id, name| [id, good_tag_ids[name]] }.to_h
    
    class TempTaggings < ActiveRecord::Base
      include EachBatch
    
      self.table_name = "taggings"
    end
    
    class TempTags < ActiveRecord::Base
      include EachBatch
    
      self.table_name = "tags"
    end
    
    Ci::BuildTag.include EachBatch;
    
    Ci::BuildTag.all.each_batch do |batch|
      batch.each do |build_tag|
        tag_id = build_tag.tag_id
    
        next unless bad_tags.include?(tag_id)
    
        build_tag.update!(tag_id: tag_remap.fetch(tag_id, tag_id))
      end
    end
    
    Ci::PendingBuild.all.each_batch do |batch|
      batch.each do |build|
        tag_ids = build.tag_ids
    
        next unless tag_ids & bad_tags
    
        build.tag_ids = tag_ids.map { |x| tag_remap.fetch(x, x) }
        build.save!
      end
    end
    
    TempTaggings.all.where(tag_id: bad_tags).each_batch(of: 10000) do |batch|
      batch.each do |row|
        new_tag_id = tag_remap.fetch(row.tag_id)
        row.update!(tag_id: new_tag_id)
      end
    end
    
    if Gitlab.ee?
      models = [Dast::ProfileTag]
    
      models.each do |model|
        model.all.where(tag_id: bad_tags).each do |row|
          new_tag_id = tag_remap.fetch(row.tag_id)
          row.update!(tag_id: new_tag_id)
        end
      end
    end
    
    TempTags.all.where(id: bad_tag_map.keys).each_batch(of: 10000) do |batch|
      batch.destroy_all
    end
    
    TempTags.connection.exec_query("DROP INDEX IF EXISTS index_tags_on_name;")
    TempTags.connection.exec_query("CREATE UNIQUE INDEX index_tags_on_name ON tags USING btree (name);")
    Edited by Jan Grewe
  • Pedro Pombeiro @pedropombeiro ·
    Maintainer

    @jangrewe actually Ci::BuildTag is very similar to TempTaggings (it should be named Ci::BuildTagging tbh), so let's harmonize it:

    ActiveRecord::Base.logger = Logger.new(STDOUT)
    
    good_tag_ids = ActiveRecord::Base.connection.select_all('SELECT name, MIN(id) AS id FROM tags GROUP BY name').rows.to_h
    bad_tag_map = ActiveRecord::Base.connection.select_all('SELECT id, name FROM tags WHERE id NOT IN (SELECT MIN(id) FROM tags GROUP BY name)').rows.to_h
    bad_tags = bad_tag_map.keys
    tag_remap = bad_tag_map.map { |id, name| [id, good_tag_ids[name]] }.to_h
    
    class TempRunnerTagging < Ci::ApplicationRecord
      self.table_name = :ci_runner_taggings
      self.primary_key = :id
    end
    
    class TempTaggings < ActiveRecord::Base
      self.table_name = "taggings"
    end
    
    class TempTags < ActiveRecord::Base
      include EachBatch
    
      self.table_name = "tags"
    end
    
    Ci::PendingBuild.all.each_batch do |batch|
      batch.each do |build|
        tag_ids = build.tag_ids
    
        next unless tag_ids & bad_tags
    
        build.tag_ids = tag_ids.map { |x| tag_remap.fetch(x, x) }
        build.save!
      end
    end
    
    tagging_models = [Ci::BuildTag, TempRunnerTagging, TempTaggings]
    tagging_models .each do |tagging_model|
      tagging_model.include EachBatch;
    
      tagging_model.all.where(tag_id: bad_tags).each_batch(of: 10000) do |batch|
        batch.each do |row|
          new_tag_id = tag_remap.fetch(row.tag_id)
          if tagging_model == TempRunnerTagging
            row_exists = tagging_model.where(tag_id: new_tag_id, runner_id: row.runner_id).exists?
            if row_exists
              row.destroy!
            else
              row.update!(tag_id: new_tag_id)
            end
          else
            row.update!(tag_id: new_tag_id)
          end
        end
      end
    end
    
    if Gitlab.ee?
      tagging_models = [Dast::ProfileTag]
    
      tagging_models.each do |tagging_model|
        tagging_model.all.where(tag_id: bad_tags).each do |row|
          new_tag_id = tag_remap.fetch(row.tag_id)
          row.update!(tag_id: new_tag_id)
        end
      end
    end
    
    TempTags.all.where(id: bad_tag_map.keys).each_batch(of: 10000) do |batch|
      batch.destroy_all
    end
    
    TempTags.connection.exec_query("DROP INDEX IF EXISTS index_tags_on_name;")
    TempTags.connection.exec_query("CREATE UNIQUE INDEX index_tags_on_name ON tags USING btree (name);")
    Edited by Pedro Pombeiro
  • Author Owner

    @pedropombeiro It sounds like we should move this script into a Rake task so we can properly test and manage it based on GitLab versions?

  • Pedro Pombeiro @pedropombeiro ·
    Maintainer

    @stanhu I feel like this is something that should be handled in a more holistic way by #370622. Do you mean having a Rake task as a stop-gap solution?

  • Thanks @pedropombeiro, i've used your harmonized version and was able to upgrade to 17.5.4 successfully, now upgrading to 17.7.0.

  • Pedro Pombeiro @pedropombeiro ·
    Maintainer

    Please note that the taggings table is being replaced by ci_runner_taggings in %17.9, so the script above would need to be adapted accordingly.

  • Pedro Pombeiro @pedropombeiro ·
    Maintainer

    Created an issue to make this script a rake task: #518698 (closed)

  • Pedro Pombeiro @pedropombeiro ·
    Maintainer

    The script will be available as gitlab-rake gitlab:db:deduplicate_tags once !181280 (merged) has finished reviews and is merged.

  • Maintainer

    Note: taggings table has been replaced by specialized tables (p_ci_build_tags, ci_runner_taggings) from %17.9.

    Reference: !179384 (merged) and !179404 (merged)

  • Author Owner

    This script has been updated with $3700665 (comment 2284396203). We also found duplicate p_ci_build_tags entries that were somehow not prevented with the unique index. The script has been updated to remove duplicates before trying to remap values.

    • Hello,

      I'm facing issue running the rub script (in CE-v17.8.7) as well as the rake task after upgrading to version CE-v17.10.5. It's the same error in both the scenarios.

      Output of rake task in CE-v17.10.5
      [rocky@gitlab-server-22 ~]$ sudo gitlab-rake gitlab:db:deduplicate_tags
      I, [2025-04-30T07:06:43.450588 #8768]  INFO -- : Deduplicating 7 tags for ci database
      I, [2025-04-30T07:06:43.547766 #8768]  INFO -- : Updated tag_ids on a batch of 52 ci_pending_builds records
      I, [2025-04-30T07:06:44.572075 #8768]  INFO -- : Updated tag_id 659 on p_ci_build_tags records to 658
      rake aborted!
      ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "ci_build_tags_100_tag_id_build_id_partition_id_idx"
      DETAIL:  Key (tag_id, build_id, partition_id)=(406, 3700360, 100) already exists.
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:88:in `block (3 levels) in deduplicate_ci_taggings'
      /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:102:in `block (2 levels) in each_batch'
      /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:102:in `block in each_batch'
      /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:72:in `step'
      /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:72:in `each_batch'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:87:in `block (2 levels) in deduplicate_ci_taggings'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:86:in `each'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:86:in `block in deduplicate_ci_taggings'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:83:in `each'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:83:in `deduplicate_ci_taggings'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:62:in `deduplicate_ci_tags'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:36:in `execute'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db/deduplicate_tags.rake:10:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:25:in `load'
      /opt/gitlab/embedded/bin/bundle:25:in `<main>'
      
      Caused by:
      PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "ci_build_tags_100_tag_id_build_id_partition_id_idx"
      DETAIL:  Key (tag_id, build_id, partition_id)=(406, 3700360, 100) already exists.
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:88:in `block (3 levels) in deduplicate_ci_taggings'
      /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:102:in `block (2 levels) in each_batch'
      /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:102:in `block in each_batch'
      /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:72:in `step'
      /opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:72:in `each_batch'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:87:in `block (2 levels) in deduplicate_ci_taggings'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:86:in `each'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:86:in `block in deduplicate_ci_taggings'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:83:in `each'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:83:in `deduplicate_ci_taggings'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:62:in `deduplicate_ci_tags'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/deduplicate_ci_tags.rb:36:in `execute'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db/deduplicate_tags.rake:10:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:25:in `load'
      /opt/gitlab/embedded/bin/bundle:25:in `<main>'
      Tasks: TOP => gitlab:db:deduplicate_tags
      (See full trace by running task with --trace)

      Since this ruby task was failing, I proceeded with deleting the duplicate tags as mentioned below

      DELETE FROM tags WHERE id NOT IN ( SELECT MIN(id) FROM tags GROUP BY name ); DROP INDEX IF EXISTS index_tags_on_name; CREATE UNIQUE INDEX index_tags_on_name ON tags USING btree (name); REINDEX INDEX CONCURRENTLY index_tags_on_name_trigram; With this method, I'm not facing the 500 error code issue when creating the runners, but I lose the job logs of all those impacted pipelines where the tag (deleted) was used.

      PS: GitLab (CE) is selfhosted using omnibus installation on Rocky Linux 9

      Could someone please help me with this issue?

      Thanks in advance.

      Edited by Pedro Pombeiro
    • @cspencer1 I think you followed similar steps like @sreeram.s as you mentioned at omnibus-gitlab#8579 (comment 1947844123), could you please tell if you loose CI job trace, after performing the mentioned steps ?

      Edited by Rajesh Jena
    • Please register or sign in to reply
  • @stanhu After using your below delete query, we are loosing the CI job trace as mentioned by @sreeram.s in above comment :

    DELETE FROM tags WHERE id NOT IN ( SELECT MIN(id) FROM tags GROUP BY name ); REINDEX INDEX CONCURRENTLY index_tags_on_name; REINDEX INDEX CONCURRENTLY index_tags_on_name_trigram;

    Could you please suggest how to regain job trace?

  • Author Owner

    Do NOT run DELETE FROM tags WHERE id NOT IN ( SELECT MIN(id) FROM tags GROUP BY name ) unless you have properly de-duped the database with the script above.

    It sounds to me the CI builds table tags were not remapped properly, but I'm not sure why the job logs aren't showing.

    Please file a ticket with support if you are a customer.

    • @stanhu We are using gitlab CE, I think we don't have possibility of customer support ?

      While running the script ($3700665 (comment 2284396203)) to fix de-dup database, we face below error:

      Error Message [2025-05-05T06:26:28.201272 #3226] DEBUG -- : TRANSACTION (0.2ms) /*application:console,db_config_database:gitlabhq_production,db_config_name:ci,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ ROLLBACK /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params': PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "ci_build_tags_100_tag_id_build_id_partition_id_idx" (ActiveRecord::RecordNotUnique) DETAIL: Key (tag_id, build_id, partition_id)=(314, 3698608, 100) already exists.
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `block (2 levels) in exec_no_cache'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:767:in `block in exec_no_cache'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:766:in `exec_no_cache'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:745:in `execute_and_clear'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/marginalia-1.11.1/lib/marginalia.rb:91:in `execute_and_clear_with_marginalia'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:67:in `exec_delete'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract/database_statements.rb:175:in `update'
          ... 90 levels...

      /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params': ERROR: duplicate key value violates unique constraint "ci_build_tags_100_tag_id_build_id_partition_id_idx" (PG::UniqueViolation) DETAIL: Key (tag_id, build_id, partition_id)=(314, 3698608, 100) already exists.

          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `block (2 levels) in exec_no_cache'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:767:in `block in exec_no_cache'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:766:in `exec_no_cache'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:745:in `execute_and_clear'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/marginalia-1.11.1/lib/marginalia.rb:91:in `execute_and_clear_with_marginalia'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:67:in `exec_delete'
          from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract/database_statements.rb:175:in `update'
          ... 90 levels...

      Could you please suggest.

    • Author Owner

      That's not the right script. The script at the top (this snippet) is the updated one and covers that case (#523146 (closed)).

      Edited by Stan Hu
    • Thanks for prompt response @stanhu, Tried running below script

      Script Used
      ActiveRecord::Base.logger = Logger.new(STDOUT)
      
      good_tag_ids = ActiveRecord::Base.connection.select_all('SELECT name, MIN(id) AS id FROM tags GROUP BY name').rows.to_h
      bad_tag_map = ActiveRecord::Base.connection.select_all('SELECT id, name FROM tags WHERE id NOT IN (SELECT MIN(id) FROM tags GROUP BY name)').rows.to_h
      bad_tags = bad_tag_map.keys
      tag_remap = bad_tag_map.map { |id, name| [id, good_tag_ids[name]] }.to_h
      
      puts tag_remap
      
      class TempRunnerTagging < Ci::ApplicationRecord
        self.table_name = :ci_runner_taggings
        self.primary_key = :id
      end
      
      class TempTaggings < ActiveRecord::Base
        self.table_name = :taggings
        self.primary_key = :id
      end
      
      class TempTags < ActiveRecord::Base
        include EachBatch
      
        self.table_name = :tags
        self.primary_key = :id
      end
      
      class CiBuildTag < ActiveRecord::Base
        self.table_name = :p_ci_build_tags
        self.primary_key = :id
      end
      
      # Identify duplicate entries in p_ci_build_tags
      duplicates_query = <<-SQL
            SELECT tag_id, build_id, partition_id, COUNT(*), ARRAY_AGG(id) as duplicate_ids
            FROM public.p_ci_build_tags
            GROUP BY tag_id, build_id, partition_id
            HAVING COUNT(*) > 1
      SQL
      
      result = CiBuildTag.connection.exec_query(duplicates_query)
      
      puts "Found #{result.count} groups of duplicates."
      
      result.each do |group|
        tag_id = group['tag_id']
        build_id = group['build_id']
        partition_id = group['partition_id']
        ids = group['duplicate_ids'].tr('{}', '').split(',').map(&:to_i)
      
        # Keep the first record and delete the rest
        ids_to_delete = ids[1..-1]  # All except the first one
      
        puts "For tag_id=#{tag_id}, build_id=#{build_id}, partition_id=#{partition_id}:"
        puts "  Keeping id=#{ids[0]}, deleting ids=#{ids_to_delete.join(', ')}"
      
        # Delete the duplicate records
        CiBuildTag.where(id: ids_to_delete).delete_all
      end
      
      Ci::PendingBuild.all.each_batch do |batch|
        batch.each do |build|
          tag_ids = build.tag_ids
      
          next unless tag_ids & bad_tags
      
          build.tag_ids = tag_ids.map { |x| tag_remap.fetch(x, x) }
          build.save!
        end
      end
      
      tagging_models = [Ci::BuildTag, TempRunnerTagging, TempTaggings]
      tagging_models.each do |tagging_model|
        tagging_model.include EachBatch;
      
        bad_tags.each do |bad_tag_id|
          tagging_model.where(tag_id: bad_tag_id).each_batch(of: 10000) do |batch|
            batch.update_all(tag_id: tag_remap.fetch(bad_tag_id))
      
            puts "Updated tag_id #{bad_tag_id} on #{tagging_model.table_name} records to #{tag_remap.fetch(bad_tag_id)}"
          end
        end
      end
      
      if Gitlab.ee?
        ee_tagging_models = [Dast::ProfileTag]
        tagging_models.concat(ee_tagging_models)
      
        ee_tagging_models.each do |tagging_model|
          tagging_model.all.where(tag_id: bad_tags).each do |row|
            new_tag_id = tag_remap.fetch(row.tag_id)
            tagging_model.where(id: row.id).update_all(tag_id: new_tag_id)
          end
        end
      end
      
      if tagging_models.any? { |tagging_model| tagging_model.where(tag_id: bad_tag_map.keys).exists? }
        puts "Something went wrong, bad tags are still in use!"
      else
        TempTags.all.where(id: bad_tag_map.keys).each_batch(of: 10000) do |batch|
          batch.destroy_all
        end
      end
      
      TempTags.connection.exec_query("DROP INDEX IF EXISTS index_tags_on_name;")
      TempTags.connection.exec_query("CREATE UNIQUE INDEX index_tags_on_name ON tags USING btree (name);")
      TempTags.connection.exec_query("REINDEX TABLE p_ci_build_tags;")
      Error Message for Script
      Loading production environment (Rails 7.0.8.7)
      irb(main):001:0> load '/tmp/main.rb'
      D, [2025-05-05T07:02:24.705345 #6064] DEBUG -- :    (2.1ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT name, MIN(id) AS id FROM tags GROUP BY name
      D, [2025-05-05T07:02:24.708593 #6064] DEBUG -- :    (1.6ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT id, name FROM tags WHERE id NOT IN (SELECT MIN(id) FROM tags GROUP BY name)
      {659=>658, 660=>406, 661=>331, 662=>201, 700=>314, 701=>588, 706=>131}
      D, [2025-05-05T07:02:28.452616 #6064] DEBUG -- :   SQL (3741.1ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/       SELECT tag_id, build_id, partition_id, COUNT(*), ARRAY_AGG(id) as duplicate_ids
            FROM public.p_ci_build_tags
            GROUP BY tag_id, build_id, partition_id
            HAVING COUNT(*) > 1
      
      Found 0 groups of duplicates.
      D, [2025-05-05T07:02:28.515889 #6064] DEBUG -- :   Ci::PendingBuild Load (1.3ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:ci,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "ci_pending_builds"."id" FROM "ci_pending_builds" ORDER BY "ci_pending_builds"."id" ASC LIMIT 1
      D, [2025-05-05T07:02:28.526342 #6064] DEBUG -- :   Ci::PendingBuild Load (0.7ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:ci,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "ci_pending_builds"."id" FROM "ci_pending_builds" WHERE "ci_pending_builds"."id" >= 2008986 ORDER BY "ci_pending_builds"."id" ASC LIMIT 1 OFFSET 1000
      D, [2025-05-05T07:02:28.527919 #6064] DEBUG -- :   Ci::PendingBuild Load (0.6ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:ci,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "ci_pending_builds".* FROM "ci_pending_builds" WHERE "ci_pending_builds"."id" >= 2008986
      D, [2025-05-05T07:02:28.586340 #6064] DEBUG -- :   Feature::FlipperGate Pluck (1.3ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "feature_gates"."key", "feature_gates"."value" FROM "feature_gates" WHERE "feature_gates"."feature_key" = 'use_sql_functions_for_primary_key_lookups'
      D, [2025-05-05T07:02:28.590997 #6064] DEBUG -- :   Feature::FlipperGate Pluck (0.5ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "feature_gates"."key", "feature_gates"."value" FROM "feature_gates" WHERE "feature_gates"."feature_key" = 'feature_flag_state_logs'
      D, [2025-05-05T07:02:28.597885 #6064] DEBUG -- :   Namespace Load (2.7ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 4011 LIMIT 1
      D, [2025-05-05T07:02:28.657292 #6064] DEBUG -- :   Namespace Load (0.9ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 900 LIMIT 1
      D, [2025-05-05T07:02:28.662724 #6064] DEBUG -- :   Namespace Load (0.9ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 4011 LIMIT 1
      D, [2025-05-05T07:02:28.667435 #6064] DEBUG -- :   Namespace Load (0.9ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 4011 LIMIT 1
      D, [2025-05-05T07:02:28.671955 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 3367 LIMIT 1
      D, [2025-05-05T07:02:28.677340 #6064] DEBUG -- :   Namespace Load (1.6ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 841 LIMIT 1
      D, [2025-05-05T07:02:28.682179 #6064] DEBUG -- :   Namespace Load (0.9ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.686999 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.691794 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.696190 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.700703 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.705395 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.709688 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.714283 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.718609 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.723031 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.727492 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.731772 #6064] DEBUG -- :   Namespace Load (0.7ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.736344 #6064] DEBUG -- :   Namespace Load (0.9ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.741055 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.745438 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.749856 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 446 LIMIT 1
      D, [2025-05-05T07:02:28.754249 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 2684 LIMIT 1
      D, [2025-05-05T07:02:28.758582 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 2684 LIMIT 1
      D, [2025-05-05T07:02:28.763180 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 4645 LIMIT 1
      D, [2025-05-05T07:02:28.767411 #6064] DEBUG -- :   Namespace Load (0.7ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 3267 LIMIT 1
      D, [2025-05-05T07:02:28.771859 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 4645 LIMIT 1
      D, [2025-05-05T07:02:28.776331 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 4645 LIMIT 1
      D, [2025-05-05T07:02:28.780499 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 4645 LIMIT 1
      D, [2025-05-05T07:02:28.784812 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 4645 LIMIT 1
      D, [2025-05-05T07:02:28.789400 #6064] DEBUG -- :   Namespace Load (0.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:main,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "namespaces"."id", "namespaces"."name", "namespaces"."path", "namespaces"."owner_id", "namespaces"."created_at", "namespaces"."updated_at", "namespaces"."type", "namespaces"."description", "namespaces"."avatar", "namespaces"."membership_lock", "namespaces"."share_with_group_lock", "namespaces"."visibility_level", "namespaces"."request_access_enabled", "namespaces"."ldap_sync_status", "namespaces"."ldap_sync_error", "namespaces"."ldap_sync_last_update_at", "namespaces"."ldap_sync_last_successful_update_at", "namespaces"."ldap_sync_last_sync_at", "namespaces"."description_html", "namespaces"."lfs_enabled", "namespaces"."parent_id", "namespaces"."shared_runners_minutes_limit", "namespaces"."repository_size_limit", "namespaces"."require_two_factor_authentication", "namespaces"."two_factor_grace_period", "namespaces"."cached_markdown_version", "namespaces"."project_creation_level", "namespaces"."runners_token", "namespaces"."file_template_project_id", "namespaces"."saml_discovery_token", "namespaces"."runners_token_encrypted", "namespaces"."custom_project_templates_group_id", "namespaces"."auto_devops_enabled", "namespaces"."extra_shared_runners_minutes_limit", "namespaces"."last_ci_minutes_notification_at", "namespaces"."last_ci_minutes_usage_notification_level", "namespaces"."subgroup_creation_level", "namespaces"."max_pages_size", "namespaces"."max_artifacts_size", "namespaces"."mentions_disabled", "namespaces"."default_branch_protection", "namespaces"."max_personal_access_token_lifetime", "namespaces"."push_rule_id", "namespaces"."shared_runners_enabled", "namespaces"."allow_descendants_override_disabled_shared_runners", "namespaces"."traversal_ids", "namespaces"."organization_id" FROM "namespaces" WHERE "namespaces"."id" = 4645 LIMIT 1
      D, [2025-05-05T07:02:28.796536 #6064] DEBUG -- :   Ci::BuildTag Load (1.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:ci,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "p_ci_build_tags"."id" FROM "p_ci_build_tags" WHERE "p_ci_build_tags"."tag_id" = 659 ORDER BY "p_ci_build_tags"."id" ASC LIMIT 1
      D, [2025-05-05T07:02:29.160300 #6064] DEBUG -- :   Ci::BuildTag Load (362.5ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:ci,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "p_ci_build_tags"."id" FROM "p_ci_build_tags" WHERE "p_ci_build_tags"."tag_id" = 660 ORDER BY "p_ci_build_tags"."id" ASC LIMIT 1
      D, [2025-05-05T07:02:29.168229 #6064] DEBUG -- :   Ci::BuildTag Load (2.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:ci,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ SELECT "p_ci_build_tags"."id" FROM "p_ci_build_tags" WHERE "p_ci_build_tags"."tag_id" = 660 AND "p_ci_build_tags"."id" >= 1895861 ORDER BY "p_ci_build_tags"."id" ASC LIMIT 1 OFFSET 10000
      D, [2025-05-05T07:02:29.176193 #6064] DEBUG -- :   Ci::BuildTag Update All (6.8ms)  /*application:console,db_config_database:gitlabhq_production,db_config_name:ci,console_hostname:euw1-gitlab-server-22,console_username:rocky*/ UPDATE "p_ci_build_tags" SET "tag_id" = 406 WHERE "p_ci_build_tags"."tag_id" = 660 AND "p_ci_build_tags"."id" >= 1895861
      /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params': PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "ci_build_tags_100_tag_id_build_id_partition_id_idx" (ActiveRecord::RecordNotUnique)
      DETAIL:  Key (tag_id, build_id, partition_id)=(406, 3700360, 100) already exists.
      
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `block (2 levels) in exec_no_cache'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:767:in `block in exec_no_cache'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:766:in `exec_no_cache'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:745:in `execute_and_clear'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/marginalia-1.11.1/lib/marginalia.rb:91:in `execute_and_clear_with_marginalia'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:67:in `exec_delete'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract/database_statements.rb:175:in `update'
              ... 40 levels...
      /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `exec_params': ERROR:  duplicate key value violates unique constraint "ci_build_tags_100_tag_id_build_id_partition_id_idx" (PG::UniqueViolation)
      DETAIL:  Key (tag_id, build_id, partition_id)=(406, 3700360, 100) already exists.
      
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:768:in `block (2 levels) in exec_no_cache'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:767:in `block in exec_no_cache'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activesupport-7.0.8.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:766:in `exec_no_cache'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql_adapter.rb:745:in `execute_and_clear'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/marginalia-1.11.1/lib/marginalia.rb:91:in `execute_and_clear_with_marginalia'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:67:in `exec_delete'
              from /opt/gitlab/embedded/lib/ruby/gems/3.2.0/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract/database_statements.rb:175:in `update'
              ... 40 levels...
      irb(main):002:0>
      Duplicate TAG Info
      SELECT * FROM tags WHERE name IN ('module_test','ipk-build','scl-win','chx','k3','owmc-shell','${RUNNER_TAG}') ORDER BY name;`
      
      | id| name| taggings_count|
      | ------ | ------ |------|
      |659 | ${RUNNER_TAG} |              0|
      |658 | ${RUNNER_TAG} |              0|
      |314 | chx           |              0|
      |700 | chx           |              0|
      |661 | ipk-build     |              0|
      |331 | ipk-build     |              0|
      |588 | k3            |              0|
      |701 | k3            |              0|
      |406 | module_test   |              0|
      |660 | module_test   |              0|
      |706 | owmc-shell    |              0|
      |131 | owmc-shell    |              0|
      |201 | scl-win       |              0|
      |662 | scl-win       |              0|
      Edited by Pedro Pombeiro
    • @stanhu Could you please suggest the needful for above error message?

    • Please register or sign in to reply
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment