As seen in https://gitlab.com/gitlab-org/gitlab/-/jobs/6097005150, db:migrate:multi-version-upgrade-2 failed with ActiveRecord::StatementInvalid: PG::DuplicateColumn: ERROR: column "encrypted_token" of relation "chat_names" already exists
As such confirmed that approach works as expected.
Nailia Iskhakovamarked the checklist item Validate that data selected approach work - create DB dump for known problematic upgrade path and verify that there is a migration error as completed
marked the checklist item Validate that data selected approach work - create DB dump for known problematic upgrade path and verify that there is a migration error as completed
Explore if there is a way to seed environment with all possible fixtures
As part of testing options in this area I configured local GDK setup. Also using Docker image for 16.3 for additional validation in non-dev env (note that Docker image has tweaks from db:migrate:multi-version-upgrade-1 code.
As long as create or build with &:save are used - ActiveRecord validations are used and unhealthy record won't be written to DB
Notes so far from few crude tests:
When using GitLab Rails console
I created this crude code for cycling through all factories and creating them:
factories=FactoryBot.factoriesresults=factories.each_with_indexdo|factory,i|# Create a new instance for each factorybeginFactoryBot.create(factory.name)rescueException=>eputs"#{factory.name} caught exception #{e}!"elseputs"success #{factory.name}"ensureputs"Factory ##{i}"endend
It worked both in GDK (test with first 5 factories) and Docker image (run full command).
Some of the factories failed with PG::UniqueViolation: ERROR for example:
But the majority finished. When checking ActiveRecords - records are in place.
When running similar script to above as part of Seeder rake
moduleGitlabmoduleDataSeederclass<<self# Seed test data using GitLab Data Seeder# @param [String] seed_file the full-path of the seed file to load (.yml, .rb)defseed(owner,seed_file)....defseed_allFactoryBot.factoriesdo|factory|# Create a new instance for each factoryGitlab::AppLogger.debugfactory.nameFactoryBot.create(factory.name)endendend
The create part silently fails due to missing variables. For example for the first factory
FactoryBot.create(factory.name)NameError:uninitializedconstantGitlab::Pagination::Keyset::Orderfrom/GitLab/gitlab-development-kit/gitlab/config/initializers/active_record_keyset_pagination.rb:22:in`block in reverse_sql_order'
To be clarified.
Failures in rake. I'm not sure yet what is difference between Rails console and running above in Rake. It looks like something is missing.
In general though it might look like that this can be an option to seed everything and then also develop a custom Seeder config with designed data for granular control?
Maintenance concern - if Factory name changes -> our customer Seeder rb file config will start failing/not creating fixtures that uses old name. How to keep it up to date/catch this? (when we move data seeding to a separate project)
The odds of this are very low. Even if this were to happen, a global refactor across the entire codebase would be necessary. Since this seed file would reside in the gitlab-org/gitlab project, this should be picked up.
In my time of Rails development, though - I've not seen any change of factory name. It would require Deletion, or Change of name of a Model of the Rails base. (For instance, if we were to rename Merge Request to Pull Request. (create(:merge_request) => create(:pull_request))
Keep in mind that the seed file must be of this format, as documented here
Yes, I tried adding something like this originally for testing
# test_data.rbclassDataSeederdefseedFactoryBot.factories.first(10)do|factory|# Create a new instance for each factoryGitlab::AppLogger.debugfactory.nameFactoryBot.create(factory.name)endendend
It didn't seed and failed with the same uninitialized constant Gitlab::Pagination::Keyset::Order, I'm probably missing something fundamental here as I'm still reading up about how factories work
I pulled the latest master in my GDK today with gitlab-org/gitlab!137582 (merged) is merged. I'm seeing the same NameError: uninitialized constant errors.
The odds of this are very low. Even if this were to happen, a global refactor across the entire codebase would be necessary.
Got it, thanks for the additional context Good to know that concern should be less as long as seed file lives in the project. I was thinking that perhaps at some point we would want to maintain few different Seeder configs for upgrade testing specifically that will be stored outside of the main repo. Will give it a further think after I get up to speed with this area first.
In the https://gitlab.com/gitlab-org/gitlab/-/jobs/6163413795 we can see that there are 351 successfully created fixtures (search ....success) and 161 failed fixtures (caught exception) if running against 16.3.0 Docker image.
When running same seeder file against local GDK( 16.9.0-pre 3a666cdff3f) I've seen 150 out of 963 fixtures failed with errors.
Fixture errors
1. No such file or directory @ rb_sysopen
Most common error - No such file or directory @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-rails/tmp/tests/gitlab-test.bundle!
Running scripts/gitaly-test-build which should build these bundles in GDK failed:
scripts/gitaly-test-build/Users/niskhakova/GitLab/gitlab-development-kit/gitlab/spec/support/helpers/gitaly_setup.rb:92:in`system': No such file or directory - make (Errno::ENOENT) from /Users/niskhakova/GitLab/gitlab-development-kit/gitlab/spec/support/helpers/gitaly_setup.rb:92:in `run_command' from /Users/niskhakova/GitLab/gitlab-development-kit/gitlab/spec/support/helpers/gitaly_setup.rb:96:in `build_gitaly'fromscripts/gitaly-test-build:21:in`run' from scripts/gitaly-test-build:42:in `<main>'
Also side note - I believe the error with above explains why MR seeded with bulk_data has missing branches:
In gitlab-test.bundle there is 0b4bc9a49b562e85de7cc9e834518ea6828729b9 refs/heads/feature ref - which should have been used in above MR.
2. PG::UniqueViolation: ERROR
Second common error. Most popular exception is for index_ci_namespace_mirrors_on_namespace_id - 41/45 cases.
Debugging:
To be done
3. Postgres factories related errors
Factory #206reindex_action caught exception PG::ObjectNotInPrerequisiteState: ERROR: cannot insert into view "postgres_indexes"DETAIL: Views that do not select from a single table or view are not automatically updatable.HINT: To enable inserting into the view, provide an INSTEAD OF INSERT trigger or an unconditional ON INSERT DO INSTEAD rule.!
I assume it's expected to fail against live envs so can be ignored?
@niskhakova As you've already found, gitaly_setup.rb will attempt to build gitaly from source unless the environment variable CI is set. That is why it is trying to run make. So for this to succeed you'll either need to add all the build tools to the docker image, or build gitaly separately like CI does and set the environment appropriately.
@proglottis many thanks for the clarification on this as well as the tip about bundle exec rspec spec/lib/backup/gitaly_backup_spec.rb that would initiate TestEnv.init in GDK Confirmed that when running this command, the bundle files were created!
The "normal" GDK way where spec_helper.rb calls TestEnv.init which then calls TestEnv.setup_forked_repo/TestEnv.setup_factory_repo that actually do the cloning/ref-setting/bundle-creating
@ddavison could you please expand on this? In this thread we're exploring how to seed all fixtures, and as found in #2354 (comment 1772988958) - there are some errors that doesn't allow to seed the environment. Resolving the errors allows fixtures to be seeded.
The data is the crucial point of catching migrations, we need a way to seed all possible data and looks like Data Seeder + fixtures is the solution here.
In general, I don't think there is concern about multi-version upgrades without building an environment because we're working on preparing PG dump - this is the only place where environment is being used, however the test itself doesn't need an env (since we import the dump). It will be pulling existing PG dump for specific upgrade stop and not rely on any environment.
27 UniqueViolation errors, out of which 23 are index_ci_namespace_mirrors_on_namespace_id.
Reproduced this in my freshly reinstalled GDK. For example rails console:
gdkrailsconsole...FactoryBot.create(:board_group_recent_visit)# factory is created=>#<BoardGroupRecentVisit:0x000000028223f508id: 1,created_at: Wed,28Feb202417:25:45.371008000UTC+00:00,updated_at: Wed,28Feb202417:25:45.371008000UTC+00:00,user_id: 70,board_id: 1,group_id: 96>FactoryBot.create(:board_group_recent_visit)# on second try failed with the same `index_ci_namespace_mirrors_on_namespace_id`ActiveRecord::RecordNotUnique:PG::UniqueViolation:ERROR:duplicatekeyvalueviolatesuniqueconstraint"index_ci_namespace_mirrors_on_namespace_id"DETAIL:Key(namespace_id)=(100)alreadyexists.FactoryBot.create(:board_group_recent_visit)# didn't fail on third time=>#<BoardGroupRecentVisit:0x0000000128e7ec48id: 2,created_at: Wed,28Feb202417:26:11.482766000UTC+00:00,updated_at: Wed,28Feb202417:26:11.482766000UTC+00:00,user_id: 73,board_id: 2,group_id: 102>
Not sure why rerunning the same command fixes this error.
Slightly changing seed_all_fixtures to retry on ActiveRecord::RecordNotUnique - resolved the issue with index_ci_namespace_mirrors_on_namespace_id:
defseed_all_fixturesFactoryBot.factories.each_with_indexdo|factory,i|retries||=0# Create a new instance for each factoryFactoryBot.create(factory.name)rescueActiveRecord::RecordNotUnique=>eputs"#{factory.name} caught exception #{e}! Attempt##{retries}"retryif(retries+=1)<3rescueException=>e# rubocop:disable Lint/RescueException -- catching all possible exceptions# We rescue exception here to make sure seeding proceeds unrelated to various unique exceptions from Factoriesputs"#{factory.name} caught exception #{e} of class #{e.class}!"elseputs"Successfully created Factory #{factory.name}"ensureputs"Factory ##{i+1}"endend
@alexives reaching out with some updates on seeding for upgrade testing in multi-version test and for assistance from Database team for your expertise.
Context
Epic goal: Add CI job which would emulate multi-version* upgrades by running bundle exec rake gitlab:db:configure against DB dump from latest known GitLab version stop with test data.
Prepares DB dump using Docker image from latest upgrade stop and seeding it using Data Seeder - code
Imports DB dump from step 1 and runs bundle exec rake gitlab:db:configure to trigger migrations against data from step 1 - code
Context A - In the current issue I'm looking at options how we can expand data variety in DB dump for step 1 - ideally seeding all possible tables. I updated existing Data Seeder config to iterate throught all existing Fixtures and seed them in Seed all fixtures with Seeder (gitlab-org/gitlab!144631 - merged). In db:migrate:multi-version-upgrade-1 job https://gitlab.com/gitlab-org/gitlab/-/jobs/6181804106 - you can see all fixtrues are being seeded. There are few errors in some fixtures but I thought it'd be better to ask for your initial overview of the approach before diving more into its debugging (some issues were already fixed).
Context B - I validated that this approach should work in general as seen in #2354 (comment 1758876296). However I'm having troubles to develop a test where we could see a failed migration (a test for multi-version upgrade test ) - this would be helpful for testing seeding changes done in this issue. I'm assuming the easiest way could be to break some migration intentionally, but I'm not sure how to do that.
Request for help
Could Database team please help with below:
Review if such approach for seeding all fixtures would be a good iteration on making data as wide as it can be? (context A)
Suggest how to break some table migration intentionally so that we could test the updated test and data seeding approach? (context B)
If this suggested approach is good - should we raise an issue about enforcing fixtures for new tables as suggested in #1919 (comment 1627260665)?
Happy to sync with your or the team if that helps!
Small update on Context B about scenario for breaking migrations intentionally for testing the test - I created gitlab-org/gitlab!144992 (closed) which adds a non-null constraint to an existing column closed_by_id in Issues model to break migrations - looks like confirmed that existing seeding option of using seed_all - works https://gitlab.com/gitlab-org/gitlab/-/jobs/6193280842#L22220, though not sure if the test as representative. If there is a way to break migration "smartly" so that it would only break multi-version migration test and not other specs - that would be great.
Context B - I validated that this approach should work in general as seen in #2354 (comment 1758876296). However I'm having troubles to develop a test where we could see a failed migration (a test for multi-version upgrade test ) - this would be helpful for testing seeding changes done in this issue. I'm assuming the easiest way could be to break some migration intentionally, but I'm not sure how to do that.
Small update on Context B about scenario for breaking migrations intentionally for testing the test - I created gitlab-org/gitlab!144992 (closed) which adds a non-null constraint to an existing column closed_by_id in Issues model to break migrations - looks like confirmed that existing seeding option of using seed_all - works https://gitlab.com/gitlab-org/gitlab/-/jobs/6193280842#L22220, though not sure if the test as representative. If there is a way to break migration "smartly" so that it would only break multi-version migration test and not other specs - that would be great.
From fed6dd8a58d75a0e053a4972765b4fc08c5814a3 Mon Sep 17 00:00:00 2001From: Jon Jenkins <jjenkins@gitlab.com>Date: Tue, 18 Apr 2023 11:14:42 -0500Subject: [PATCH] Designed to fail - do not merge--- db/migrate/30000000000000_designed_to_fail.rb | 31 +++++++++++++++++++ db/schema_migrations/30000000000000 | 1 + 2 files changed, 32 insertions(+) create mode 100644 db/migrate/30000000000000_designed_to_fail.rb create mode 100644 db/schema_migrations/30000000000000diff --git a/db/migrate/30000000000000_designed_to_fail.rb b/db/migrate/30000000000000_designed_to_fail.rbnew file mode 100644index 00000000000000..cdf3afeeee69c5--- /dev/null+++ b/db/migrate/30000000000000_designed_to_fail.rb@@ -0,0 +1,31 @@+# frozen_string_literal: true++# See https://docs.gitlab.com/ee/development/migration_style_guide.html+# for more information on how to write migrations for GitLab.++class DesignedToFail < Gitlab::Database::Migration[2.1]+ # When using the methods "add_concurrent_index" or "remove_concurrent_index"+ # you must disable the use of transactions+ # as these methods can not run in an existing transaction.+ # When using "add_concurrent_index" or "remove_concurrent_index" methods make sure+ # that either of them is the _only_ method called in the migration,+ # any other changes should go in a separate migration.+ # This ensures that upon failure _only_ the index creation or removing fails+ # and can be retried or reverted easily.+ #+ # To disable transactions uncomment the following line and remove these+ # comments:+ # disable_ddl_transaction!+ #+ # Configure the `gitlab_schema` to perform data manipulation (DML).+ # Visit: https://docs.gitlab.com/ee/development/database/migrations_for_multiple_databases.html+ # restrict_gitlab_migration gitlab_schema: :gitlab_main++ def up+ raise 'this is supposed to fail because we\'re testing a DB testing pipeline failure'+ end++ def down++ end+enddiff --git a/db/schema_migrations/30000000000000 b/db/schema_migrations/30000000000000new file mode 100644index 00000000000000..b46bdb79ef7ab7--- /dev/null+++ b/db/schema_migrations/30000000000000@@ -0,0 +1 @@+70a752b17fff409330858ed34d010afed59319398d5d6bc113ab0df59cced123\ No newline at end of file-- GitLab
During the job execution, we can apply the patch:
git am --3way < add-intentionally-failing-migration.patch
It looks like all other migration testing jobs failed similarly as well - https://gitlab.com/gitlab-org/gitlab/-/jobs/6245272627#L795. In this way it replicates a bit issue in #2354 (comment 1778884274). Is there a way to break migration so that db:check-migrations and other non multi-version upgrade tests are passing, but db:migrate:multi-version-upgrade fails? Currently latest upgrade stop is 16.7.6 - this is the version from which PG dump file is created. Is it possible to introduce some known breaking change to current master that would break migration from 16.7.6 but won't break it for usual migrations? Hope that context helps Happy to discuss this further
@l.rosa thanks for example The goal is to ensure that multi-version upgrade job is able to catch migration errors that default DB CI jobs are not able. As multi-version upgrade runs migration against DB imported from last known upgrade stop. It was slightly verified with #2354 (comment 1758876296) but I wanted to ask if there is a way to intentionally cause a break in some existing table that would affect upgrade 16.7.6 -> master.
@l.rosa could you please clarify if it's a viable option after discussion with the team? I read up on DB Group weekly notes from last week and seems that there are concerns/problems.
To add to the list, another problem is that GitLab.com doesn't follow self-managed release cycle and it's being upgraded daily multiple times. So snapshot from for example %16.7 last date != 16.7.0 release. Additionally it won't cover patches - like 16.7.2.
So I believe there is no option of using snapshot in isolation even if it's possible, we will need to use custom seeded DBs in all cases. And hopefully with time and analysis for test gaps, we can expand it further.
Database team mention an option to use fixtures for seeding. Can be triggered with MASS_INSERT=1 rails db:seed_fu. It should be more robust solution with less errors since fixtures are handcrafted to work.
db/fixtures code from latest master can't be used for seeding older GitLab images - it should always be in sync with current GitLab version. So for example if running against 16.7.2 - it should use db/fixtures from 16.7.2 ref - otherwise there are compatibility errors like NoMethodError: undefined method skipping_transaction_check' for Gitlab::ExclusiveLease:Class
there are known seeding errors like ActiveRecord::RecordInvalid: Validation failed: Email has already been taken, Username has already been taken for db/fixtures/development/01_admin.rb which was coincidentally fixed yesterday - gitlab-org/gitlab@d1f714eb.
however all fixes that will be done can't be added retroactively => in example above the db/fixtures/development/01_admin.rb was fixed in 16.10 and we need to run seeding in 16.7 where it's still broken.
Based on above we need to run each seed file in separation and skip to next one, otherwise if running just db:seed_fu without FILTER => seeding will just fail on 01_admin.rb and won't continue.
To be clarified:
What are criteria for adding db/fixture files? Is it ad-hoc? If yes, we can't enforce that teams should create such file for each table?
Is there a way to continue seeding with db:seed_fu even if one of the seed file fixtures fails?
Listing all errors I've seen locally in Docker/GDK when trying different seeding commands and configs.
Click to expand
== Seed from db/fixtures/development/01_admin.rb/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/zeitwerk-2.6.7/lib/zeitwerk/kernel.rb:38: warning: ⛔️ WARNING: Sidekiq testing API enabled, but this is not the test environment. Your jobs will not go to Redis.rake aborted!ActiveRecord::RecordInvalid: Validation failed: Email has already been taken, Username has already been taken/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'(eval):4:in `block (3 levels) in run_file'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:94:in `block (3 levels) in quiet'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:122:in `without_new_note_notifications'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:93:in `block (2 levels) in quiet'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:132:in `without_statement_timeout'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:92:in `block in quiet'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:148:in `without_database_logging'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:91:in `quiet'(eval):3:in `block (2 levels) in run_file'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:359:in `block in transaction'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:358:in `transaction'/opt/gitlab/embedded/bin/bundle:25:in `load'/opt/gitlab/embedded/bin/bundle:25:in `<main>'Tasks: TOP => db:seed_fu(See full trace by running task with --trace)== Seed from db/fixtures/development/07_milestones.rbrake aborted!NoMethodError: undefined method `id' for nil:NilClass author_id: current_user.id ^^^/opt/gitlab/embedded/service/gitlab-rails/app/services/event_create_service.rb:263:in `create_event'/opt/gitlab/embedded/service/gitlab-rails/app/services/event_create_service.rb:193:in `create_record_event'/opt/gitlab/embedded/service/gitlab-rails/app/services/event_create_service.rb:73:in `open_milestone'/opt/gitlab/embedded/service/gitlab-rails/app/services/milestones/create_service.rb:11:in `execute'(eval):12:in `block (5 levels) in run_file'(eval):5:in `times'(eval):5:in `block (4 levels) in run_file'(eval):4:in `block (3 levels) in run_file'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:94:in `block (3 levels) in quiet'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:122:in `without_new_note_notifications'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:93:in `block (2 levels) in quiet'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:132:in `without_statement_timeout'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:92:in `block in quiet'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:148:in `without_database_logging'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:91:in `quiet'(eval):3:in `block (2 levels) in run_file'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:359:in `block in transaction'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:358:in `transaction'/opt/gitlab/embedded/bin/bundle:25:in `load'/opt/gitlab/embedded/bin/bundle:25:in `<main>'Tasks: TOP => db:seed_fu(See full trace by running task with --trace)== Seed from /opt/gitlab/embedded/service/gitlab-rails/db/fixtures/development/03_project.rbrake aborted!NoMethodError: undefined method `skipping_transaction_check' for Gitlab::ExclusiveLease:Class(eval):187:in `create_real_project!'(eval):114:in `block in create_real_projects!'(eval):113:in `each'(eval):113:in `each_with_index'(eval):113:in `create_real_projects!'(eval):61:in `block in seed!'(eval):60:in `seed!'(eval):237:in `block (3 levels) in run_file'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:94:in `block (3 levels) in quiet'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:122:in `without_new_note_notifications'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:93:in `block (2 levels) in quiet'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:132:in `without_statement_timeout'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:92:in `block in quiet'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:148:in `without_database_logging'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/seeder.rb:91:in `quiet'(eval):235:in `block (2 levels) in run_file'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:359:in `block in transaction'/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:358:in `transaction'/opt/gitlab/embedded/bin/bundle:25:in `load'/opt/gitlab/embedded/bin/bundle:25:in `<main>'Tasks: TOP => db:seed_fu(See full trace by running task with --trace)== Seed from /Users/niskhakova/GitLab/gitlab-development-kit/gitlab/db/fixtures/development/03_project.rbrake aborted!NoMethodError: undefined method `_run_after_commit_queue' for nil:NilClass(eval):196:in `block (2 levels) in create_real_project!'/Users/niskhakova/GitLab/gitlab-development-kit/gitlab/config/initializers/forbid_sidekiq_in_transactions.rb:10:in `skipping_transaction_check'(eval):188:in `block in create_real_project!'/Users/niskhakova/GitLab/gitlab-development-kit/gitlab/lib/gitlab/exclusive_lease.rb:104:in `skipping_transaction_check'(eval):187:in `create_real_project!'(eval):114:in `block in create_real_projects!'(eval):113:in `each'(eval):113:in `each_with_index'(eval):113:in `create_real_projects!'(eval):61:in `block in seed!'/Users/niskhakova/GitLab/gitlab-development-kit/gitlab/vendor/gems/sidekiq-7.1.6/lib/sidekiq/testing.rb:17:in `__set_test_mode'/Users/niskhakova/GitLab/gitlab-development-kit/gitlab/vendor/gems/sidekiq-7.1.6/lib/sidekiq/testing.rb:47:in `inline!'(eval):60:in `seed!'(eval):237:in `block (3 levels) in run_file'
I'm not sure if it's fine to drop/recreate DB for seeding, it seems like it might skew PG dump building and also affecting Seeder fixture seeding For now will iterate on using both Seeder and fixtures without DB drop.
For SeedFu the decided approach is to run seed_fu rake for each found fixture one by one, because when running just "db:seed_fu" without FILTER - whole seeding will stop if one of the fixtures failed to be seeded and there is no option to ignore failures with SeedFu. The failures for seeding are expected and resolved in gitlab-org/gitlab!147090 (merged) and also earlier in gitlab-org/gitlab@8f6166db and gitlab-org/gitlab@d1f714eb - and they can't be added retroactively to 16.7 => so we need to seed each fixture in isolation to bypass this.
Created Investigate db/fixture seeding errors with Seed... (#2513 - closed) to track investigation of NameError: uninitialized constant Gitlab::Seeder::Projects and NoMethodError: undefined method '_run_after_commit_queue' for nil:NilClass errors when seeding project fixtures.
Quick iteration in gitlab-org/gitlab!147090 (merged) didn't resolve the issue, it looks like more in-depth knowledge is needed for this.
there are known seeding errors like ActiveRecord::RecordInvalid: Validation failed: Email has already been taken, Username has already been taken for db/fixtures/development/01_admin.rb
I am surprised this was the case, as fixtures are executed/seeded on new GDK install, and also covered by run-dev-fixtures* jobs.
I was also a bit confused why it wasn't caught earlier, and with regards to run-dev-fixtures* - I took a further look into it at #2354 (comment 1804126206) and I believe it's because the job drops the DB bundle exec rake db:drop db:create db:schema:load db:migrate gitlab:db:lock_writes - so admin seed works fine. In GDK I believe something similar is happening, though the fix for 01_admin.rb - gitlab-org/gitlab@d1f714eb - originated from another GDK issue.
92% of Data Tables (Base Tables) are seeded with above approach With factory rubocop in place and future discussion on enforcing fixtures for tables - the number should improve.
It's worth noting however that further iteration on data seeding will be done once we review known migration errors that customers faced and if some of those scenarios can be reproduced with PG Dump Generator.