Expand data seeding configuration for multi-version upgrades testing

marked this issue as blocked by #1919 (closed)

changed milestone to %Next 1-3 releases

added Quality teamSelf-Managed Platform typemaintenance labels

assigned to @niskhakova

added to epic &19 (closed)

mentioned in issue #1919 (closed)

mentioned in epic &19 (closed)

mentioned in merge request gitlab-org/gitlab!137582 (merged)

changed the description

changed milestone to %16.9

mentioned in merge request gitlab-org/gitlab!143647 (closed)

Catching up with Seeder based on MR Add multi-version upgrade test (gitlab-org/gitlab!137582 - merged)

Before validating, I'm focusing on getting hands on experience with Seeder and up to speed with changes in Add multi-version upgrade test (gitlab-org/gitlab!137582 - merged)

I used db:migrate:multi-version-upgrade-1 guidance to build 16.3.x container locally and run Seeder.

On first run the Seeding failed with:

root@192:/opt/gitlab/embedded/service/gitlab-rails# gitlab-rake "ee:gitlab:seed:data_seeder[bulk_data.rb]"
Seeding data for Administrator
.rake aborted!
ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_ci_namespace_mirrors_on_namespace_id"
DETAIL:  Key (namespace_id)=(2) already exists.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:133:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:204:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:122:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:61:in `block (2 levels) in <class:ConnectionProxy>'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:133:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:204:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:122:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:92:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:133:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:204:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:122:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:390:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:389:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/cross_database_modification.rb:83:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/spec/factories/namespaces.rb:19:in `block (3 levels) in <top (required)>'
<internal:kernel>:90:in `tap'

Relevant container logs:

==> /var/log/gitlab/gitlab-rails/application_json.log <==
{"severity":"INFO","time":"2024-02-02T17:12:38.782Z","message":"Group \"bulk_data\" was created"}

==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2024-02-02T17:12:39.269Z","retry":25,"queue":"default","version":0,"args":[],"class":"Namespaces::ProcessSyncEventsWorker","jid":"846a2a7fa1ecb3d1a8b18431","created_at":"2024-02-02T17:12:39.208Z","meta.feature_category":"cell","correlation_id":"711af00011819c0d03ed581266739105","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:99b01e5f7dc2a9b47ce31029252a21c1a7e02ef97b2a00a70d65582147f34ddb","size_limiter":"validated","enqueued_at":"2024-02-02T17:12:39.265Z","job_size_bytes":2,"pid":837,"message":"Namespaces::ProcessSyncEventsWorker JID-846a2a7fa1ecb3d1a8b18431: start","job_status":"start","scheduling_latency_s":0.004002}

==> /var/log/gitlab/postgresql/current <==
2024-02-02_17:12:39.38328 ERROR:  duplicate key value violates unique constraint "index_ci_namespace_mirrors_on_namespace_id"

2024-02-02_17:12:39.38437 DETAIL:  Key (namespace_id)=(2) already exists.
2024-02-02_17:12:39.38457 STATEMENT:  /*application:web,db_config_name:ci*/ INSERT INTO "ci_namespace_mirrors" ("namespace_id", "traversal_ids") VALUES (2, '{2}') RETURNING "id"

After rerunning the same gitlab-rake "ee:gitlab:seed:data_seeder[bulk_data.rb]" second time against the same container - it passed

Not sure what might have caused the error first time.

mentioned in merge request gitlab-org/gitlab!143821 (closed)

Validate that data selected approach work

Verified that general approach for testing is working at gitlab-org/gitlab!143821 (closed), where testing upgrade path 16.5.6 -> 16.7.2 with a known error - gitlab-org/gitlab#438215 (closed)

As seen in https://gitlab.com/gitlab-org/gitlab/-/jobs/6097005150, db:migrate:multi-version-upgrade-2 failed with ActiveRecord::StatementInvalid: PG::DuplicateColumn: ERROR: column "encrypted_token" of relation "chat_names" already exists

As such confirmed that approach works as expected.

The next steps are to expand data shape to be used for Seeding to catch migrations. Once adjusted Data Seeder config, would rerun validation test that depends on specific data. Some of the examples that can be used for that - gitlab-org/gitlab#437309 (closed), gitlab-org/gitlab#428107 (comment 1607561859) or gitlab-org/gitlab#409331 (closed)

marked the checklist item Validate that data selected approach work - create DB dump for known problematic upgrade path and verify that there is a migration error as completed

Explore if there is a way to seed environment with all possible fixtures

As part of testing options in this area I configured local GDK setup. Also using Docker image for 16.3 for additional validation in non-dev env (note that Docker image has tweaks from db:migrate:multi-version-upgrade-1 code.

Prior to this I haven't had experience with FactoryBot, so I'm reading up on https://github.com/thoughtbot/factory_bot/blob/main/GETTING_STARTED.md.

Notes so far from my very basic understanding:

Each Class has a Factory
As long as create or build with &:save are used - ActiveRecord validations are used and unhealthy record won't be written to DB

Notes so far from few crude tests:

When using GitLab Rails console

I created this crude code for cycling through all factories and creating them:

factories = FactoryBot.factories

results = factories.each_with_index do |factory, i|
  # Create a new instance for each factory
  begin
    FactoryBot.create(factory.name)
  rescue Exception => e
      puts "#{factory.name} caught exception #{e}!"
  else
    puts "success #{factory.name}"
  ensure
      puts "Factory ##{i}"
  end
end

It worked both in GDK (test with first 5 factories) and Docker image (run full command). Some of the factories failed with PG::UniqueViolation: ERROR for example:

vulnerability_read caught exception PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_ci_namespace_mirrors_on_namespace_id"
DETAIL:  Key (namespace_id)=(1599) already exists.

But the majority finished. When checking ActiveRecords - records are in place.

When running similar script to above as part of Seeder rake

module Gitlab
  module DataSeeder
    class << self
      # Seed test data using GitLab Data Seeder
      # @param [String] seed_file the full-path of the seed file to load (.yml, .rb)
      def seed(owner, seed_file)
....

      def seed_all
        FactoryBot.factories do |factory|
          # Create a new instance for each factory
          Gitlab::AppLogger.debug factory.name
          FactoryBot.create(factory.name)
        end
      end
    end

The create part silently fails due to missing variables. For example for the first factory

FactoryBot.create(factory.name)
NameError: uninitialized constant Gitlab::Pagination::Keyset::Order
from /GitLab/gitlab-development-kit/gitlab/config/initializers/active_record_keyset_pagination.rb:22:in `block in reverse_sql_order'

To be clarified.

Failures in rake. I'm not sure yet what is difference between Rails console and running above in Rake. It looks like something is missing.

In general though it might look like that this can be an option to seed everything and then also develop a custom Seeder config with designed data for granular control?

Maintenance concern - if Factory name changes -> our customer Seeder rb file config will start failing/not creating fixtures that uses old name. How to keep it up to date/catch this? (when we move data seeding to a separate project)

A few notes →

Keep in mind that the seed file must be of this format, as documented here

class DataSeeder
  def seed
    ...
  end
end

Add multi-version upgrade test (gitlab-org/gitlab!137582 - merged) will introduce a Logger for the seeder to use, which will be usable by @logger.debug 'msg' in the seed file.

Failures in rake. I'm not sure yet what is difference between Rails console and running above in Rake. It looks like something is missing.

This might be fixed with gitlab-org/gitlab!137582 (merged), in particular, the change to data_seeder.rb to include rspec, and a few other pertinent things.

if Factory name changes

The odds of this are very low. Even if this were to happen, a global refactor across the entire codebase would be necessary. Since this seed file would reside in the gitlab-org/gitlab project, this should be picked up.

In my time of Rails development, though - I've not seen any change of factory name. It would require Deletion, or Change of name of a Model of the Rails base. (For instance, if we were to rename Merge Request to Pull Request. (create(:merge_request) => create(:pull_request))

@ddavison many thanks for sharing your thoughts

Keep in mind that the seed file must be of this format, as documented here

Yes, I tried adding something like this originally for testing

# test_data.rb
class DataSeeder
  def seed
    FactoryBot.factories.first(10) do |factory|
      # Create a new instance for each factory
      Gitlab::AppLogger.debug factory.name
      FactoryBot.create(factory.name)
    end
  end
end

It didn't seed and failed with the same uninitialized constant Gitlab::Pagination::Keyset::Order, I'm probably missing something fundamental here as I'm still reading up about how factories work

This might be fixed with gitlab-org/gitlab!137582 (merged), in particular, the change to data_seeder.rb to include rspec, and a few other pertinent things.

I pulled the latest master in my GDK today with gitlab-org/gitlab!137582 (merged) is merged. I'm seeing the same NameError: uninitialized constant errors.

The odds of this are very low. Even if this were to happen, a global refactor across the entire codebase would be necessary.

Got it, thanks for the additional context Good to know that concern should be less as long as seed file lives in the project. I was thinking that perhaps at some point we would want to maintain few different Seeder configs for upgrade testing specifically that will be stored outside of the main repo. Will give it a further think after I get up to speed with this area first.

Created a crude MR for testing all fixture seeding using Seeder and snippets above - Seed all fixtures with Seeder (gitlab-org/gitlab!144631 - merged)

In the https://gitlab.com/gitlab-org/gitlab/-/jobs/6163413795 we can see that there are 351 successfully created fixtures (search ....success) and 161 failed fixtures (caught exception) if running against 16.3.0 Docker image.

When running same seeder file against local GDK( 16.9.0-pre 3a666cdff3f) I've seen 150 out of 963 fixtures failed with errors.

Fixture errors

1. No such file or directory @ rb_sysopen

Most common error - No such file or directory @ rb_sysopen - /opt/gitlab/embedded/service/gitlab-rails/tmp/tests/gitlab-test.bundle!

Cause: no Gitaly binaries compiled before the seeding such as done in CI - https://gitlab.com/gitlab-org/gitlab/-/blob/5915b2f8daa3023dd98dc3ee6cd23c4e518529b4/.gitlab/ci/rails.gitlab-ci.yml#L27.

Debugging:

Running scripts/gitaly-test-build which should build these bundles in GDK failed:

scripts/gitaly-test-build
/Users/niskhakova/GitLab/gitlab-development-kit/gitlab/spec/support/helpers/gitaly_setup.rb:92:in `system': 
No such file or directory - make (Errno::ENOENT)
        from /Users/niskhakova/GitLab/gitlab-development-kit/gitlab/spec/support/helpers/gitaly_setup.rb:92:in `run_command'
        from /Users/niskhakova/GitLab/gitlab-development-kit/gitlab/spec/support/helpers/gitaly_setup.rb:96:in `build_gitaly'
        from scripts/gitaly-test-build:21:in `run'
        from scripts/gitaly-test-build:42:in `<main>'

I found test: Install Gitaly incrementally (gitlab-org/gitlab!76558 - merged) where it looks like this command was moved to CI only run.

Also side note - I believe the error with above explains why MR seeded with bulk_data has missing branches:

In gitlab-test.bundle there is 0b4bc9a49b562e85de7cc9e834518ea6828729b9 refs/heads/feature ref - which should have been used in above MR.

2. `PG::UniqueViolation: ERROR`

Second common error. Most popular exception is for index_ci_namespace_mirrors_on_namespace_id - 41/45 cases.

Debugging:

To be done

3. Postgres factories related errors

Factory #206
reindex_action caught exception PG::ObjectNotInPrerequisiteState: ERROR:  cannot insert into view "postgres_indexes"
DETAIL:  Views that do not select from a single table or view are not automatically updatable.
HINT:  To enable inserting into the view, provide an INSTEAD OF INSERT trigger or an unconditional ON INSERT DO INSTEAD rule.
!

I assume it's expected to fail against live envs so can be ignored?

@niskhakova As you've already found, gitaly_setup.rb will attempt to build gitaly from source unless the environment variable CI is set. That is why it is trying to run make. So for this to succeed you'll either need to add all the build tools to the docker image, or build gitaly separately like CI does and set the environment appropriately.

@proglottis many thanks for the clarification on this as well as the tip about bundle exec rspec spec/lib/backup/gitaly_backup_spec.rb that would initiate TestEnv.init in GDK Confirmed that when running this command, the bundle files were created!

The "normal" GDK way where spec_helper.rb calls TestEnv.init which then calls TestEnv.setup_forked_repo/TestEnv.setup_factory_repo that actually do the cloning/ref-setting/bundle-creating

It helps, many thanks

Confirmed that with running these commands from Seeder config resolved all No such file or directory @ rb_sysopen https://gitlab.com/gitlab-org/gitlab/-/jobs/6181804106 - commit gitlab-org/gitlab!144631 (32b84867)

@niskhakova — I'm sure it's not a dealbreaker, but I do wonder how much we're deviating from the original intent of the epic

multi-version upgrades without building an environment

@ddavison could you please expand on this? In this thread we're exploring how to seed all fixtures, and as found in #2354 (comment 1772988958) - there are some errors that doesn't allow to seed the environment. Resolving the errors allows fixtures to be seeded.

The data is the crucial point of catching migrations, we need a way to seed all possible data and looks like Data Seeder + fixtures is the solution here.

In general, I don't think there is concern about multi-version upgrades without building an environment because we're working on preparing PG dump - this is the only place where environment is being used, however the test itself doesn't need an env (since we import the dump). It will be pulling existing PG dump for specific upgrade stop and not rely on any environment.

`PG::UniqueViolation: ERROR`

Following up on this - #2354 (comment 1772988958)

Second common error. Most popular exception is for index_ci_namespace_mirrors_on_namespace_id - 41/45 cases.

As mentioned above this was the second most common issue. Latest example when seeding was run 16.7.6 - https://gitlab.com/gitlab-org/quality/pg-dump-generator/-/jobs/6270181897#L1305

27 UniqueViolation errors, out of which 23 are index_ci_namespace_mirrors_on_namespace_id.

Reproduced this in my freshly reinstalled GDK. For example rails console:

gdk rails console
...
FactoryBot.create(:board_group_recent_visit)
# factory is created
=> #<BoardGroupRecentVisit:0x000000028223f508
 id: 1,
 created_at: Wed, 28 Feb 2024 17:25:45.371008000 UTC +00:00,
 updated_at: Wed, 28 Feb 2024 17:25:45.371008000 UTC +00:00,
 user_id: 70,
 board_id: 1,
 group_id: 96>
FactoryBot.create(:board_group_recent_visit)
# on second try failed with the same `index_ci_namespace_mirrors_on_namespace_id`
ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_ci_namespace_mirrors_on_namespace_id"
DETAIL:  Key (namespace_id)=(100) already exists.

FactoryBot.create(:board_group_recent_visit)
# didn't fail on third time
=> #<BoardGroupRecentVisit:0x0000000128e7ec48
 id: 2,
 created_at: Wed, 28 Feb 2024 17:26:11.482766000 UTC +00:00,
 updated_at: Wed, 28 Feb 2024 17:26:11.482766000 UTC +00:00,
 user_id: 73,
 board_id: 2,
 group_id: 102>

Not sure why rerunning the same command fixes this error.

Relevant docs found during investigation: https://docs.gitlab.com/ee/development/database/ci_mirrored_tables.html

Slightly changing seed_all_fixtures to retry on ActiveRecord::RecordNotUnique - resolved the issue with index_ci_namespace_mirrors_on_namespace_id:

  def seed_all_fixtures
    FactoryBot.factories.each_with_index do |factory, i|
      retries ||= 0
      # Create a new instance for each factory
      FactoryBot.create(factory.name)
    rescue ActiveRecord::RecordNotUnique => e
      puts "#{factory.name} caught exception #{e}! Attempt##{retries}"
      retry if (retries += 1) < 3
    rescue Exception => e # rubocop:disable Lint/RescueException -- catching all possible exceptions
      # We rescue exception here to make sure seeding proceeds unrelated to various unique exceptions from Factories
      puts "#{factory.name} caught exception #{e} of class #{e.class}!"
    else
      puts "Successfully created Factory #{factory.name}"
    ensure
      puts "Factory ##{i + 1}"
    end
  end

Example output

output

Factory #244
.incident_management_timeline_event_tag caught exception PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_ci_namespace_mirrors_on_namespace_id"
DETAIL:  Key (namespace_id)=(503) already exists.
! Attempt#0
...Successfully created Factory incident_management_timeline_event_tag
Factory #245

...

Factory #296
...Successfully created Factory telegram_integration
Factory #297
.internal_id caught exception PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_ci_namespace_mirrors_on_namespace_id"
DETAIL:  Key (namespace_id)=(611) already exists.
! Attempt#0
.internal_id caught exception PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "index_ci_namespace_mirrors_on_namespace_id"
DETAIL:  Key (namespace_id)=(612) already exists.
! Attempt#1
...Successfully created Factory internal_id

mentioned in merge request gitlab-com/content-sites/handbook!3404 (merged)

changed milestone to %16.10

Expand data seeding configuration for multi-version upgrades testing

Summary

Details

Context

Designs

Child items ...

Activity

Catching up with Seeder based on MR Add multi-version upgrade test (gitlab-org/gitlab!137582 - merged)

Validate that data selected approach work

Explore if there is a way to seed environment with all possible fixtures

To be clarified.

Fixture errors

1. No such file or directory @ rb_sysopen

2. `PG::UniqueViolation: ERROR`

3. Postgres factories related errors

`PG::UniqueViolation: ERROR`

Expand data seeding configuration for multi-version upgrades testing

Summary

Details

Context

Is blocked by

Relates to

Activity

Catching up with Seeder based on MR Add multi-version upgrade test (gitlab-org/gitlab!137582 - merged)

Validate that data selected approach work

Explore if there is a way to seed environment with all possible fixtures

To be clarified.

Fixture errors

1. No such file or directory @ rb_sysopen

2. PG::UniqueViolation: ERROR

3. Postgres factories related errors

PG::UniqueViolation: ERROR

2. `PG::UniqueViolation: ERROR`

`PG::UniqueViolation: ERROR`