Improve vulnerabilities database seed performance
The following discussion from !101546 (merged) should be addressed:
-
@leetickett-gitlab started a discussion: (+2 comments) @bwill is it possible we accidentally changed this from seeding vulnerabilities for 5 projects to all projects?
I'm just doing a fresh GCK install and noticed how long it took to run the 20_vulnerabilities seed
Thanks
👍
Recreation Steps
git@gitlab-rails-web:~/gitlab$ bundle exec rails console
Vulnerabilities::Identifier.delete_all
Vulnerabilities::Scanner.delete_all
Ci::Pipeline.delete_all
git@gitlab-rails-web:~/gitlab$ bundle exec rake db:seed_fu FILTER=20_vuln
== Filtering seed files against regexp: /20_vuln/
== Seed from ee/db/fixtures/development/20_vulnerabilities.rb
2023-11-01T11:29:19+00:00
Seeding vulnerabilities for the 'toolbox/gitlab-smoke-tests' project
..............................
Done.
Seeding vulnerabilities for the 'gitlab-org/gitlab-test' project
..............................
Done.
Seeding vulnerabilities for the 'gitlab-org/gitlab-shell' project
..............................
Done.
Seeding vulnerabilities for the 'gnuwget/Wget2' project
..............................
Done.
Seeding vulnerabilities for the 'Commit451/lab-coat' project
..............................
Done.
Seeding vulnerabilities for the 'jashkenas/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'flightjs/Flight' project
..............................
Done.
Seeding vulnerabilities for the 'twitter/Typeahead.Js' project
..............................
Done.
Seeding vulnerabilities for the 'gaynell/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'claribel/gitlab-smoke-tests' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-1-1698447830/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-0-1698447824/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'lisha.powlowski/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'windy.stehr/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-2-1698447827/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-2-1698447799/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-0-1698447799/gitlab-smoke-tests' project
..............................
Done.
Seeding vulnerabilities for the 'len/gitlab-smoke-tests' project
..............................
Done.
2023-11-01T13:30:07+00:00
OK
Implementation Guide
Modify lib/tasks/gitlab/seed.rake
to limit the number of projects?
Perhaps something like projects = projects_from_args(args).shuffle(5)
?
It would be great to dig further into ee/lib/quality/seeders/vulnerabilities.rb
to see why this seed can be so slow.
Edited by Lee Tickett