Improve vulnerability db seed performance
requested to merge gitlab-community/gitlab:430094-improve-vulnerabilities-seed-performance into master
What does this MR do and why?
Improves vulnerability db seed performance 250x
It seems gitlab-rails/app/validators/addressable_url_validator.rb is really slow (probably doing DNS lookups and maybe trying to make a request to the URL).
Switching to localhost seems to avoid the lookup/performance issue.
I will create a follow up issue to see where else this might be hindering performance and better understand it.
Screenshots or screen recordings
Before: 2hrs
git@gitlab-rails-web:~/gitlab$ bundle exec rake db:seed_fu FILTER=20_vuln
== Filtering seed files against regexp: /20_vuln/
== Seed from ee/db/fixtures/development/20_vulnerabilities.rb
2023-11-01T11:29:19+00:00
Seeding vulnerabilities for the 'toolbox/gitlab-smoke-tests' project
..............................
Done.
Seeding vulnerabilities for the 'gitlab-org/gitlab-test' project
..............................
Done.
Seeding vulnerabilities for the 'gitlab-org/gitlab-shell' project
..............................
Done.
Seeding vulnerabilities for the 'gnuwget/Wget2' project
..............................
Done.
Seeding vulnerabilities for the 'Commit451/lab-coat' project
..............................
Done.
Seeding vulnerabilities for the 'jashkenas/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'flightjs/Flight' project
..............................
Done.
Seeding vulnerabilities for the 'twitter/Typeahead.Js' project
..............................
Done.
Seeding vulnerabilities for the 'gaynell/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'claribel/gitlab-smoke-tests' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-1-1698447830/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-0-1698447824/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'lisha.powlowski/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'windy.stehr/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-2-1698447827/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-2-1698447799/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-0-1698447799/gitlab-smoke-tests' project
..............................
Done.
Seeding vulnerabilities for the 'len/gitlab-smoke-tests' project
..............................
Done.
2023-11-01T13:30:07+00:00After: 29s
git@gitlab-rails-web:~/gitlab$ bundle exec rake db:seed_fu FILTER=20_vuln
== Filtering seed files against regexp: /20_vuln/
== Seed from ee/db/fixtures/development/20_vulnerabilities.rb
2023-11-02T15:18:50+00:00
Seeding vulnerabilities for the 'toolbox/gitlab-smoke-tests' project
..............................
Done.
Seeding vulnerabilities for the 'gitlab-org/gitlab-test' project
..............................
Done.
Seeding vulnerabilities for the 'gitlab-org/gitlab-shell' project
..............................
Done.
Seeding vulnerabilities for the 'gnuwget/Wget2' project
..............................
Done.
Seeding vulnerabilities for the 'Commit451/lab-coat' project
..............................
Done.
Seeding vulnerabilities for the 'jashkenas/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'flightjs/Flight' project
..............................
Done.
Seeding vulnerabilities for the 'twitter/Typeahead.Js' project
..............................
Done.
Seeding vulnerabilities for the 'gaynell/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'claribel/gitlab-smoke-tests' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-1-1698447830/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-0-1698447824/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'windy.stehr/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-2-1698447827/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-2-1698447799/Underscore' project
..............................
Done.
Seeding vulnerabilities for the 'i-user-0-1698447799/gitlab-smoke-tests' project
..............................
Done.
Seeding vulnerabilities for the 'len/gitlab-smoke-tests' project
..............................
Done.
2023-11-02T15:19:19+00:00
How to set up and validate locally
- Clean up existing records from the rails console (
bundle exec rails console):Vulnerabilities::Identifier.delete_all Vulnerabilities::Scanner.delete_all Ci::Pipeline.delete_all - Run the seed script
bundle exec rake db:seed_fu FILTER=20_vuln - Profit!
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #430094 (closed)