Investigate and improve performance of /api/:version/projects/:id/dependencies

Why are we doing this work

This endpoint has been reported as underperforming and eating our error budget. See analysis in parent epic.

We must find out where the performance hit comes from and fix it to bring these requests within the acceptable threshold.

Relevant links

Non-functional requirements

  • Documentation:
  • Feature flag:
  • Performance:
  • Testing:

Implementation plan

  1. identify N+1 queries

    Use bullet to identify N+1 queries. I've already identified one such query in ee/app/models/ee/ci/build.rb:

    USE eager loading detected: Ci::Build => [:project] Add to your query: .includes([:project])

    This might be the only one, but we should confirm this.

  2. investigate the effect of adding pagination to Dependencies API - does it result in better performance when the default number of 20 items per page is returned?

  3. Profile Security::DependencyListService.execute:

    def execute
  collection = init_collection
  collection = filter_by_package_manager(collection)
  collection = filter_by_vulnerable(collection)
  sort(collection)
end

    The init_collection method of the execute method is responsible for:

    1. finding the latest dependency list reports
    2. parsing each dependency list report
      1. parse dependency names
      2. parse vulnerabilities
    3. returning the dependencies from the dependency list report

    We need to profile this execute method and determine where the majority of the time is spent in the above process, and figure out how to optimize this code path or add caching to improve performance.

Testing

This looks like an ideal candidate to introduce a performance test. SET to baseline current performance. Execute API call against performance improvement implementation, compare results. Compare to any quoted threshold.

Edited by Will Meek