Improve Performance on Project Overview Page Load - O(n) Loads
Summary
Currently the files loaded on the Project Overview
> Details
and the Repository
> Files
page are loaded 25 at a time and in a single-threaded manner and each request for the next 25 results relies on the past request being completed. This implementation works when a project effectively relies on nesting (using directories) within project structures but when there is a need for a lot of files to be stored in a single directory, the user experience suffers with a time complexity of O(n).
Total Page Load time for a directory with 1,600 files is 1.2 minutes.
Example Project: poffey21/lots-of-files
Improvements
How can GitLab load the results in a way that scales up when the number of files increases to keep overall load time down?
This proposal simply changes the algorithm used to determine how request all files.
Option 1:
Add attribute in the TreeSummary
object that queries for total number of files which is subsequently added to the header (maybe call it More-Logs-Total
). Use that attribute to know how many files are needed to request and rely on the fetchLogsTree
JS method to determine how aggressive it should get in requesting files - including making multiple requests at once.
Option 2:
This option is not preferred given the issue: #30346 (closed)
We are able to accomplish this in the Web IDE... maybe the solution is to back-port whatever we built out there:
Risks
As we increase the complexity of this solution, we must ensure we account for additional situations.
- How do we employ a proper retry mechanism for failed requests?
- How do we ensure that we don't introduce performance problems on the server-side as we request additional information up-front?
Involved components
-
lib/gitlab/tree_summary.rb
: Add attribute that queries for total number of files. -
app/controllers/projects/refs_controller.rb
: Usetotal number of files
attribute and provide the result through aMore-Logs-Total
header. -
app/assets/javascripts/repository/log_tree.js
: Utilize newly proposed header & introduce retries.