Improve processing performance for bulk watched file changes being sent by clients
We've already improved the performance of these kinds of requests somewhat in #325 (closed), but there is room for more improvements.
If you clear the cache on a Symfony project, the container will be rebuilt. Clients such as VSCode will send a single watched file change notification with a large file list. Currently we reindex the files in this list one by one, but we could bundle these into a single reindex by refactoring the indexer.
This would be most advantageous if many of these changes happen in the same directory - which is the case for the Symfony cache -, as the IndexableFileIterator
creates a Symfony Finder
instance for the folder when scanning a single file (which then happens many times, one for each file), which would then become a single Finder
instance checking many files at once.
As an example: var/cache/test/containerXYZ
containing over 2000 files isn't exceptional for Symfony projects, such as those using Sylius, which we are currently all processing individually, which means we're scanning this folder 2000 times by creating 2000 Finder
instances, ouch.
Although, bundling files into a single request also means that they are processed simultaneously. This means handling all subsequent requests will be delayed again until the server is done with the full list, which is especially painful if no files share the same folder. The upside of processing them individually is that yes, it is less efficient in its totality, but it improves responsiveness by demuxing them into small requests, which are then low in priority.
An alternative solution is to group files with the same folder together in the didChangeWatchedFiles
handler and send these off in separate indexing requests, or use some kind of threshold of "at most 10 files".