[Skims] Analyze big repositories
What is the current bug behavior? how can we reproduce it?
When running Skims over the Java Juliet Test Suite, the execution is broken.
What is the expected correct behavior?
It should complete the analysis and report the vulnerabilities found.
Relevant logs and/or screenshots
Exception in thread QueueManagerThread:
Traceback (most recent call last):
File "/nix/store/7ssj1jid9dqikk513rvg340kpbxrv0p0-python3-3.8.11/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/nix/store/7ssj1jid9dqikk513rvg340kpbxrv0p0-python3-3.8.11/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/nix/store/7ssj1jid9dqikk513rvg340kpbxrv0p0-python3-3.8.11/lib/python3.8/concurrent/futures/process.py", line 394, in _queue_management_worker
work_item.future.set_exception(bpe)
File "/nix/store/7ssj1jid9dqikk513rvg340kpbxrv0p0-python3-3.8.11/lib/python3.8/concurrent/futures/_base.py", line 547, in set_exception
raise InvalidStateError('{}: {!r}'.format(self._state, self))
concurrent.futures._base.InvalidStateError: CANCELLED: <Future at 0x7f62985ade20 state=cancelled>
/nix/store/kwggs4lgw8gxadh1lfpih3clnsmmh0j0-skims/core/scan.py:241: RuntimeWarning: coroutine 'reset' was never awaited
reset_ephemeral_state()
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
[WARNING] Function: __main__.cli_scan_wrapped, type: A process in the process pool was terminated abruptly while the future was running or pending.
Traceback (most recent call last):
File "/nix/store/kwggs4lgw8gxadh1lfpih3clnsmmh0j0-skims/utils/function.py", line 178, in wrapper
return function(*args, **kwargs)
File "/nix/store/kwggs4lgw8gxadh1lfpih3clnsmmh0j0-skims/cli/__init__.py", line 301, in cli_scan_wrapped
success: bool = core.scan.main(
File "/nix/store/kwggs4lgw8gxadh1lfpih3clnsmmh0j0-skims/core/scan.py", line 237, in main
return execute_skims(token)
File "/nix/store/kwggs4lgw8gxadh1lfpih3clnsmmh0j0-skims/core/scan.py", line 94, in execute_skims
analyze_root(stores=stores)
File "/nix/store/kwggs4lgw8gxadh1lfpih3clnsmmh0j0-skims/lib_root/analyze.py", line 93, in analyze
graph_db = parse.get_graph_db(tuple(unique_paths))
File "/nix/store/kwggs4lgw8gxadh1lfpih3clnsmmh0j0-skims/sast/parse.py", line 457, in get_graph_db
for index, shard in enumerate(parse_many(paths), start=1):
File "/nix/store/kwggs4lgw8gxadh1lfpih3clnsmmh0j0-skims/sast/parse.py", line 431, in parse_many
for parsed in worker.map(
File "/nix/store/7ssj1jid9dqikk513rvg340kpbxrv0p0-python3-3.8.11/lib/python3.8/concurrent/futures/process.py", line 484, in _chain_from_iterable_of_lists
for element in iterable:
File "/nix/store/7ssj1jid9dqikk513rvg340kpbxrv0p0-python3-3.8.11/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
yield fs.pop().result()
File "/nix/store/7ssj1jid9dqikk513rvg340kpbxrv0p0-python3-3.8.11/lib/python3.8/concurrent/futures/_base.py", line 444, in result
return self.__get_result()
File "/nix/store/7ssj1jid9dqikk513rvg340kpbxrv0p0-python3-3.8.11/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.
[INFO] Success: False
Possible fixes
-
Does the ProcessPool cannot live that long? -
Is it due to the amount of files in the repository to analyze (47k+)? -
Does it happen when building the graph for a particular file?
Edited by Diego Restrepo