Recent update to bandit 1.6.2 in SAST breaks Python scans
The update of bandit from the 1.5 series to 1.6.2 has resulted in bandit failing. In my case, its scanning around 10K files of python (and that's a problem see below).
Since SAST doesn't clean up containers, I was able to simply restart the container and run commands within it.
When running bandit manually via /usr/local/bin/python /usr/local/bin/bandit ...
the, container simply exits. Here's the output from that (beginning from when I ran docker exec -ti bash):
root@97300742bf0a:/tmp/app# time /usr/local/bin/python /usr/local/bin/bandit -a vuln -x tests,build,deploy,docs,.tox -r . [main] INFO profile include tests: None [main] INFO profile exclude tests: None [main] INFO cli include tests: None [main] INFO cli exclude tests: None [main] INFO running on Python 3.7.4 10572 [0.. 50.. 100.. 150.. 200.. 250.. 300.. 350.. 400.. 450.. 500.. 550.. 600.. 650.. 700.. 750.. 800.. 850.. 900.. 950.. 1000.. 1050.. 1100.. 1150.. 1200.. 1250.. 1300.. 1350.. 1400.. 1450.. 1500.. 1550.. 1600.. 1650.. 1700.. 1750.. 1800.. 1850.. 1900.. 1950.. 2000.. 2050.. 2100.. 2150.. 2200.. 2250.. 2300.. 2350.. 2400.. 2450.. [ ERROR Bandit internal error running: hashlib_new on file ./.tox/dev/lib/python3.6/site-packages/glanceclient/v2/images.py at line 245: 'NoneType' object has no attribute 'lower'Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/bandit/core/tester.py", line 64, in run_tests result = test(context) File "/usr/local/lib/python3.7/site-packages/bandit/plugins/hashlib_new_insecure_functions.py", line 57, in hashlib_new if name.lower() in ('md4', 'md5'): AttributeError: 'NoneType' object has no attribute 'lower'
2500.. 2550.. 2600.. 2650.. 2700.. 2750.. 2800.. 2850.. 2900.. 2950.. 3000.. 3050.. 3100.. 3150.. 3200.. 3250.. 3300.. 3350.. 3400.. 3450.. 3500.. 3550.. 3600.. 3650.. 3700.. 3750.. 3800.. 3850.. 3900.. 3950.. 4000.. 4050.. 4100.. 4150.. 4200.. 4250.. 4300.. 4350.. 4400.. 4450.. 4500.. 4550.. 4600.. 4650.. 4700.. 4750.. 4800.. 4850.. 4900.. 4950.. 5000.. 5050.. 5100.. 5150.. 5200.. 5250.. 5300.. 5350.. 5400.. 5450.. 5500.. 5550.. 5600.. 5650.. 5700.. 5750.. 5800.. 5850.. 5900.. 5950.. 6000.. 6050.. 6100.. 6150.. 6200.. 6250.. 6300.. 6350.. 6400.. 6450.. 6500.. 6550.. 6600.. 6650.. 6700.. 6750.. 6800.. 6850.. 6900.. 6950.. 7000.. 7050.. 7100.. 7150.. 7200.. 7250.. 7300.. 7350.. 7400.. 7450.. 7500.. 7550.. 7600.. 7650.. 7700.. 7750.. 7800.. 7850.. 7900.. 7950.. 8000.. [tlevi@lfp-runner ~]$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 97300742bf0a registry.gitlab.com/gitlab-org/security-products/analyzers/bandit:2 "/analyzer run" 3 hours ago Exited (0) 12 seconds ago hungry_keldysh
From the above, it only processed 8000 (or so) of the roughly 10K files it found. Also, the container simply exited and kicked me to the shell. The python exception in the output is a documented bug in bandit.
When running bandit manually via /usr/local/bin/bandit within that same container (again after starting it and running docker exec -ti ...), it gets a bit further (around 9700) before exiting. In both case, it seems to abort after around 20 minutes. These same scans work perfectly using bandit 1.5.1 and in far, far less than 20 minutes - in fact they ran in about 2 minutes per SAST logs. In those logs, the number of files scanned was 360 for the exact same project:
... 2019/08/21 12:11:08 [bandit] Starting analyzer... 2: Pulling from gitlab-org/security-products/analyzers/bandit Digest: sha256:dfa240ec684c9f8f0d0d6bc675714a249f8b887e5e39331350c62e8b118963f6 Status: Image is up to date for registry.gitlab.com/gitlab-org/security-products/analyzers/bandit:2 Found project in /tmp/app [main] INFO profile include tests: None [main] INFO profile exclude tests: None [main] INFO cli include tests: None [main] INFO cli exclude tests: None 360 [0.. 50.. 100.. 150.. 200.. 250.. 300.. 350.. ] [json] INFO JSON output written to file: /tmp/bandit.json 2019/08/21 12:13:27 [secrets] Detect project using plugin ...
So this seem to be a problem where bandit no longer honors the exclude and, therefore, takes longer to process. Here's one bug pointing at exactly that: https://github.com/PyCQA/bandit/issues/488. The most common workaround is to pin back to 1.5.1.