Reduce complexity of count operation during policy violation check
Problem
- We are using
count > vulnerabilities_allowed
andpreexisting_count > vulnerabilities_allowed
logic during the policy violation check.
The drawback with this implementation is that:
we need to count all the vulnerabilities. The problem is that if there are thousands of UUIDs and if the
vulnerabilities_allowed
is just 1, then we will be looping unnecessarily.
Is it possible to count until we reach
vulnerabilities_allowed
?
See related discussion.
-
with_finding_by_uuid_and_state
is inefficient and it was observed that queries run > 100ms which is not ideal. Consider if we can useid
instead ofuuid
during the counting operation.
See related discussion
Solution proposal
Possible solutions for problem 1:
-
One micro optimization we could try is to refactor the code to break out of each_slice as soon as we have count + findings_count greater than vulnerabilities_allowed That would reduce the no, of queries. Because ideally customers would not set vulnerabilities_allowed to be more than 1-10
-
Evaluate if the
count_by_uuid
operation can be moved to using PostgreSQLexists
by including thevulnerabilities_allowed
parameter.
verification
From the logs capture:
-
-
SyncReportsToReportApprovalRulesWorker
run time should have reduced as we have removed the parse findings and update approvals_required via !115798 (merged) and !115825 (merged)
-
-
- Lets say the run time of the above worker reduced is x duration from its previous known avg run time.
-
- The new worker
SyncFindingsToApprovalRulesWorker
should ideally have a run time less then x duration or close toSyncReportsToReportApprovalRulesWorker
, since it has the optimisations.
- The new worker