SAST FP detection flow response - not valid json

Problem to solve

We've observed a lot of json decode errors in the FP detection evaluation. Upon further inspection we found the cause.

• Initially in the flow will output a valid json string as the final answer.

  Click to expand

  {\n  \"false_positive_likelihood\": 95,\n  \"explanation\": \"**Summary: This is a FALSE POSITIVE - Test Fixture Code**\\n\\n**Context Analysis:**\\n\\nThe reported vulnerability is located in the file `qa/fixtures/js/cjs/src/js/nodejs_crypto.cjs` at lines 11-13 ...}

• Recently this behavior has changed, and the final answer contains other text along with a json block

  Click to expand

  Based on my comprehensive analysis of the SAST finding, here is my assessment:\n\n```json\n{\n  \"false_positive_likelihood\": 95,\n  \"explanation\": \"**Summary: This is a FALSE POSITIVE - Test Fixture Code**\\n\\n**Context Analysis:**\\n\\nThe reported vulnerability is located in the file `qa/fixtures/js/cjs/src/js/nodejs_crypto.cjs` at lines 11-13 ...}

This has caused most examples to fail because we cannot decode the final answer as a json.

Proposal

• Determine the intended behavior of FP detection then adjust CEF accordingly.
• Temporarily change CEF's eval pipeline to expect a text block as oppose to a json block, so our evaluations can run without too many errors.

Further details

Links / references