Vulnerability report with Unicode NULL (U+0000) cannot be ingested
Some DAST API checks use the Unicode character U+0000
(the "null" character) in injection attacks. If these attacks result in vulnerabilities, the null character is written to the report in the evidence for the vulnerability. The resulting report cannot be ingested by the monolith, and an error is shown on the "Security" tab of the pipeline:
The problem is that the vulnerability with the null character cannot be stored in the database:
ActiveRecord::StatementInvalid
PG::UntranslatableCharacter: ERROR: unsupported Unicode escape sequence
LINE 1: ...2ff5588c5e66d45508253a26a914337b', 3820655, TRUE, '{"name":"...
^
DETAIL: \u0000 cannot be converted to text.
CONTEXT: JSON data, line 1: ...ation_byte_offsets":[],"raw_source_code_extract":...
PG::UntranslatableCharacter
ERROR: unsupported Unicode escape sequence
LINE 1: ...2ff5588c5e66d45508253a26a914337b', 3820655, TRUE, '{"name":"...
^
DETAIL: \u0000 cannot be converted to text.
CONTEXT: JSON data, line 1: ...ation_byte_offsets":[],"raw_source_code_extract":...
(note that the character being pointed to is the start of the string that contains the null character, not the location of the null character itself)
This is a known (though poorly documented) limitation of Postgres; see the chr ( integer ) function description. Postgres uses null-terminated strings internally and therefore cannot support the null character inside of a string.
See also: https://www.commandprompt.com/blog/null-characters-workarounds-arent-good-enough/
Related Sentry issue: https://new-sentry.gitlab.net/organizations/gitlab/issues/284657/?referrer=gitlab_integration
Workaround
Remove null characters using an after_script
:
include:
- Security/DAST-API.gitlab-ci.yml
dast_api:
after_script:
- sed -i 's/\\u0000//g' gl-dast-api-report.json
Implementation plan
a) Call gsub
on the finding_data
before handing it off to PostgreSQL OR
b) Create a function that basically calls regexp_replace(stringWithNull, '\\u0000', '', 'g')
on the finding_data
before saving
c) Replace the value with something else that would indicate that \u0000
is the value used there