Ultimate Scanners: DAST: issue with Proxy-Authorization header

Summary

We've been experiencing an issue when trying to use DAST (Dynamic Application Security Testing) with a Proxy-Authorization header.

When we provide this header it seems that a crash occurs internally preventing the crawling of the link to start. However, the READY module that check if our target is available works as expected with the Proxy-Authorization header.

Context

We want to use the Proxy-Authorization header to authenticate the requests on the IAP (Identity-Aware-Proxy) GCP service that we use in front of our apps. We can't use Authorization since it will collide with the internal application authentication.

cf:

Steps to reproduce

  • Go to https://webhook.site/
  • Copy a webhook URL link
  • Run command: docker run --env DAST_WEBSITE="https://webhook.site/YOUR_WEBHOOK_URL" --env DAST_REQUEST_HEADERS="proxy-authorization:Bearer thisIsMyToken" -i -t registry.gitlab.com/security-products/dast:latest
  • 💥 You will see warning messages in the logs of the container
2024-12-19T16:53:29.053 INF NAVDB adding sitemap as a seed URL sitemap="https://webhook.site/sitemap.xml"
2024-12-19T16:53:29.059 INF CRAWL crawled path nav_id="585d356053a3d25ea23c850b7d1e11380db383b8" path="LoadURL [https://webhook.site/sitemap.xml]" timeout_in="23h59m59.993s"
2024-12-19T16:53:29.059 INF CRAWL crawled path nav_id="7d8c408921e1ee4b35c395db9162d5cd5fff7c8c" path="LoadURL [https://webhook.site/17d1272b-8bbc-44b9-94e5-44a751667338]" timeout_in="23h59m59.993s"
2024-12-19T16:53:29.093 WRN CONTA request failed, attempting to continue scan index="0" requestID="D1F354CEEC7AA47F99C7B557544F5850" url="https://webhook.site/17d1272b-8bbc-44b9-94e5-44a751667338" error="net::ERR_INVALID_ARGUMENT"
2024-12-19T16:53:29.094 WRN CONTA request failed, attempting to continue scan index="0" requestID="A5B62378B86331C5963C80894BE6AAB5" url="https://webhook.site/sitemap.xml" error="net::ERR_INVALID_ARGUMENT"
2024-12-19T16:53:29.096 WRN CRAWL failed to process navigation, continuing browser_id="777797561472497289" nav_id="585d356053a3d25ea23c850b7d1e11380db383b8" of="1" step="1" error="crawler failed to process navigation LoadURL[https://webhook.site/sitemap.xml]: failed to execute navigation: failed to load URL: net::ERR_INVALID_ARGUMENT: error in navigation"
2024-12-19T16:53:29.096 WRN CRAWL failed to process navigation, continuing browser_id="3775261597121223817" nav_id="7d8c408921e1ee4b35c395db9162d5cd5fff7c8c" of="1" step="1" error="crawler failed to process navigation LoadURL[https://webhook.site/17d1272b-8bbc-44b9-94e5-44a751667338]: failed to execute navigation: failed to load URL: net::ERR_INVALID_ARGUMENT: error in navigation"
  • 💥 You will also be able to see that the webhook URL was only accessed once on the website webhook.site

Expected behavior:

  • DAST should not fail when we use a header Proxy-Authorization.
  • DAST should give an output similar to when the command is run without the header: docker run --env DAST_WEBSITE="https://webhook.site/YOUR_WEBHOOK_URL" --env DAST_REQUEST_HEADERS="Authorization:Bearer thisIsMyToken" -i -t registry.gitlab.com/security-products/dast:latest

Example Project

What is the current bug behavior?

When providing the Proxy-Authorization header it seems that a crash occurs internally preventing the crawling of the link to start. However, the READY module that check if our target is available works as expected with the Proxy-Authorization header.

What is the expected correct behavior?

  • DAST should not fail when we use a header Proxy-Authorization.
  • DAST should give an output similar to when the command is run without the header: docker run --env DAST_WEBSITE="https://webhook.site/YOUR_WEBHOOK_URL" --env DAST_REQUEST_HEADERS="Authorization:Bearer thisIsMyToken" -i -t registry.gitlab.com/security-products/dast:latest

Relevant logs and/or screenshots

  • 💥 You will see warning messages in the logs of the container
2024-12-19T16:53:29.053 INF NAVDB adding sitemap as a seed URL sitemap="https://webhook.site/sitemap.xml"
2024-12-19T16:53:29.059 INF CRAWL crawled path nav_id="585d356053a3d25ea23c850b7d1e11380db383b8" path="LoadURL [https://webhook.site/sitemap.xml]" timeout_in="23h59m59.993s"
2024-12-19T16:53:29.059 INF CRAWL crawled path nav_id="7d8c408921e1ee4b35c395db9162d5cd5fff7c8c" path="LoadURL [https://webhook.site/17d1272b-8bbc-44b9-94e5-44a751667338]" timeout_in="23h59m59.993s"
2024-12-19T16:53:29.093 WRN CONTA request failed, attempting to continue scan index="0" requestID="D1F354CEEC7AA47F99C7B557544F5850" url="https://webhook.site/17d1272b-8bbc-44b9-94e5-44a751667338" error="net::ERR_INVALID_ARGUMENT"
2024-12-19T16:53:29.094 WRN CONTA request failed, attempting to continue scan index="0" requestID="A5B62378B86331C5963C80894BE6AAB5" url="https://webhook.site/sitemap.xml" error="net::ERR_INVALID_ARGUMENT"
2024-12-19T16:53:29.096 WRN CRAWL failed to process navigation, continuing browser_id="777797561472497289" nav_id="585d356053a3d25ea23c850b7d1e11380db383b8" of="1" step="1" error="crawler failed to process navigation LoadURL[https://webhook.site/sitemap.xml]: failed to execute navigation: failed to load URL: net::ERR_INVALID_ARGUMENT: error in navigation"
2024-12-19T16:53:29.096 WRN CRAWL failed to process navigation, continuing browser_id="3775261597121223817" nav_id="7d8c408921e1ee4b35c395db9162d5cd5fff7c8c" of="1" step="1" error="crawler failed to process navigation LoadURL[https://webhook.site/17d1272b-8bbc-44b9-94e5-44a751667338]: failed to execute navigation: failed to load URL: net::ERR_INVALID_ARGUMENT: error in navigation"
  • 💥 You will also be able to see that the webhook URL was only accessed once on the website webhook.site

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of: \\\`sudo gitlab-rake gitlab:env:info\\\`) (For installations from source run and paste the output of: \\\`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production\\\`)

Results of GitLab application Check

Expand for output related to the GitLab application check 

(For installations with omnibus-gitlab package run and paste the output of: \`sudo gitlab-rake gitlab:check SANITIZE=true\`) (For installations from source run and paste the output of: \`sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true\`) (we will only investigate if the tests are passing)

Possible fixes

Describe what you would like to see improved.

Edited by Mathilde Riou