Fix flaky DAST AJAX spider end-to-end test
Problem
There is a flaky test in the DAST end-to-end tests which regularly cause pipelines to fail. It's also helped condition engineers to expect failing AJAX end-to-end tests, such that when the AJAX spider actually broken it was unintentionally released to production.
Failure
The end-to-end-test 1/6
fails with the following output:
Error Output
Running test_ajax_spider_starts_at_target_url ... --- /dev/fd/63
+++ /dev/fd/62
@@ -25,11 +25,6 @@
{
"method": "GET",
"type": "url",
- "url": "http://nginx"
- },
- {
- "method": "GET",
- "type": "url",
"url": "http://nginx/bacon.html"
},
{
@@ -79,10 +74,6 @@
"urls": {
"items": [
{
- "href": "http://nginx",
- "type": "url"
- },
- {
"href": "http://nginx/bacon.html",
"type": "url"
},
@@ -111,12 +102,12 @@
"request": {
"headers": [],
"method": "GET",
- "url": "http://nginx"
+ "url": "http://nginx/bacon.html"
},
"response": {
"headers": [],
- "reason_phrase": "Not Found",
- "status_code": 404
+ "reason_phrase": "OK",
+ "status_code": 200
},
"summary": ""
},
@@ -178,10 +169,6 @@
"urls": {
"items": [
{
- "href": "http://nginx",
- "type": "url"
- },
- {
"href": "http://nginx/robots.txt",
"type": "url"
},
@@ -198,7 +185,7 @@
"request": {
"headers": [],
"method": "GET",
- "url": "http://nginx"
+ "url": "http://nginx/robots.txt"
},
"response": {
"headers": [],
@@ -247,10 +234,6 @@
"urls": {
"items": [
{
- "href": "http://nginx",
- "type": "url"
- },
- {
"href": "http://nginx/bacon.html",
"type": "url"
},
@@ -279,12 +262,12 @@
"request": {
"headers": [],
"method": "GET",
- "url": "http://nginx"
+ "url": "http://nginx/bacon.html"
},
"response": {
"headers": [],
- "reason_phrase": "Not Found",
- "status_code": 404
+ "reason_phrase": "OK",
+ "status_code": 200
},
"summary": "nginx/1.22.0"
},
FAILURE
Analyze results differ from expectations
expected [0] but was [1]
./end-to-end-test-helpers.sh:16:assert_output()
test-ajax-spider.sh:41:test_ajax_spider_starts_at_target_url()
An example failing job can be found at https://gitlab.com/gitlab-org/security-products/dast/-/jobs/5435862063.
Proposal
Fix the problem. Simplify the test if necessary, as long the test verifies that the AJAX spider ran correctly.
Edited by Cameron Swords