Skip to content

Paginate projects pages for git traffic

Nailia Iskhakova requested to merge 8-paginate-git-traffic into main

Paginate through projects API list for git traffic

Verified that with this fix in place, cmbr crawls continuously. The below is the output with additional debugging to highlight that pages are being paginated:

go run cmbr.go --instance https://staging.gitlab.com/api/v4/projects -concurrency 4 -duration 55m -traffic git
INFO[0001] debugging...                                  Path=/api/v4/projects nextPage=2 q="map[page:[2]]" url="https://staging.gitlab.com/api/v4/projects?page=2"
ERRO[0009] request failed                                correlation_id=01G5MFRX95GRT5X4TA1R0ARZJ1 error=Forbidden gitlab_server=git-https-gke-us-east1-d status=403 url="https://staging.gitlab.com/gitlab-qa-sandbox-group-4/qa-test-2022-06-15-19-22-11-64ed23113e462410/pipeline-editor-project-bd0f0b8e7452bde6-deleted-7685783.git/info/refs?service=git-receive-pack&repository_path=gitlab-qa-sandbox-group-4/qa-test-2022-06-15-19-22-11-64ed23113e462410/pipeline-editor-project-bd0f0b8e7452bde6-deleted-7685783"
INFO[0010] debugging...                                  Path=/api/v4/projects nextPage=3 q="map[page:[3]]" url="https://staging.gitlab.com/api/v4/projects?page=3"
ERRO[0011] request failed                                correlation_id=01G5MFRYWARWG0SD9MTD35R3KK error=Forbidden gitlab_server=git-https-gke-us-east1-c status=403 url="https://staging.gitlab.com/gitlab-qa-sandbox-group-4/group-to-test-access-termination-a71b40ab935038fe/project-for-user-group-access-termination-ae15665e7fc32968-deleted-7685781.git/info/refs?service=git-receive-pack&repository_path=gitlab-qa-sandbox-group-4/group-to-test-access-termination-a71b40ab935038fe/project-for-user-group-access-termination-ae15665e7fc32968-deleted-7685781"
INFO[0012] crawling...                                   count=100
...
INFO[0085] debugging...                                  Path=/api/v4/projects nextPage=11 q="map[page:[11]]" url="https://staging.gitlab.com/api/v4/projects?page=11"
ERRO[0094] request failed                                correlation_id=01G5MFVFZ9YKY23NPZ5ZAJSDW1 error=Forbidden gitlab_server=git-https-gke-us-east1-b status=403 url="https://staging.gitlab.com/gitlab-qa-sandbox-group-4/qa-test-2022-06-15-19-21-30-11ee0e57361f82e7/badge-test-project-c3e147f39462d971-deleted-7685596.git/info/refs?service=git-receive-pack&repository_path=gitlab-qa-sandbox-group-4/qa-test-2022-06-15-19-21-30-11ee0e57361f82e7/badge-test-project-c3e147f39462d971-deleted-7685596"
INFO[0094] debugging...                                  Path=/api/v4/projects nextPage=12 q="map[page:[12]]" url="https://staging.gitlab.com/api/v4/projects?page=12"
INFO[0096] crawling...                                   count=800
^Csignal: interrupt

Closes #8 (closed)

Merge request reports