Skip to content

Introduce Git only crawling to CMBR for HTTPs traffic

Nailia Iskhakova requested to merge 4-git-crawler into main

Adds traffic option to CMBR with web value by default. When specified as git it will only send git related HTTP traffic. It goes through projects via API and sends requests to git-upload-pack and git-receive-pack services. These requests need basic auth token to authorise.

Example output with debug:

 go run cmbr.go --instance https://staging.gitlab.com/api/v4/projects --concurrency 1 --duration 1m -output color -git -debug
DEBU[0000] Creating request for https://staging.gitlab.com/api/v4/projects  url="https://staging.gitlab.com/api/v4/projects"
DEBU[0000] Adding link                                   url="https://staging.gitlab.com/gitlab-qa-sandbox-group-3/qa-test-2022-06-14-11-18-24-72872b8f413209f7/push_rules-7e930446c00f4710.git/info/refs?service=git-upload-pack"
DEBU[0000] Adding link                                   url="https://staging.gitlab.com/gitlab-qa-sandbox-group-3/qa-test-2022-06-14-11-18-24-72872b8f413209f7/push_rules-7e930446c00f4710.wiki.git/info/refs?service=git-upload-pack"
DEBU[0000] Adding link                                   url="https://staging.gitlab.com/gitlab-qa-sandbox-group-3/qa-test-2022-06-14-11-18-24-72872b8f413209f7/push_rules-7e930446c00f4710.git/info/refs?service=git-receive-pack&repository_path=gitlab-qa-sandbox-group-3/qa-test-2022-06-14-11-18-24-72872b8f413209f7/push_rules-7e930446c00f4710"
DEBU[0000] Adding link                                   url="https://staging.gitlab.com/api/v4/projects/7675467"
DEBU[0000] Creating request for https://staging.gitlab.com/gitlab-qa-sandbox-group-3/qa-test-2022-06-14-11-18-24-72872b8f413209f7/push_rules-7e930446c00f4710.git/info/refs?service=git-upload-pack  url="https://staging.gitlab.com/gitlab-qa-sandbox-group-3/qa-test-2022-06-14-11-18-24-72872b8f413209f7/push_rules-7e930446c00f4710.git/info/refs?service=git-upload-pack"
DEBU[0000] Adding link                                   url="https://staging.gitlab.com/gitlab-qa-sandbox-group-3/qa-test-2022-06-14-11-18-24-72872b8f413209f7/push_rules-15b507ab4d53572c.git/info/refs?service=git-upload-pack"

Worth calling out that some of these requests may fail when target project doesn't have wiki or overall the project was deleted.

Verification that git traffic is now generated

Screenshot_2022-06-14_at_14.32.52

Closes #4 (closed)

Edited by Nailia Iskhakova

Merge request reports