Skip to content

Remove orphaned registry folders

Craig Smith requested to merge clean_up_orphaned_registry_folders_325013 into master

What does this MR do?

For every feature branch a DAST engineer creates, a new image repository is created to store the current and future image of the branch to run the e2e tests against. However, once the branch is merged and deleted an empty image repository is left behind. There are currently over 200 orphaned image repositories in the dast container registry which contributes to making the registry very hard to navigate.

This MR adds the task clean orphaned image repositories, that once a branch has been merged into master, runs and deletes all empty image repositories that no longer have a corresponding branch.

The script cleanup-registry.py currently runs as --dry-run which only lists the repositories to be deleted and doesn't actually delete anything. Once this MR is merged, I'd like to review the behaviour of the script before removing the --dry-run flag.

An example of this job running as a dry run can be found here: https://gitlab.com/gitlab-org/security-products/dast/-/jobs/1122845382

The following image repos will be removed:

List of image repositories that will be removed: 
Will remove registry.gitlab.com/gitlab-org/security-products/dast/run-scan-on-list-of-urls-214120
Will remove registry.gitlab.com/gitlab-org/security-products/dast/print-alerts-rules-and-write-report
Will remove registry.gitlab.com/gitlab-org/security-products/dast/share-dup-risk-codes-293971
Will remove registry.gitlab.com/gitlab-org/security-products/dast/wait-for-passive-scan
Will remove registry.gitlab.com/gitlab-org/security-products/dast/da-readd-config-file
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-zap-config-file-vars
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-unused-zap-code
Will remove registry.gitlab.com/gitlab-org/security-products/dast/fix-docker-uid-volume-mapping-issue
Will remove registry.gitlab.com/gitlab-org/security-products/dast/rename-underscore-functions
Will remove registry.gitlab.com/gitlab-org/security-products/dast/check-changelog-225438
Will remove registry.gitlab.com/gitlab-org/security-products/dast/url-scan-does-not-require-a-target-214120
Will remove registry.gitlab.com/gitlab-org/security-products/dast/ensure-api-scheme-is-in-zap-wrk
Will remove registry.gitlab.com/gitlab-org/security-products/dast/excluded-rules-dont-execute
Will remove registry.gitlab.com/gitlab-org/security-products/dast/document-zap-server-configuration
Will remove registry.gitlab.com/gitlab-org/security-products/dast/auto-create-issue-to-update-zap-addons
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-test-invoke-tasks
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-zap-base-image-to-w2020-06-30
Will remove registry.gitlab.com/gitlab-org/security-products/dast/exclude-vulnerabilities-not-for-allowed-hosts-50
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-gitlab-supported-runner
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-missing-changelog-zap-user
Will remove registry.gitlab.com/gitlab-org/security-products/dast/refactor-python-modules
Will remove registry.gitlab.com/gitlab-org/security-products/dast/post-server-start-tasks
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-changelog-version
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-changelog
Will remove registry.gitlab.com/gitlab-org/security-products/dast/229068-fix-log-debug
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-integration-tests
Will remove registry.gitlab.com/gitlab-org/security-products/dast/migrate-spider
Will remove registry.gitlab.com/gitlab-org/security-products/dast/url-scan-passive-scan-214120
Will remove registry.gitlab.com/gitlab-org/security-products/dast/incorporate-browserker
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-location-from-description-field
Will remove registry.gitlab.com/gitlab-org/security-products/dast/trigger-dast-benchmark-on-release
Will remove registry.gitlab.com/gitlab-org/security-products/dast/fix-output-of-rules-executed
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-browserker-user-to-gitlab
Will remove registry.gitlab.com/gitlab-org/security-products/dast/docs-add-instructions-to-read-zap-db
Will remove registry.gitlab.com/gitlab-org/security-products/dast/fix-docker-in-dast-pipeline
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-scan-delays
Will remove registry.gitlab.com/gitlab-org/security-products/dast/dont-run-pipeline-on-tags
Will remove registry.gitlab.com/gitlab-org/security-products/dast/227549-dast-version-guidelines
Will remove registry.gitlab.com/gitlab-org/security-products/dast/validate-target-273355
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-license-check-290958
Will remove registry.gitlab.com/gitlab-org/security-products/dast/use-zap-as-a-proxy
Will remove registry.gitlab.com/gitlab-org/security-products/dast/mask-http-header-values
Will remove registry.gitlab.com/gitlab-org/security-products/dast/improve-full-scan-debug-output
Will remove registry.gitlab.com/gitlab-org/security-products/dast/access-scanned-resources-from-the-db
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-useful-flake8-rules
Will remove registry.gitlab.com/gitlab-org/security-products/dast/profile-python-memory-useage
Will remove registry.gitlab.com/gitlab-org/security-products/dast/sethgitlab-secretdetection
Will remove registry.gitlab.com/gitlab-org/security-products/dast/207335-fail-if-auth-fails
Will remove registry.gitlab.com/gitlab-org/security-products/dast/aggregated-vulnerabilities-should-match-schema
Will remove registry.gitlab.com/gitlab-org/security-products/dast/fix-reported-schema-version
Will remove registry.gitlab.com/gitlab-org/security-products/dast/verify-active-scan-rules-can-be-excluded
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-use-context-option
Will remove registry.gitlab.com/gitlab-org/security-products/dast/address-feedback-exclude-rules-2
Will remove registry.gitlab.com/gitlab-org/security-products/dast/grab-filename-from-path
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-progress-file
Will remove registry.gitlab.com/gitlab-org/security-products/dast/scan-urls-found-by-browserker-49
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-missing-changelog-entries
Will remove registry.gitlab.com/gitlab-org/security-products/dast/post-deploy-integration-test-213621
Will remove registry.gitlab.com/gitlab-org/security-products/dast/dast-target-availability-timeout-238448
Will remove registry.gitlab.com/gitlab-org/security-products/dast/consistent-exception-handling
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-addons-instructions
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-start-server-tasks
Will remove registry.gitlab.com/gitlab-org/security-products/dast/ping-after-zap-start
Will remove registry.gitlab.com/gitlab-org/security-products/dast/bump-browserker-0-0-20
Will remove registry.gitlab.com/gitlab-org/security-products/dast/invoke-multipage-site
Will remove registry.gitlab.com/gitlab-org/security-products/dast/dast-fails-when-zap-fails
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-naming-guidelines
Will remove registry.gitlab.com/gitlab-org/security-products/dast/switch-on-zap-server-debug-logging-218972
Will remove registry.gitlab.com/gitlab-org/security-products/dast/show-filtered-rules-as-skipped
Will remove registry.gitlab.com/gitlab-org/security-products/dast/run-memory-test-on-master-only
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-legacy-entrypoints
Will remove registry.gitlab.com/gitlab-org/security-products/dast/ignore-hsql-log-statements
Will remove registry.gitlab.com/gitlab-org/security-products/dast/access-messages-from-the-zap-db
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-pancakes
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-browserker-to-vulnerability-scan-name-323878
Will remove registry.gitlab.com/gitlab-org/security-products/dast/allow-dast-paths-file-to-be-an-artifact
Will remove registry.gitlab.com/gitlab-org/security-products/dast/run-as-non-zap-user
Will remove registry.gitlab.com/gitlab-org/security-products/dast/create-scan-coordinator-without-strict-di
Will remove registry.gitlab.com/gitlab-org/security-products/dast/migrate-target-processing
Will remove registry.gitlab.com/gitlab-org/security-products/dast/323454-docker-20-10-5
Will remove registry.gitlab.com/gitlab-org/security-products/dast/e2e-full-scan-future-test
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-dast-paths-to-browserker-scan-70
Will remove registry.gitlab.com/gitlab-org/security-products/dast/upgrade-schema-to-13-1-0
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-dvwa-invoke-task
Will remove registry.gitlab.com/gitlab-org/security-products/dast/build-url-using-model-248998
Will remove registry.gitlab.com/gitlab-org/security-products/dast/222781-touch-dockerenv
Will remove registry.gitlab.com/gitlab-org/security-products/dast/refactor-alerts
Will remove registry.gitlab.com/gitlab-org/security-products/dast/fix-flakey-cli-options-test
Will remove registry.gitlab.com/gitlab-org/security-products/dast/replace-webdriver-auth-with-browserker-scans
Will remove registry.gitlab.com/gitlab-org/security-products/dast/aggregate-descriptions-254043
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-auth-help-text-280472
Will remove registry.gitlab.com/gitlab-org/security-products/dast/prevent-running-ajax-spider-with-api-scan
Will remove registry.gitlab.com/gitlab-org/security-products/dast/fix-dast-pipeline
Will remove registry.gitlab.com/gitlab-org/security-products/dast/247789-fail-for-500s
Will remove registry.gitlab.com/gitlab-org/security-products/dast/enable-api-rules
Will remove registry.gitlab.com/gitlab-org/security-products/dast/docs-editing-zap-code
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-browserker-auth
Will remove registry.gitlab.com/gitlab-org/security-products/dast/upgrade-firefox-37928
Will remove registry.gitlab.com/gitlab-org/security-products/dast/first-class-messages
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-properties-to-alert
Will remove registry.gitlab.com/gitlab-org/security-products/dast/revert-remove-zap-target-ping
Will remove registry.gitlab.com/gitlab-org/security-products/dast/267403-default-host-start-to-false
Will remove registry.gitlab.com/gitlab-org/security-products/dast/dast-website-should-default-to-none
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-build-tool
Will remove registry.gitlab.com/gitlab-org/security-products/dast/bring-auth-first-submit-field-name-xpath-inline-with-auth-submi
Will remove registry.gitlab.com/gitlab-org/security-products/dast/build-future
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-gitlab-supported-runner-tag
Will remove registry.gitlab.com/gitlab-org/security-products/dast/214298-add-discovered-at-to-vulnerabilities
Will remove registry.gitlab.com/gitlab-org/security-products/dast/use-dast-website-with-dast-urls-214120
Will remove registry.gitlab.com/gitlab-org/security-products/dast/mask-sensitive-values
Will remove registry.gitlab.com/gitlab-org/security-products/dast/upgrade-addons-with-zap-api
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-changelog-to-enable-deploy
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-zap-try-catch
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-large-memory-message-values
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-zap-addons-08-07-2020
Will remove registry.gitlab.com/gitlab-org/security-products/dast/chrome
Will remove registry.gitlab.com/gitlab-org/security-products/dast/dry-up-baseline-end-to-end-tests-33915
Will remove registry.gitlab.com/gitlab-org/security-products/dast/handle-headers-with-colons
Will remove registry.gitlab.com/gitlab-org/security-products/dast/improve-target-access-error-messages
Will remove registry.gitlab.com/gitlab-org/security-products/dast/extract-vulnerability-report-formatter
Will remove registry.gitlab.com/gitlab-org/security-products/dast/dev-guidelines
Will remove registry.gitlab.com/gitlab-org/security-products/dast/use-sed-i-to-inplace-edit
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-env-var-name-from-help-text
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-addons-280767
Will remove registry.gitlab.com/gitlab-org/security-products/dast/begin-removing-zap-scripts
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-base-to-docker-20-10-5-in-python-3-9
Will remove registry.gitlab.com/gitlab-org/security-products/dast/improve-ajax-tests
Will remove registry.gitlab.com/gitlab-org/security-products/dast/dont-tag-urls-because-performance
Will remove registry.gitlab.com/gitlab-org/security-products/dast/exclude-elements-from-browserker-scan
Will remove registry.gitlab.com/gitlab-org/security-products/dast/memory-spike
Will remove registry.gitlab.com/gitlab-org/security-products/dast/dast-contributing-review-slo
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-browserker-tests
Will remove registry.gitlab.com/gitlab-org/security-products/dast/refactor-url-scan-configuration-parsing
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-e2e-test-for-dast-latest-gitlab-ci-yml
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-deprecated-auth-env-variables
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-output-as-artifact-to-e2e-tests
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-task-to-write-a-plugin
Will remove registry.gitlab.com/gitlab-org/security-products/dast/browserker-rename-disable-headless-to-headless-42
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-tool-versions
Will remove registry.gitlab.com/gitlab-org/security-products/dast/use-browserker-future-to-build-dast-future
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-selenium-to-3-141-0
Will remove registry.gitlab.com/gitlab-org/security-products/dast/warn-about-host-override
Will remove registry.gitlab.com/gitlab-org/security-products/dast/continue-remove-zap-scripts
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-multi-page-test-website-214120
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-dast-paths-file-258825
Will remove registry.gitlab.com/gitlab-org/security-products/dast/allow-user-to-specify-session-name-258958
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-peak-memory-useage-to-metrics-report-227729
Will remove registry.gitlab.com/gitlab-org/security-products/dast/fix-openshift-changelog
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-urls-to-scan-from-active-scan
Will remove registry.gitlab.com/gitlab-org/security-products/dast/move-zaproxy-to-zap-gateway
Will remove registry.gitlab.com/gitlab-org/security-products/dast/setup-zap-project
Will remove registry.gitlab.com/gitlab-org/security-products/dast/exclude-rules-follow-up
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-zap-scripts
Will remove registry.gitlab.com/gitlab-org/security-products/dast/start-zap-server
Will remove registry.gitlab.com/gitlab-org/security-products/dast/configure-browserker-log-level
Will remove registry.gitlab.com/gitlab-org/security-products/dast/replace-dast-auth-browserker-auth
Will remove registry.gitlab.com/gitlab-org/security-products/dast/migrate-api-spec-loading
Will remove registry.gitlab.com/gitlab-org/security-products/dast/ssarka-master-patch-76422
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-request-headers-browserker-e2e
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-typing-metrics-report
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-dast-auth-exclude-urls-after-14-289959
Will remove registry.gitlab.com/gitlab-org/security-products/dast/aggregate-vulnerabilities-254043
Will remove registry.gitlab.com/gitlab-org/security-products/dast/setup-webgoat-as-a-fixture
Will remove registry.gitlab.com/gitlab-org/security-products/dast/migrate-active-scan-in-smaller-chunks
Will remove registry.gitlab.com/gitlab-org/security-products/dast/explicitly-call-custom-hooks
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-integration-test-docs
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-typing
Will remove registry.gitlab.com/gitlab-org/security-products/dast/292434-set-env-to-disable-tzdata-console
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-context-to-url-scan-247782
Will remove registry.gitlab.com/gitlab-org/security-products/dast/hsqldb-server-mode-238113
Will remove registry.gitlab.com/gitlab-org/security-products/dast/configure-browserker
Will remove registry.gitlab.com/gitlab-org/security-products/dast/document-invoke-tasks
Will remove registry.gitlab.com/gitlab-org/security-products/dast/229067-allow-skip-target-check
Will remove registry.gitlab.com/gitlab-org/security-products/dast/use-utc-timezone
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-dup-scan-creation
Will remove registry.gitlab.com/gitlab-org/security-products/dast/migrate-active-scan
Will remove registry.gitlab.com/gitlab-org/security-products/dast/bump-browserker-0-0-19
Will remove registry.gitlab.com/gitlab-org/security-products/dast/turn-aggregated-vulnerabilities-ff-on-by-default
Will remove registry.gitlab.com/gitlab-org/security-products/dast/log-scanned-urls-after-spider-completes-209043
Will remove registry.gitlab.com/gitlab-org/security-products/dast/verify-scan-does-not-run-out-of-memory
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-deploy-script-to-deal-with-quotes-285443
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-danger-line-count-warning
Will remove registry.gitlab.com/gitlab-org/security-products/dast/update-changelog-entry-mr
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-mypy-job
Will remove registry.gitlab.com/gitlab-org/security-products/dast/disabled-deprecated-config-arguments-299919
Will remove registry.gitlab.com/gitlab-org/security-products/dast/allow-failure-on-benchmark
Will remove registry.gitlab.com/gitlab-org/security-products/dast/alias-dast-auth-exclude-urls-as-dast-exclude-urls-289959
Will remove registry.gitlab.com/gitlab-org/security-products/dast/rename-executed-rule-to-rule
Will remove registry.gitlab.com/gitlab-org/security-products/dast/move-wrk-dir-env-var-to-shared-class-273334
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-ability-to-prevent-dast-start-at-host
Will remove registry.gitlab.com/gitlab-org/security-products/dast/reproduce-cli-opts-issue-223827
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-legacy-zaproxy-fields-33915
Will remove registry.gitlab.com/gitlab-org/security-products/dast/document-factory-usage
Will remove registry.gitlab.com/gitlab-org/security-products/dast/disable-unix-timestamp-check-253990
Will remove registry.gitlab.com/gitlab-org/security-products/dast/pin-firefox-version
Will remove registry.gitlab.com/gitlab-org/security-products/dast/show-skipped-rules-in-log
Will remove registry.gitlab.com/gitlab-org/security-products/dast/clean-up-unit-tests
Will remove registry.gitlab.com/gitlab-org/security-products/dast/remove-zap-target-ping
Will remove registry.gitlab.com/gitlab-org/security-products/dast/add-type-for-configuration
Will remove registry.gitlab.com/gitlab-org/security-products/dast/upgrade-addons-231278
Will remove registry.gitlab.com/gitlab-org/security-products/dast/da-fix-spider-timeout
Will remove registry.gitlab.com/gitlab-org/security-products/dast/migrate-ajax-spider
Will remove registry.gitlab.com/gitlab-org/security-products/dast/enable-request-response
Will remove registry.gitlab.com/gitlab-org/security-products/dast/avoid-duplicate-url-count
Will remove registry.gitlab.com/gitlab-org/security-products/dast/extract-zap-server-configuration
Will remove registry.gitlab.com/gitlab-org/security-products/dast/error-when-dast-paths-is-too-large-255232
Will remove registry.gitlab.com/gitlab-org/security-products/dast/load-scripts-from-directory
Will remove registry.gitlab.com/gitlab-org/security-products/dast/sethgitlab-dangerbot-test
Will remove registry.gitlab.com/gitlab-org/security-products/dast/log-dast-version

What are the relevant issue numbers?

gitlab-org/gitlab#325013 (closed)

Does this MR meet the acceptance criteria?

Edited by Craig Smith

Merge request reports