### Context The entire premise of Unified Secret Detection scan engine is that the engine should have first-class support for portability, performant at scale and resource efficient. In https://gitlab.com/gitlab-org/gitlab/-/issues/526227, we concluded to use the Vectorscan regex engine ported in Go. In this phase, we focus on building the core scan engine ground-up and make it modular enough for Engine Frontends (gRPC server/CLI/Domain-specific Adapters) to import it in their source. As outlined in the [ADR](https://gitlab.com/gitlab-com/content-sites/handbook/-/blob/1668f78462b090428e86f91617ca66dce93bb1c9/content/handbook/engineering/architecture/design-documents/secret_detection/decisions/007_switch_to_go_scan_engine.md), we're transitioning to a new scan engine that meets all the [required characteristics](https://gitlab.com/groups/gitlab-org/-/epics/17911#characteristics-of-a-unified-scan-engine), making it compatible across different types of scan targets. Read more about the idea of unifying the scan engine [here](https://gitlab.com/groups/gitlab-org/-/epics/17911). This issue tracks the progress of building the scan engine in Go and also switching the regex engine to [Vectorscan](https://github.com/VectorCamp/vectorscan). ### Implementation Plan * [x] Build the core logic of Secret Detection scan using Vectorscan regex engine. Include tests to confirm the behavior. * [x] Build the features to level the feature parity of [existing scan engine](https://gitlab.com/gitlab-org/security-products/secret-detection/secret-detection-service/-/tree/main/lib/gitlab/secret_detection/core?ref_type=heads) ex: accepting scan parameters, ruleset parsing, etc. * [x] Add Unit/Integration tests confirming the feature parity wherever applicable. MR: https://gitlab.com/gitlab-org/security-products/secret-detection/secret-detection-service/-/merge_requests/111 #### Desired Outcome A core engine Go module having the feature parity with existing [Ruby SD gem](https://gitlab.com/gitlab-org/security-products/secret-detection/secret-detection-service/-/tree/0a088a7e6f2bd5a43e70f1d9f272a5aa8d6b0e33/lib/gitlab) source and also is eligible for importing in other Go modules within the same Go workspace.