Unified Secret Detection P1: Build Go-based Scan Engine

Context

The entire premise of Unified Secret Detection scan engine is that the engine should have first-class support for portability, performant at scale and resource efficient. In #526227 (closed), we concluded to use the Vectorscan regex engine ported in Go. In this phase, we focus on building the core scan engine ground-up and make it modular enough for Engine Frontends (gRPC server/CLI/Domain-specific Adapters) to import it in their source.

As outlined in the ADR, we're transitioning to a new scan engine that meets all the required characteristics, making it compatible across different types of scan targets. Read more about the idea of unifying the scan engine here. This issue tracks the progress of building the scan engine in Go and also switching the regex engine to Vectorscan.

Implementation Plan

• Build the core logic of Secret Detection scan using Vectorscan regex engine. Include tests to confirm the behavior.
• Build the features to level the feature parity of existing scan engine ex: accepting scan parameters, ruleset parsing, etc.
• Add Unit/Integration tests confirming the feature parity wherever applicable.

MR: gitlab-org/security-products/secret-detection/secret-detection-service!111 (merged)

Desired Outcome

A core engine Go module having the feature parity with existing Ruby SD gem source and also is eligible for importing in other Go modules within the same Go workspace.