Limiter: Re-implement Resizable Semaphore
Resizable sempahore was a new data structure. As the same suggested, it is a data structure to synchronize request concurrency. It has the ability to adjust the size dynamically. The implementation spawns a dedicated goroutine to manage the state. Unfortunately, in !6183 (merged), the integration didn't clean up the semaphore properly, leading to goroutine leaking incident (gitlab-com/gl-infra/production#16187 (closed)).
This issue's goal is to re-implement the semaphore. The new implementation should eliminate the need for spawning a goroutine. sync.Weighted is a good reference. Although we cannot use that data structure directly, its inner workings can inspire this new implementation.
The prior incident could have been avoided by a feature flag. The prior release wasn't supposed to change any functionality. The limiter was still static. However, we must be extra careful this time. The new semaphore must be controlled by a feature flag, even though the limiter is static.