Simplifiy PipelineCreations::Requests Redis data structure

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Work on this issue
• Close this issue

Implement simplification proposed in !174577 (comment 2253031186). Specifically:

Having said that and in light of this new behavior, I'm wondering whether we could refactor the way we treat the MRs requests in Redis to use the "generic" requests-by-ID of REQUEST_REDIS_KEY as SSoT.

When creating a request for an MR, it would be like:

• Store the generic key-value: "pipeline_creation:projects:{123}:request:{the-request-id}" => { 'status' => 'in_progress', 'pipeline_id' => nil, 'error' => nil }
• We treat the MR key-value store not as a Hash but Array: instead of hset use lpush to push new values to the array.
• Then store also the request id above into the merge requests key-value store: `pipeline_creation:projects:{123}:mrs:{22334} => ["the-request-id"]

This mechanism would allow us to use the basic request key as SSoT for tracking the state of the request and we model the "state for the MR" as a list of requests.

The advantage of this design is that even for merge requests, the frontend could fetch the status directly by having the request ID. All pipeline requests are stored consistently regardless whether they are from MRs, API, etc.

The downside of this is however that in order to know which requests are completed for a merge request we would need to fetch all the requests for a given MR and check each of them using the related request key. This however should not be a big list.