Add feature flag to allow maintainers to create MR pipelines async (!161407) · Merge requests · GitLab.org / GitLab

Dylan Griffith requested to merge 463355-async-pipeline-creation-for-merge-requests into master Aug 01, 2024

What does this MR do and why?

This is related to #463355 (closed) as this endpoint often times out. Allowing it to be created asynchronously will mean that we can create pipelines that take longer than 60s to create.

From my local testing this change has minimal UX impact on the merge requests pipeline page. That is because we are already polling for new pipelines constantly so having it created async shouldn't really matter to the user. I even tested the scenario where the pipeline config is broken and both provide a similar UX before and after. The UX differences are:

The spinner stops immediately because the async request returns immediately and the user has no visual indicator that it's still being created async
You might have to wait slightly longer for the polling interval to see the new pipeline when you create it async
If the pipeline has validation errors then the user won't get that feedback at all for async creation

Follow up UX improvements

I would like to introduce the UX improvements in a follow up issue #476627 (closed) because:

It's behind a user scoped feature flag and I don't intend to roll this out to customers yet
This provides immediate value to GitLab maintainers that understand the UX tradeoff because at present we often can't even create pipelines. Personally I see over 60% of my pipeline creation fails and I can't do my job
Making GitLab maintainers more productive ASAP means we'll be able to improve the product faster and fix the underlying causes (e.g. performance) faster
The async polling for the specific pipeline failure is quite a bit more complicated to implement as we need to store the information about recent pipeline attempts somewhere

There are more ideas in #463355 (closed) about how we might like to stream/poll from some endpoint to know if the specific pipeline I just tried to create failed or not. From a UX perspective this only seems relevant to timeouts today as it seems these are the ones where a user gets a flash notice of the failure. But the async process basically makes the timeouts go away so it is less relevant. In any case this is behind a feature flag and it will make me and other GitLab maintainers much more productive right now as we often cannot even start a pipeline and we're wasting a lot of resources trying and timing out constantly.

In future, if we need, we can update this API to return an identifier related to the async pipeline you created and then we can add a new endpoint to see whether or not your request succeeded. This could be done by streaming or polling and we could either use ActionCable subscriptions to notify the clients or we could store these recent jobs in redis with a short timeout if that's simpler than a whole new GraphQL endpoint.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.