Review Request: maven virtual registry, multiple usptreams support
ℹ️ Scaling Request
The feature/improvement we'd like some assistance with is: Multiple upstream support in the maven virtual registry.
The epic and relevant issues are:
- Maven virtual registry MVC (API only interactions) (gitlab-org&14137 - closed) • David Fernandez, Moaz Khalifa • 18.1
- Investigate: How to add support for multiple up... (gitlab-org/gitlab#516087 - closed) • David Fernandez • 17.10 • At risk
The reason we're asking for a scaling review on this item is:
The multiple upstream support can be simply described by the following: given an ordered (limited) set of urls, we need to HEAD each one and find the first one (respecting the order) that is successful.
We did an first analysis here with different ideas. Given that the virtual registry works under pretty strong constraints (everything is inlined in a web request = we can't use background jobs), there is a need for an efficient support of multiple upstreams.
The ask here is to review the investigation to find the proper implementation approach. We might have missed some alternative ideas available to implement this.
The maven virtual registry is still in its first iteration behind a wip feature flag. Now is a good time to stop and properly think how to support these multiple upstreams. This is not specific to the Maven package format, we will re-use the same logic for package formats that behave in a similar way as Maven.
In particular, we are concerned about:
-
Memory -
Migrations -
N+1 -
Queueing -
Design implementation -
Execution time
We're hoping to release this as part of milestone: we're still a few milestones away to have everything ready for the maven virtual registry, thus I would say %18.0 or beyond.
🔮 Follow ups
We're currently focusing hard on the Maven package format. Next in line is NPM. For NPM, we will have an additional constraint: we need to get (or stream) the response body from the upstream and modify it on the fly while returning it to the client. This within the multiple upstream support.
Thus, the solution selected for the Maven virtual registry should be compatible or at least provide leads to solve that technical challenge that we have for NPM.