Container Registry metadata database import job
The following discussion from !3752 (merged) should be addressed:
-
@Alexand started a discussion: (+1 comment) non-blocking thought
Users have reported step one import completed at rates of 2 to 4 TB per hour. At the slower speed, registries with over 100TB of data could take longer than 48 hours.
If this can take that long, running it with
kubectl exec
will be prone to network failures, Node failures, autoscaler scaling up/down, etc. In general, pods are ephemeral resources.I don't mind starting with documenting this manual approach. But we should discuss moving this to a Kubernetes Job. Kubernetes jobs will take care of retrying failed pods. If this task can be parallelized, one could also configure a Job to start multiple processes to do this at the same time and provide a quicker result.
I'll leave this thread open in case the maintainer has any thoughts, but we could probably consider this in a follow-up.
Sample job #5292 (comment 1874368684)