Failing to sync rpm gitlab/gitlab-ce repo with Pulp
Status update 2022-11-08:
We've documented a workaround here https://docs.gitlab.com/omnibus/troubleshooting.html#mirroring-the-gitlab-yum-repository-with-pulp-or-red-hat-satellite-fails
And have also logged https://github.com/rpm-software-management/createrepo_c/issues/329 upstream that would get us closer to a fix in rpm mirroring.
We are also continuing our own investigation to see if we can make changes that allow us to turn filelist back on in gitlab-org/distribution/team-tasks#1145 (closed)
Status update 2022-07-29:
We utilize packagecloud for our package distribution for serving our repos, and this is actually an intentional choice which causing this failure. This documentation from packagecloud shows the exact feature we are disabling - https://packagecloud.io/docs#disable_filelists
The two situations we are running into with our packagecloud repo which have required us to disable filelist metadata are as follows:
- The filelist for our gitlab-ee repo is several hundred MB, and undesirable for users to download during metadata sync operations #6971
- The db table used to generate the filelist in packagecloud is 100s of Gigs large, and causing our new package uploads hours to index during a release.
As a first step, in Distribution, we will schedule some research issues for workarounds related to Pulp. And following that we will continue to look at resolving the underlying issues with the repository that prevent us from having these filelists.
Overview
When i'm trying to sync gitlab rpm repository with Pulp and i'm getting error Malformed repository: metadata is specified for different set of packages in filelists.xml and in other.xml
. At the same time downloading metadata with yum occurs without problem.
Here is the full log of failed syncronization
Sep 8 16:01:01 localhost systemd: Started Session 5 of user root.
Sep 8 16:01:01 localhost systemd: Starting Session 5 of user root.
Sep 8 16:01:49 localhost pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[1f4aa43a-d0c4-47c2-9162-84ac28fca8e1]
Sep 8 16:01:49 localhost pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[3e5f9b05-cebe-4956-a4ba-e91fb644e090]
Sep 8 16:01:49 localhost pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[fece44e1-ea7c-490c-ad29-f65403a7aae5]
Sep 8 16:01:49 localhost pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._queue_reserved_task[1f4aa43a-d0c4-47c2-9162-84ac28fca8e1] succeeded in 0.0405119440002s: None
Sep 8 16:01:49 localhost pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [3e5f9b05] Downloading metadata from http://packages.gitlab.com/gitlab/gitlab-ce/el/7/x86_64/.
Sep 8 16:01:49 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): packages.gitlab.com
Sep 8 16:01:53 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): packages.gitlab.com
Sep 8 16:01:58 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): packages-gitlab-com.s3-accelerate.amazonaws.com
Sep 8 16:02:02 localhost pulp: nectar.downloaders.threaded:INFO: Download succeeded: http://packages.gitlab.com/gitlab/gitlab-ce/el/7/x86_64/repodata/repomd.xml.
Sep 8 16:02:03 localhost pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [3e5f9b05] Parsing metadata.
Sep 8 16:02:03 localhost pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [3e5f9b05] Downloading metadata from http://packages.gitlab.com/gitlab/gitlab-ce/el/7/x86_64/.
Sep 8 16:02:03 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): packages.gitlab.com
Sep 8 16:02:07 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): packages.gitlab.com
Sep 8 16:02:12 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): packages-gitlab-com.s3-accelerate.amazonaws.com
Sep 8 16:02:16 localhost pulp: nectar.downloaders.threaded:INFO: Download succeeded: http://packages.gitlab.com/gitlab/gitlab-ce/el/7/x86_64/repodata/repomd.xml.
Sep 8 16:02:17 localhost pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [3e5f9b05] Parsing metadata.
Sep 8 16:02:17 localhost pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [3e5f9b05] Downloading metadata from http://packages.gitlab.com/gitlab/gitlab-ce/el/7/x86_64/.
Sep 8 16:02:17 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): packages.gitlab.com
Sep 8 16:02:21 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): packages.gitlab.com
Sep 8 16:02:26 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): packages-gitlab-com.s3-accelerate.amazonaws.com
Sep 8 16:02:31 localhost pulp: nectar.downloaders.threaded:INFO: Download succeeded: http://packages.gitlab.com/gitlab/gitlab-ce/el/7/x86_64/repodata/repomd.xml.
Sep 8 16:02:31 localhost pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [3e5f9b05] Parsing metadata.
Sep 8 16:02:31 localhost pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [3e5f9b05] Downloading metadata from http://packages.gitlab.com/gitlab/gitlab-ce/el/7/x86_64/.
Sep 8 16:02:31 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): packages.gitlab.com
Sep 8 16:02:35 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): packages.gitlab.com
Sep 8 16:02:40 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): packages-gitlab-com.s3-accelerate.amazonaws.com
Sep 8 16:02:45 localhost pulp: nectar.downloaders.threaded:INFO: Download succeeded: http://packages.gitlab.com/gitlab/gitlab-ce/el/7/x86_64/repodata/repomd.xml.
Sep 8 16:02:45 localhost pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [3e5f9b05] Parsing metadata.
Sep 8 16:02:45 localhost pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [3e5f9b05] Downloading metadata files.
Sep 8 16:02:45 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (2): packages.gitlab.com
Sep 8 16:02:45 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (3): packages.gitlab.com
Sep 8 16:02:46 localhost pulp: nectar.downloaders.threaded:INFO: Download succeeded: http://packages.gitlab.com/gitlab/gitlab-ce/el/7/x86_64/repodata/af89eefe50453876ce8ee32ce6075bb344b64910-filelists.xml.gz.
Sep 8 16:02:49 localhost pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): packages.gitlab.com
Sep 8 16:02:50 localhost pulp: nectar.downloaders.threaded:INFO: Download succeeded: http://packages.gitlab.com/gitlab/gitlab-ce/el/7/x86_64/repodata/da8335dcde4cd2844b082f25a461e6870c457662-other.xml.gz.
Sep 8 16:02:55 localhost pulp: nectar.downloaders.threaded:INFO: Download succeeded: http://packages.gitlab.com/gitlab/gitlab-ce/el/7/x86_64/repodata/f081779cb84658c5089d36c81c3d02eaad94fe9d-primary.xml.gz.
Sep 8 16:02:55 localhost pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [3e5f9b05] Generating metadata databases.
Sep 8 16:02:55 localhost pulp: pulp.server.async.tasks:INFO: [3e5f9b05] Task failed : [3e5f9b05-cebe-4956-a4ba-e91fb644e090] : Malformed repository: metadata is specified for different set of packages in filelists.xml and in other.xml
Sep 8 16:02:55 localhost pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.sync.sync[3e5f9b05-cebe-4956-a4ba-e91fb644e090] raised expected: PulpCodedException()
Sep 8 16:02:55 localhost pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[fece44e1-ea7c-490c-ad29-f65403a7aae5] succeeded in 0.00701914000001s: None
This issue was discussed in the bug tracker of pulp here https://pulp.plan.io/issues/1287 and the conclusion was that the problem is in malformed metadata of repository.
Can you fix metadata of your RPM repository?