fix: Publish the Pulp repository if the uploaded RPM is not published
What does this MR do?
With PULP_CLI_PUBLISH=false, RPM uploads use --no-publish, and Pulp publishes the repository inside the upload task (autopublish). If the Pulp worker dies after the new repository version is saved, that version gets no publication. A retry finds the package already in the repository, so Pulp makes no new version and does not publish. The job passes, but the package is never served. This happened on 2026-11-11 for 4 gitlab-pre-release-* repositories during 19.4.7.
After each RPM upload with --no-publish, this MR:
- Finds the newest publication of the repository:
pulp rpm publication list --repository <repo> --ordering -pulp_created --limit 1 - Checks if the repository version of that publication contains the package, by the SHA256 of the file:
pulp rpm content -t package list --sha256 <sha256> --repository-version <repository_version> --limit 1 - Publishes the repository if the package is not in that version, if the repository has no publication, or if the check fails:
pulp --format none rpm publication create --repository <repo>
| Case | Publish? |
|---|---|
Normal upload (autopublish finished inside the upload task) |
No, only 2 read calls |
| Another job saved a newer version, and its publish still runs | No. An incomplete publication is not listed, so the newest publication is ours. |
| Another job's publish has finished | No. Its repository version also has our package. |
| Retry after a worker crash | Yes |
| The repository has no publication, the output is not JSON, or a check command times out | Yes, with a warning |
| The publish command fails | PackageUploadError, so Retriable retries |
Default or PULP_CLI_PUBLISH=true, and deb |
No check, no extra calls |
The check does not use pulp rpm publication list --repository <repo> --content '["<href>"]'. In pulpcore, PublicationQuerySet#with_content builds a new query from Publication.objects instead of self, so the content filter drops the repository filter. The list then has publications of all repositories that contain the package. The el and ol uploads share one package, so a manual test on the pre Pulp showed that the ol-9 check found the el-9 publication and did not publish ol-9. A regression spec covers this case.
The SHA256 uses OpenSSL::Digest::SHA256, because the Fips/OpenSSL cop requires it. All options are available in pulp-cli 0.35.1, the version in the builder images. The description of PULP_CLI_PUBLISH in doc/development/ci-variables.md explains the check. The specs unset PULP_CLI_PUBLISH by default, because the project CI/CD variables set it to false for the nightlies.
After this MR and its backports merge, PULP_CLI_PUBLISH on dev.gitlab.org goes back to false. It is true now as a short-term protection.
Test
Manual test on the pre Pulp (packages.pre.gitlab.net). I ran the upload code locally with PULP_CLI_PUBLISH=false against gitlab-pre-release-el-9-x86_64 and gitlab-pre-release-ol-9-x86_64. The test package was a small el9 RPM. To simulate a worker crash, I deleted the publications of the latest version, so that version had the package but no publication.
| Case | Result |
|---|---|
Default path (PULP_CLI_PUBLISH unset) |
|
Normal upload with --no-publish |
|
Worker crash, only ol-9 |
ol-9 is published |
| Worker crash, both repositories |
The first version of the check used pulp rpm publication list --repository <repo> --content '["<href>"]'. The test showed that pulpcore drops the repository filter when content is set (PublicationQuerySet#with_content builds a new query). So the ol-9 check found the el-9 publication and did not publish ol-9. c7df1028 fixes this: the check now uses the newest publication of the repository and its repository version. A regression spec covers the case.
The test package was removed from both repositories after the test.
Related issues
Related to gitlab-com/gl-infra/delivery#22556
Checklist
See Definition of done.
For anything in this list which will not be completed, please provide a reason in the MR discussion.
Required
- MR title and description are up to date, accurate, and descriptive.
- MR targeting the appropriate branch.
- Latest Merge Result pipeline is green.
- When ready for review, MR is marked as ready (not a draft) and a reviewer is requested. Pick the reviewer suggested by the reviewer roulette, or one of the project maintainers.
For GitLab team members
If you don't have access to this, the reviewer should trigger these jobs for you during the review process.
- The manual
Trigger:ee-packagejobs have a green pipeline running against latest commit.- To debug QA failures, refer to the Investigate QA failures section.
- If
config/softwareorconfig/patchesdirectories are changed, make sure thebuild-package-on-all-osjob within theTrigger:ee-packagedownstream pipeline succeeded. - If you are changing anything SSL related, then the
Trigger:package:fipsmanual job within theTrigger:ee-packagedownstream pipeline must succeed. - If CI configuration is changed, the branch must be pushed to
dev.gitlab.orgto confirm regular branch builds aren't broken.
Expected (please provide an explanation if not completing)
- Test plan indicating conditions for success has been posted and passes.
- Documentation created/updated.
- Tests added.
- Integration tests added to GitLab QA.
- Equivalent MR/issue for the GitLab Chart opened.
- Validate potential values for new configuration settings. Formats such as integer
10, duration10s, URIscheme://user:passwd@host:portmay require quotation or other special handling when rendered in a template and written to a configuration file.
Reviewer checklist
- As soon as you begin reviewing, select Start a review on your first comment so the MR shows Reviewer started review.