## Summary The `AlmaLinux-8` staging upload job for `18.11.13+ce.0` failed because it exceeded the 60-minute job timeout. The same job for `AlmaLinux-10` in the same release passed in 29 minutes. The cause is lock contention on the `gitlab-pre-release-el-8-x86_64` repository in Pulp. Repository `autopublish` and job retries make it much worse. The cause is not the network, the CI job, the database or the number of Pulp workers. See [Root cause](#root-cause) and the analysis in the comments. - Failed: https://dev.gitlab.org/gitlab/omnibus-gitlab/-/jobs/39827538 - Passed: https://dev.gitlab.org/gitlab/omnibus-gitlab/-/jobs/39818863 ``` ERROR: Job failed: execution took longer than 1h0m0s seconds ``` ## Background - Each EL package is uploaded and published twice: once to the `el-N` repository and once to the `ol-N` repository (`PulpRepository#add_oracle_linux_platform`). One job therefore does two full upload and publish cycles. - The Pulp CLI runs with `--timeout 0`, so the CI job timeout is the only limit. - `.staging_upload_template` has `timeout: 60 minutes` on `18-11-stable`, `19-3-stable` and `19-4-stable`. gitlab-org/omnibus-gitlab!9813 increased it to 120 minutes on `master` and `19-5-stable` only. ## Comparison Both jobs use the same tag (`18.11.13+ce.0`, commit `f40012d5`), the same code and RPMs of about the same size (146 and 147 upload chunks). The phase times come from the Pulp task IDs (UUIDv7) and from the `pulp_created` and `pulp_last_updated` fields in the job logs. | Phase | Failed: el-8 (21:26 UTC) | Passed: el-10 (19:20 UTC) | Passed: ol-10 | |-------|--------------------------|---------------------------|---------------| | Chunk upload (client to Pulp) | 2m 48s | 4m 16s | 3m 14s | | Add-content task (wait + run) | 11m 53s (10m 44s wait + 1m 00s run) | 5m 32s | 7m 14s | | Publish task: wait before start | **10m 40s** | 1s | 1s | | Publish (metadata generation) | 11m 00s | 4m 45s | 3m 45s | | **Total per repository** | **\~36 min** | **\~14.5 min** | **\~14 min** | The failed job finished `el-8` in about 36 minutes. It was then stopped during the `ol-8` upload. About 21 of the 36 minutes were queue wait. The wait and run split for the failed job comes from `pulp task show`. The passed job's add-content times are not split. ## Observations 1. The network is not the cause. The chunk upload in the failed job was the fastest of the three. 2. The add-content and publish tasks of the failed job each waited about 10m 40s in the queue. The add-content task ran for only 1 minute. 3. The number of Pulp workers is not the cause. 15 workers are online, and at most 7 tasks on the `el-8` repository ran at the same time. 4. Publishes of the same repository that run together slow each other. One publish alone took about 6m 15s, and 7 publishes together took up to 11 minutes. The database is not the cause: CloudSQL CPU was at a maximum of 40%, and p99 query latency was at a maximum of 25 ms. 5. Publish time grows with repository size. Publishes that ran alone took 6.2 - 6.6 min on `el-8` (395 packages) and 3.7 - 4.7 min on `el-10` (253 packages). 6. The failed job is a retry. Its `el-8` add-content task made no change, because an earlier attempt had already added the package. On 2026-10-20 the AlmaLinux 8 and 9 jobs (CE and EE) timed out after 1 hour many times, and passed only after some retries. 7. `tasks_unblocked_queue{namespace="pulp"}` was 0 during the incident, so no task waited for a free worker. ## Root cause From 20:35 to 22:02 UTC, 28 tasks used `gitlab-pre-release-el-8-x86_64` (first analysis): | Task type | Lock | Count | Median wait | Max wait | Median run | Total run | |-----------|------|-------|-------------|----------|------------|-----------| | `general_create` (add content) | exclusive | 11 | 14.3 min | 34.6 min | 0.7 min | 15 min | | `publish` | shared | 17 | 11.7 min | 41.0 min | 7.9 min | 137 min | 1. Pulp starts tasks in order of creation. A task does not start before an older task that needs a conflicting lock. 2. An add-content task needs an exclusive lock on the repository. It waits until all running publishes end. 3. A publish needs a shared lock. Publishes that arrive after a waiting add-content task wait behind that task. 4. Many jobs that upload to the same repository at the same time therefore make a loop: a batch of publishes runs for about 8 minutes, then add-content tasks run one at a time, then the next batch of publishes runs. 5. The RPM repositories have `autopublish: true` (set by `gitlab-com/gl-infra/pulp-resources-automation`). Each new package therefore runs a full publish inside the add-content task, under the exclusive lock (7 - 8.5 min). The `pulp` CLI then starts a second publish, because `--publish` is the default. 6. A retried job uploads, locks and publishes the packages that the earlier attempt already added. It then continues to the next repository. From 20:00 to 22:35 UTC on `el-8`, 23 add-content tasks added only 5 new packages (18.11.13 FIPS and EE, 19.0.10 CE, FIPS and EE). The other 18 tasks (78%) were retries that made no change. 32 publishes ran for these 5 packages and used 246 minutes of worker time. Version 388 alone was published 11 times. The exclusive lock was held for 54 minutes in total. This is not a temporary problem with Pulp. It will happen again in each release that uploads many packages to the same repository at the same time, and each timeout and retry adds more load. ## Possible fixes | Fix | Where | Effect | Decision | |-----|-------|--------|----------| | Turn off `autopublish` on the RPM repositories | `pulp-resources-automation` | Removes the 7 - 8.5 minute publish under the exclusive lock | Not now. `autopublish` stays the default. | | `--no-publish` on each RPM upload | omnibus-gitlab | Removes the redundant CLI publishes. Pulp still publishes each new package inside the upload task. | **Do now:** #22495 | | Skip the upload if the package is already in the repository | omnibus-gitlab | A retry continues where the earlier attempt stopped | Not now. With `--no-publish`, a retry costs only about 1 minute of lock time. | | Backport the 120-minute timeout (gitlab-org/omnibus-gitlab!9813) | omnibus-gitlab stable branches | Short-term risk reduction | **Do now:** #22496 | `--no-publish` was tested on the Pulp pre environment (see the comments). With `autopublish: true`, Pulp still publishes inside the upload task, so each package is published by its own job. With `autopublish: false`, `--no-publish` leaves the package unpublished, and the job still passes. `PULP_CLI_PUBLISH=true` turns the CLI publish back on. gitlab-org/build/team-tasks#292 describes a similar problem for the gitlab-runner deb repositories. A publish pins its repository version when it is created, not when it starts. So skipping a publish when a newer publish of the same repository is waiting is also safe. ## Investigation tasks - [x] Confirm the queue wait: run `pulp task show --href /pulp/api/v3/tasks/01a15b42-6d8d-7748-a0b3-6c3051d0b760/` and compare `pulp_created` with `started_at`. - [x] Check the Pulp worker count and task queue for 2026-10-20, 21:00 - 22:30 UTC. Result: 15 workers, and lock contention on one repository. - [x] Check the Pulp database load during the publish batches. Result: CloudSQL CPU at most 40%, p99 latency at most 25 ms. - [x] Check the CPU usage and throttling of the Pulp worker pods during the publish batches. - [x] Check if publish time grows with the repository version count or size. Result: yes, about in proportion to the package count. - [x] Check if earlier attempts of the `AlmaLinux-8` staging upload job for `18.11.13+ce.0` also failed. Result: yes, AlmaLinux 8 and 9 (CE and EE) timed out many times. - [x] Decide if gitlab-org/omnibus-gitlab!9813 must be backported to `18-11-stable`, `19-3-stable` and `19-4-stable`. - [x] Confirm which jobs created the add-content tasks on `gitlab-pre-release-el-8-x86_64`. Result: 5 new packages and 18 retries, all from `omnibus-uploader`. - [x] Choose a long-term fix from [Possible fixes](#possible-fixes). Result: see Possible fixes.