<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/engineering/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Work on this issue](https://contributors.gitlab.com/manage-issue?action=work&projectId=278964&issueIid=631098) - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=631098) </details> <!--IssueSummary end--> ### Summary Since approximately 2026-09-25, group-level vulnerability report exports (CSV) on GitLab.com fail during segment generation. `Gitlab::Export::SegmentedExportWorker` raises `ActiveRecord::RecordNotFound: Couldn't find Vulnerabilities::Read with 'vulnerability_id'=<id>` in `VulnerabilityExports::ExportService#export_segment`. The failing part retries repeatedly with the same error, the export is marked failed, and the user never receives the "export ready" email. The export request itself succeeds (`201`) and `VulnerabilityExports::ExportWorker` completes normally, so the user sees the "Report export in progress" banner but never receives the email. Suspected cause: commit b87a93e15b448d908497fe7c227c164f4e5fa12d ("Speed up filtered vulnerability CSV export enqueue"). For **group** exports it stores `vulnerability_reads.id` as the part boundaries, but `export_segment` still looks them up with `Vulnerabilities::Read.find`, whose primary key is `vulnerability_id`. Project-level exports do not appear to be affected. Customer report: ZD [755872](https://gitlab.zendesk.com/agent/tickets/755872) (Ultimate, GitLab.com). ### Steps to reproduce 1. On GitLab.com, choose a group whose vulnerabilities have `vulnerability_reads.id` values that differ from their `vulnerability_id` (true of nearly all real data). 2. Request a group vulnerability export, either from **Secure > Vulnerability report > Export** or via `POST /api/v4/security/groups/:id/vulnerability_exports`. 3. The request returns `201` and `VulnerabilityExports::ExportWorker` finishes with `done` and logs "Queued segmented export". 4. One or more `Gitlab::Export::SegmentedExportWorker` jobs fail with `ActiveRecord::RecordNotFound` and keep failing on retry. 5. `Gitlab::Export::SegmentedExportFinalisationWorker` runs but the export never completes, and no `ActionMailer::MailDeliveryJob` is sent to the requester. ### Example Project N/A. Seen on customer groups on GitLab.com; details are in the linked ZD ticket. It should reproduce on any group large enough that `vulnerability_reads.id` and `vulnerability_id` diverge. ### What is the current *bug* behavior? - Group exports never finish and the export email is never sent. - Failed parts retry many times (`retry_count` reached 10 in our logs), always raising the same exception for the same boundary ID. - `export_segment` rescues the error, marks the export failed and schedules its deletion. - Some group or subgroup exports still complete, possibly when a stored `vulnerability_reads.id` happens to match an existing `vulnerability_id`. If so, the segment range is built from the wrong record, and the resulting CSV **may be missing or duplicating rows**. This is unverified and needs engineering confirmation. GitLab.com-wide impact from Kibana (`pubsub-sidekiq-inf-gprd`, `json.class = Gitlab::Export::SegmentedExportWorker`), counts of log events per day: | Day (UTC) | start | done | fail | |---|---|---|---| | 2026-09-22 | 1,601 | 1,482 | 58 | | 2026-09-23 | 876 | 814 | 25 | | 2026-09-24 | 813 | 768 | 28 | | 2026-09-25 | 5,281 | 574 | **4,544** | | 2026-09-26 | 8,173 | 305 | **7,737** | | 2026-09-27 | 7,386 | 401 | **6,792** | | 2026-09-28 (partial) | 2,536 | 220 | **2,216** | The failures are spread across many root namespaces (well over 10 on each day since 2026-09-25). ### What is the expected *correct* behavior? All export parts are generated using the correct boundary records, the finalisation worker combines them into the CSV, the export is marked finished, and the requester receives the download email, as happened before 2026-09-25. ### Relevant logs and/or screenshots Failing part (Sidekiq, one of 96 failures for the affected customer on 2026-09-28): ```json { "class": "Gitlab::Export::SegmentedExportWorker", "job_status": "fail", "retry_count": 10, "exception.class": "ActiveRecord::RecordNotFound", "exception.message": "Couldn't find Vulnerabilities::Read with 'vulnerability_id'=421285435", "exception.backtrace": [ "activerecord (7.2.3.1) lib/active_record/core.rb:268:in `find'", "ee/app/services/vulnerability_exports/export_service.rb:43:in `block in export_segment'", "ee/app/services/vulnerability_exports/export_service.rb:209:in `default_iterator'", "ee/app/services/vulnerability_exports/export_service.rb:40:in `export_segment'", "ee/app/services/gitlab/segmented_export/segment_exporter_service.rb:17:in `block in execute'", "ee/app/workers/gitlab/export/segmented_export_worker.rb:16:in `perform'" ] } ``` For that customer, all 96 failures cite just 3 boundary IDs: `421285435` (78), `406710850` (9) and `425847384` (9). Sequence for one export (correlation ID `a41de3b2ab3153e7-SYD`): ``` 22:19:21 VulnerabilityExports::ExportWorker start 22:19:27 Gitlab::Export::SegmentedExportWorker (x2) start 22:19:27 "Export part generation failed" <- JID 61ccdbebc4dec86a57c3b902 22:19:27 Gitlab::Export::SegmentedExportWorker done "Export part uploaded" 22:19:27 "Queued segmented export" 22:19:27 VulnerabilityExports::ExportWorker done (5.19s) 22:19:28 SegmentedExportFinalisationWorker deduplicated 22:19:51 SegmentedExportFinalisationWorker done (0.18s) 22:20:18 … 23:42:30 JID 61ccdbebc4dec86a57c3b902 retried, fails each time with the same RecordNotFound (no ActionMailer::MailDeliveryJob for the requester) ``` "Cannot obtain an exclusive lease. There must be another instance already in execution." is logged next to both successful and failed parts, so it does not appear to be the cause. Kibana query used for the per-day counts: ``` GET /pubsub-sidekiq-inf-gprd/_search { "size": 0, "query": { "bool": { "filter": [ { "range": { "@timestamp": { "gte": "2026-09-22T00:00:00Z", "lte": "2026-09-28T07:00:00Z" } } }, { "term": { "json.class.keyword": "Gitlab::Export::SegmentedExportWorker" } } ] } }, "aggs": { "per_day": { "date_histogram": { "field": "@timestamp", "calendar_interval": "day" }, "aggs": { "status": { "terms": { "field": "json.job_status.keyword" } } } } } } ``` ### Output of checks This bug happens on GitLab.com #### Results of GitLab environment info <details> <summary>Expand for output related to GitLab environment info</summary> <pre> GitLab.com (SaaS), 19.5.0 at time of report. Not applicable (not self-managed). </pre> </details> #### Results of GitLab application Check <details> <summary>Expand for output related to the GitLab application check</summary> <pre> Not applicable (GitLab.com). </pre> </details> ### Possible fixes In `ee/app/services/vulnerability_exports/export_service.rb`: `enqueue_segmented_export` saves part boundaries using `boundary_column`, which is `:id` for groups: ```ruby start_id: records.first[boundary_column], end_id: records.last[boundary_column], def boundary_column exportable.is_a?(Group) ? :id : :vulnerability_id end ``` `export_segment` looks up group boundaries with `find`, which searches the primary key (`vulnerability_id` for `Vulnerabilities::Read`): ```ruby first_record = Vulnerabilities::Read.find(part.start_id) last_record = Vulnerabilities::Read.find(part.end_id) ``` Before b87a93e15b448d908497fe7c227c164f4e5fa12d, parts stored `records.first.id`. On `Vulnerabilities::Read` that is the primary key value, which is `vulnerability_id`, so the lookup matched. `records.first[:id]` returns the raw `id` column instead, a different sequence. Options: 1. Store the primary key for groups as well, for example `boundary_column` returning `:vulnerability_id` in both cases (or use `records.first.id` for groups). This restores the previous behavior. 2. Or keep storing `id` and look it up with `Vulnerabilities::Read.find_by!(id: part.start_id)` / `find_by!(id: part.end_id)`. Follow-ups: - Exports already queued with bad parts will keep failing. Consider cleaning up or re-queuing affected exports. - The spec updated in 4137103f6480c849a19e0cb4ce1d1786ce258c67 now expects `:id` as the group boundary, matching the new behavior. Test data likely has `vulnerability_reads.id == vulnerability_id`, which would hide the mismatch. Add a spec where the two differ. - Confirm whether group exports that completed since the deploy produced correct output, given the possible wrong-range lookup described above. Related commits: b87a93e15b448d908497fe7c227c164f4e5fa12d, 4137103f6480c849a19e0cb4ce1d1786ce258c67. ### Patch release information for backports If the bug fix needs to be backported in a [patch release](https://handbook.gitlab.com/handbook/engineering/releases/patch-releases) to a version under [the maintenance policy](https://docs.gitlab.com/policy/maintenance/), please follow the steps on the [patch release runbook for GitLab engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/patch/engineers.md). Refer to the [internal "Release Information" dashboard](https://dashboards.gitlab.net/d/delivery-release_info/delivery3a-release-information?orgId=1) for information about the next patch release, including the targeted versions, expected release date, and current status. #### High-severity bug remediation To remediate high-severity issues requiring an [internal release](https://handbook.gitlab.com/handbook/engineering/releases/internal-releases/) for single-tenant SaaS instances, refer to the [internal release process for engineers](https://gitlab.com/gitlab-org/release/docs/-/blob/master/general/internal-releases/engineers.md?ref_type=heads).