Backport of 'Skip conflicting inserts for finding identifiers' 19.4
What does this MR do and why?
Backport of !256534 (merged)
Makes Security::Ingestion::Tasks::IngestFindingIdentifiers use ON CONFLICT DO NOTHING instead of DO UPDATE.
vulnerability_occurrence_identifiers holds only (occurrence_id, identifier_id). Its unique key covers the whole row, so a conflicting insert has nothing to update — but DO UPDATE rewrote the row anyway, taking an exclusive row lock held until the slice transaction committed.
Measured over one hour on patroni-sec:
| Value | |
|---|---|
| Updates per hour | 2,132,063 |
| Inserts per hour | 199,259 |
| Updates per insert | 10.7 |
| Index entries written per hour by the non-HOT share | 3,049,396 |
Every one of those 2.1M updates is a no-op by construction. That is ~592 wasted row rewrites per second, each one also taking a lock on a table that is under lock contention.
How it is implemented
Rather than special-casing this task, Gitlab::Ingestion::BulkInsertableTask gains an on_conflict setting so any task whose unique key covers every column worth writing can opt in:
self.unique_by = %i[occurrence_id identifier_id].freeze
self.on_conflict = :nothingThe DO NOTHING path now also passes unique_by through to bulk_insert!. Without it Postgres would emit a bare ON CONFLICT DO NOTHING with no conflict target, which skips on any constraint violation rather than the intended key. Existing callers of that path pass no unique_by, so nothing changes for them.
The task sets no self.uses, so nothing consumed RETURNING and no caller depended on the update.
No feature flag: the table has no mutable payload, so there is no behavioural difference beyond not rewriting rows. The rollback is a revert.
References
- Closes #629989 (closed)
- RCA, including the measurements above: #629530
- Parent epic: &23596
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
- This MR is backporting a bug fix, documentation update, or spec fix, previously merged in the default branch.
- The MR that fixed the bug on the default branch has been deployed to GitLab.com (not applicable for documentation or spec changes).
- The MR title is descriptive (e.g. "Backport of 'title of default branch MR'"). This is important, since the title will be copied to the patch blog post.
- Required labels have been applied to this merge request
- severity label and bug subtype labels (if applicable)
- If this MR fixes a bug that affects customers, the customer label has been applied.
- This MR has been approved by a maintainer (only one approval is required).
- Ensure the
E2E Omnibus GitLab EEpipeline (triggered by thee2e:test-on-omnibus-eejob) has succeeded. If it has failed (or shows warnings), investigate the failures. If you determine the failures are unrelated, you may proceed. If you need assistance investigating, request help in the #s_developer_experience Slack channel to confirm the failures are unrelated to the merge request.
Note to the merge request author and maintainer
If you have questions about the patch release process, please:
- Refer to the patch release runbook for engineers and maintainers for guidance.
- Ask questions on the
#releasesSlack channel (internal only). - Once the backport has been merged, the commit changes will be automatically deployed to a release environment that can be used for manual validation. See after merging runbook for details.