Backport of 'Resume the CVE enrichment sync across export files' to 19.3
What does this MR do and why?
Backport of !258833 (merged) to 19-3-stable-ee.
The CVE enrichment sync stores no checkpoint, so when the export spans more than one file (today it is one), each hourly run starts again at the first and stops at its time budget, and CVEs in later files are never imported. This change gives the sync a pm_checkpoints row, so each run resumes after the last finished file, and makes the GCP connector list v2/cve_enrichment/ so each file parses as its own chunk.
Please merge this only after !260538 (closed), the 19.3 backport of !253401 (merged): the new syncing a CVE enrichment export across runs examples use the verbatim mirror layout and fail without it.
Besides the migration milestone, three files differ from master's change, because 19-3 keeps the malware marker code that !250924 (merged) removed (19-3's specs still cover it):
sync_service.rb: master's CVE marker line moves into 19-3'scheckpoint_position, and 19-3'sfinalize_full_sync, a no-op on a nil marker, replaces master'sfinish_full_sync.checkpoint.rb: thefirst_sync?comment keeps 19-3's first sentence before master's new one.sync_service_spec.rb: the 19-3 malware examplewhen interrupted mid datasetgets thestop?stub master gives the generic stop example.
Database migrations
The four regular migrations keep master's versions, so an upgrade to 19.5 skips them. A db:check-migrations job run after 17:00 UTC on 18 October fails: it rejects added migrations over 21 days old. Their milestone is '19.3', not master's '19.5', following review on the backport in !253669 (merged). Please merge this with or after !260674 (merged), the 19.4 backport, in the same patch release, so a 19.3 instance with these migrations can upgrade to a 19.4 patch with the same schema. Database reviewer, can you confirm '19.3' is right?
Testing
In an RSpec run, the migration spec and the package metadata model, service, worker and library specs pass on 19-3-stable-ee with both backports applied. The db:migrate tasks ran the migrations up, down and up again, and the migrated schema matches db/structure.sql.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
- This MR is backporting a bug fix, documentation update, or spec fix, previously merged in the default branch.
- The MR that fixed the bug on the default branch has been deployed to GitLab.com (not applicable for documentation or spec changes).
- The MR title is descriptive (e.g. "Backport of 'title of default branch MR'"). This is important, since the title will be copied to the patch blog post.
- Required labels have been applied to this merge request
- severity label and bug subtype labels (if applicable)
- If this MR fixes a bug that affects customers, the customer label has been applied.
- This MR has been approved by a maintainer (only one approval is required).
- Ensure the
E2E Omnibus GitLab EEpipeline (triggered by thee2e:test-on-omnibus-eejob) has succeeded. If it has failed (or shows warnings), investigate the failures. If you determine the failures are unrelated, you may proceed. If you need assistance investigating, request help in the #s_developer_experience Slack channel to confirm the failures are unrelated to the merge request.
Note to the merge request author and maintainer
If you have questions about the patch release process, please:
- Refer to the patch release runbook for engineers and maintainers for guidance.
- Ask questions on the
#releasesSlack channel (internal only). - Once the backport has been merged, the commit changes will be automatically deployed to a release environment that can be used for manual validation. See after merging runbook for details.