Backport of 'Resume mid-delta on the package metadata label path' into 19-3-stable-ee

What does this MR do and why?

Backport of Resume mid-delta on the package metadata label ... (!255559 - merged) • Orin Naaman • 19.5 ("Resume mid-delta on the package metadata label path"), merged to master in 19.5.

  • The GCP bucket-label shortcut in ee/lib/gitlab/package_metadata/connector/gcp.rb decided a sync was caught up from the export sequence number alone, ignoring the chunk number.
  • A sync interrupted partway through a multi-file export therefore looked finished and never read the remaining files. filtered_entries dropped the partially-read export entirely, so both had to change together.
  • Package metadata v2 exports are incremental deltas and are never republished, so a skipped window is lost. The instance keeps serving incomplete license data with no error and no alert.
  • Observed case: npm's checkpoint sat at chunk 4 of an 11-file export, so six files were never read, and the sync reported success every 5 minutes for six days.
  • Introduced in 19.0 by 57fcdcfb, so every maintained version is affected. Bug report: PackageMetadata v2 sync skips unread chunks whe... (#628038) • Orin Naaman • 19.5
  • Clean cherry-pick of ec40ad2f. No conflicts and no manual changes.

Note

On this branch the affected path is still behind the package_metadata_sync_using_labels feature flag, but that flag is default_enabled: true, so instances hit the bug by default.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

  • This MR is backporting a bug fix, documentation update, or spec fix, previously merged in the default branch.
  • The MR that fixed the bug on the default branch has been deployed to GitLab.com (not applicable for documentation or spec changes).
  • The MR title is descriptive (e.g. "Backport of 'title of default branch MR'"). This is important, since the title will be copied to the patch blog post.
  • Required labels have been applied to this merge request
  • This MR has been approved by a maintainer (only one approval is required).
  • Ensure the e2e:test-on-omnibus-ee job has succeeded, or if it has failed, investigate the failures. If you determine the failures are unrelated, you may proceed. If you need assistance investigating, request help in the #s_developer_experience Slack channel to confirm the failures are unrelated to the merge request.

Note to the merge request author and maintainer

If you have questions about the patch release process, please:

Edited by Orin Naaman

Merge request reports

Loading
Loading