Allows administrators to configure a maximum number of simultaneous
exports (default 25) to prevent exhausting resources.

Changelog: added

Changelog: other

The import source hostname is used as one of the scopes for subsequent
imports to use an existing `Import::SourceUser` record
(through `Import::SourceUserMapper`).

Capturing the port and scheme in the hostname allows us to make this
scope a bit tighter.

https://gitlab.com/gitlab-com/gl-security/product-security/appsec/appsec-team/-/issues/845#note_2135545803

Introduce the class Import::BulkImports::EphemeralData to allow the
feature flag bulk_import_importer_user_mapping to be enabled or
disabled without interfering with in-flight migrations. Also, refactored
the Import::SourceUserMapper class to return an Import::SourceUser.

Instead of waiting for all import jobs, only wait for user related
jobs to finish before starting user mapping.

Changelog: performance

James Nutt authored 7 months ago and GitLab Release Tools Bot committed 7 months ago

Merge branch 'security/limit-access-to-exports' into 'master'

See merge request gitlab-org/security/gitlab!4080

Changelog: security

Cache any user reference columns in a set so that they
can be pulled after all exports have finished in a dedicated
relation. This prevents the need for ugly queries or out of
pattern exports.

Changelog: added

Changelog: other

Instead of using just the `id`, we now use `full_path` which is
provided by the backend.

We also add a fallback in case the project / group failed to be created.

Changelog: changed

Direct Transfer API shows failures for the things that we could not
import. If a subrelation of a top level relation failed to import
(e.g. MR diff note) - we still show top level relation as the one
that failed to import. In reality, the top level relation is imported,
but a subrelation did not.

Update relation to include the failed to import subrelation, so it's
more obvious what exactly failed to import.

Changelog: added

* Previously, a NoMethodError was raised when status was nil
* Now, we handle the nil case and return 0 for total object count
* Fixes #439819

Changelog: fixed

Update Direct Transfer API entities endpoint to include
new `stats` key to show statistics about what relations
are imported. It has 3 counters:
  - `source` - how many records the source instance states there are
  - `fetched` - how many records we read from the downloaded export
  - `imported` - how many records we persisted in the database

Changelog: added

Fix exception during BulkImports::Failure creation when
provided source title or url
is longer than 255 characters. Truncate the data instead.

Changelog: fixed

Bojan Marjanovic authored 1 year ago and Bojan Marjanovic committed 1 year ago

This feature flag was introduced in 16.2 and is now being removed.

Changelog: other

This limits the number of concurrent in-progress
`BulkImports::PipelineBatchWorker` workers that will run for a single
Direct Transfer migration.

This is a performance measure to avoid any one Direct Transfer migration
over-saturing resources.

https://gitlab.com/gitlab-org/gitlab/-/issues/429863

Also introducing scope for trackers

Changelog: performance

Max Fan authored 1 year ago and George Koltsov committed 1 year ago

Now imports that have been updated in the past 8 hours
are exempt from being killed by the stuckImportWorker

When a Direct Transfer entity fails and is marked as failed, this status
is not passed up to the parent import. This can result in a misleading
situation where an import is marked as `has_failures: false` when in
fact there are failures.

#432546

Changelog: fixed

Update BulkImports::ExportUploader to upload exported relation in
a after_commit callback in order to not idle in a db transaction.

Changelog: fixed

Max Fan authored 1 year ago and George Koltsov committed 1 year ago

Bulks imports will now start failing entity if the pipeline has aborts_on_failure enabled
Previously it only failed the trackers, not the bulk import itself

Changelog: fixed

Increase the period from 8 hours to 24 hours

Rodrigo Tomonari authored 1 year ago and Bojan Marjanovic committed 1 year ago

Update PipelineWorker and PipelineBatchWorker workers to use the
`defer_on_database_health_signal` option.

Also, update defer_on_database_health_signal to receive a block
to lazy evaluate the schema and table as the workers modify
different tables depending on the relation being imported

Rodrigo Tomonari authored 1 year ago and Luke Duncalfe committed 1 year ago

Add stderr to the exception message raised when gunzip or gzip
commands fail

Revert "Merge branch 'bmarjanovic/feature-flag-rollout-bulk_imports_batched_import_export' into 'master'"

This reverts merge request !132537

Bojan Marjanovic authored 1 year ago and Stan Hu committed 1 year ago

This feature flag was introduced in 16.2 and is now being removed.

Changelog: other

Rodrigo Tomonari authored 1 year ago and Bojan Marjanovic committed 1 year ago

Updates EntityWorker to enqueue itself instead of being called by
pipeline workers when they finish to prevent multiple EntityWorkers
to be executed

Changelog: fixed

James Nutt authored 1 year ago and Rodrigo Tomonari committed 1 year ago

This MR is a pre-cursor to extracting the BulkImportWorker code into a
separate service, allowing the Worker itself to focus primarily on
scheduling/retry concerns.

It implements the in-lining of the pipeline tracker creation as per this
code review comment:

<!57153 (comment 547953119)>

In 52d3a674 we deleted the actual
regexes from `Gitlab::Regex::BulkImports`, leaving just the validation
messages.

This change deletes the module as a follow-up clean up as it no longer
has regexes.

#424432

Allow BulkImports::BatchTracker state machine status to transit from
`started` to `started` so allow the PipelineBatchWorker worker to
multiple times

Changelog: fixed

Luke Duncalfe authored 1 year ago and GitLab Release Tools Bot committed 1 year ago

Merge branch 'security-415067-redos-in-bulk_imports-api' into 'master'

See merge request gitlab-org/security/gitlab!3464

Changelog: security

Import 1k records per batch when importing collection
relations using Direct Transfer if source instance
supports it.

Changelog: added

Fix Layout/ArgumentAlignment offenses in models

Rodrigo Tomonari authored 1 year ago and George Koltsov committed 1 year ago

Allows GitLab Direct Transfer of type project to be created via
API

Changelog: added