GitLab Next

Updates verification state records using batches

We need to regularly re-verify data on the primary
to protect against data corruption. In combination with
https://gitlab.com/gitlab-org/gitlab-ee/issues/13842,
this will ensure that packages are regularly re-verified
on all secondary nodes as well.

In this commit we introduce the service and worker that
will allow us to move groups' repository storage in
bulk.

This classes will only be used through the API.

In this commit we introduce a worker that will handle
group repository storage through the group update service.

- Queue files and barebone jobs

In this commit we extract the existing logic of the
`GitGarbageCollectWorker` into a concern, in order to reuse it later
with the new wiki garbage collector that we're introding as well in
this commit.

This commit removes tech debts on the service.

Run it in StoreReportService and automatically
create auto-fix MRs

This worker will auto-approve blocked users when the setting
require_admin_approval_after_user_signup gets disabled. This setting is
enabled by default in new instances.

The `epic_tracking` category appears to have been added manually; the
other changed categories are currently unused.

On a monthly basis we create snapshots
of devops features current instance uses

Verification concurrency is managed by the verification_started_at time.
If a `VerificationBatchWorker` or `VerificationWorker` is lost or
killed, then one or more records would be stuck in the
"verification_started" state. This worker unsticks those records. It is
called every minute.

Performance should be fine, but if desired, we could call this worker
directly with sidekiq-cron every hour instead of piggybacking on the
`trigger_background_verification` method, without noticeable impact.

- Triggered by Geo::VerificationCronWorker every minute
- The work is done by Geo::VerificationBatchWorker
- Reuses LimitedCapacity::Worker concern
- Concurrent workers do not pick up the same records
- Fixes a bug in LimitedCapacity::Worker so it passes the job args
- To do: Killed jobs can cause stuck "started" records. A follow up MR
  is in progress.

The trigger `#trigger_background_verification` will be implemented in
the following commit.

It will decrease the boilerplate of SSF destroy worker

Apply 1 suggestion(s) to 1 file(s)

Apply 1 suggestion(s) to 1 file(s)

Assures that todos targeting inaccessible epics are deleted
(either because user left the group or epic became confidential).

Also adds a background migration which checks accessibility for
all existing confidential epics.

!48836

If the admin_new_user_signups_cap setting is defined, users will be
marked as blocked_pending_approval. After the current user count is
checked and the cap is not exceeded the user will be automatically
activated.

If the :admin_new_user_signups_cap feature is enabled and the cap is
increased by an admin, users in blocked_pending_approval state should
be auto approved.

- Fix an issue of delayed or failed project creation when using
  group level custom templates. Since it's using Project Export
  under the hood, regular project export requests can delay
  this type of project creation.
- In order to speed this process up, move Template Export into
  a separate worker/queue

Ensure changes to project visibility settings also update the associated
issue documents for that project. This is required to keep the issue
documents updated once the visibility permissions are stored within
the issue document in preparation for global search joins being
removed from the Elasticsearch issue query.

- Add job to queues
- Add specs

This commit introduces the auto rollback facility.

Move the dependency proxy feature from
premium to core.

This MR implements elastic migration framework, which looks and behaves
similar to rails database migrations.
Migrations are stored in ee/elastic/migrate/ with
YYYYMMDDHHMMSS_migration_name.rb file name format

This commit executes the external HTTP request in
sidekiq worker.

Since it is reusable on primaries and secondaries.

When the worker is given a top-level group and an array of
SAML group links, it will add users to groups based on those
group links. The next iteration will trigger this worker on SAML
sign-in when the IdP provides a list of group memberships.

- Add worker & update queues file

- Add docs page

- Update cron queue
- Add service for adding label
- add service for applying label

* Queues marked with this require some form of access for data
that is shared among various components.  An example might be that the
queue writes data to a local directory that is quickly displayed to the
user.
* In scenarios where large installations exist, this requires some form
of shared stored, commonly NFS, to be configured where sidekiq can write
the data, but then a set of web front ends have the same NFS share such
that they are capable of reading that data and vice versa.
* This commit marks all queues _known_ to have such a dependency
* As this dependency is removed, this tag should be removed

Once we deploy this new worker, security scan entries and findings will
be saved by the new worker class.
We will stop using the old worker but we can't remove it completely as
there might be some jobs enqueued for the worker.

LicenseMailer is a mailer for sending
emails related to licenses, for example
sending an email when EE instance
is approaching the active user count threshold.

Adds a new Sidekiq worker, state machine and service for validation.