MR : !65073

gitlab-com/www-gitlab-com!70219
reorganised some of these categories. Most of the items that were under
`collection` can go to `usage_ping`, but the historical usage export
probably belongs under `provision` (as it's to do with license usage),
and the survey response are - tangentially - under `navigation`.

To be more resilient to network errors, usage ping
will raise an exception when it fails, and retry
via sidekiq

This commit's goal is to add a cop (Scalability::IdempotentWorker)
that will catch all workers that don't call idempotent! in its scope.

Calling it will label the worker as idempotent, which will end up in
our Sidekiq logs for further visibility.

This also introduces a shared example and a perform_multiple helper
method for executing jobs multiple times through unit tests.

Also in this commit we set an example in an already idempotent worker
ExpireJobCacheWorker. For this worker we call idempotent! in its
scope and add tests with the new test helpers.

AdminEmailWorker only send repository check emails, so it belongs to
source code management. (It also doesn't run on GitLab.com.)

GitlabUsagePingWorker belongs to the collection category.

PruneOldEventsWorker relates to events which are used on the user
profile, so I've gone with users. This could also go under audit events,
potentially.

AdminEmailsWorker sends an admin-defined email to all users, so I put it
under issue tracking as that currently owns notifications (!). (This
also doesn't run on GitLab.com.)

Bob Van Landuyt authored 5 years ago and Alessio Caiazza committed 5 years ago

These workers don't need context because they don't operate scoped to
a user, namespace or project.

This disables the cops for current offences, so we can get to them
fixed 1 by 1.

Add a new class to lib that caches usage data
and write to grafana embed usage data to the Rails
cache during the usage ping background job prior to
sending the usage ping POST request.

Andrew Newdigate authored 5 years ago and Kamil Trzciński committed 5 years ago

We should consider having a clearer mapping between product/categories
and sidekiq queues. This adds feature_category attribution for each
Sidekiq worker in the GitLab application.

* Splay the start by sleeping up to one minute.
* Adjust sideikiq retries to try up to approximately three hours.
* Randomize the minute fully within the hour.
* Psudo-randomize the day of the week based on the instance UUID.

Sean McGivern authored 7 years ago and Rémy Coutable committed 7 years ago

CE port of https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/735

Sean McGivern authored 7 years ago and Rémy Coutable committed 7 years ago

CE port of https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/557

This builds on
https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7006 and adjusts
various parts to take care of EE specific workers.

Closes #1044

Robert Speicher authored 8 years ago and Ruben Alexis committed 8 years ago

Add more usage data to EE ping

This MR loads the EE usage ping asynchronously on the admin application settings page and now includes the counts of the following items:

* Comments
* Groups
* Users
* Projects
* Issues
* Labels
* CI builds
* Snippets
* Milestones
* Todos
* Pushes
* Merge requests
* Environments
* Triggers
* Deploy keys
* Pages
* Project Services
* Issue Boards
* CI Runners
* Deployments
* Geo Nodes
* LDAP Groups
* LDAP Keys
* LDAP Users
* LFS objects
* Protected branches
* Releases
* Remote mirrors
* Web hooks

Closes #997

See merge request !735

Load it asynchronously to prevent blocking a page load.

Closes #997

Add a container around usage data help-block to prevent horizontal scroll in Firefox

Closes #808

Closes #380