Attribute Sidekiq workers to feature categories
What does this MR do?
Merge sequence: !18462 (merged)
Adds feature_category
attribution for each Sidekiq worker in the GitLab application.
Closes gitlab-com/gl-infra/scalability#32 (closed)
cc @marin
Tasks
-
Attribute each worker -
Provide documentation in doc/development/sidekiq_style_guide.md
-
Add CI checks to validate that all workers have declared feature category attributes see spec/workers/every_sidekiq_worker_spec.rb
-
Add CI checks to validate that all feature category attributes match values from https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml see spec/workers/every_sidekiq_worker_spec.rb
-
Provide a script to re-import the list of feature mappings from https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml: see bin/update-feature-categories
-
Notify all backend engineering managers via a ping on this MR about this change, so that they has first right of refusal on any of the assignments
Mapping
Sidekiq Class | Feature Category | Backend Engineering Manager for Feature |
---|---|---|
NewEpicWorker |
agile_portfolio_management |
John Hope |
LdapGroupSyncWorker |
authentication_and_authorization |
Liam McAndrew |
RepositoryPushAuditEventWorker |
authentication_and_authorization |
Liam McAndrew |
DeleteUserWorker |
authentication_and_authorization |
Liam McAndrew |
RemoveExpiredGroupLinksWorker |
authentication_and_authorization |
Liam McAndrew |
AuthorizedProjectsWorker |
authentication_and_authorization |
Liam McAndrew |
LdapSyncWorker |
authentication_and_authorization |
Liam McAndrew |
RemoveExpiredMembersWorker |
authentication_and_authorization |
Liam McAndrew |
LdapAllGroupsSyncWorker |
authentication_and_authorization |
Liam McAndrew |
NamespacelessProjectDestroyWorker |
authentication_and_authorization |
Liam McAndrew |
AutoDevops::DisableWorker |
auto_devops |
Seth Engelhard (Interim) |
Chaos::LeakMemWorker |
chaos_engineering |
Seth Engelhard (Interim) |
Chaos::SleepWorker |
chaos_engineering |
Seth Engelhard (Interim) |
Chaos::DbSpinWorker |
chaos_engineering |
Seth Engelhard (Interim) |
Chaos::KillWorker |
chaos_engineering |
Seth Engelhard (Interim) |
Chaos::CpuSpinWorker |
chaos_engineering |
Seth Engelhard (Interim) |
ChatNotificationWorker |
chatops |
Nicholas Klick |
CleanupContainerRepositoryWorker |
container_registry |
Daniel Croft |
DeleteContainerRepositoryWorker |
container_registry |
Daniel Croft |
AutoMergeProcessWorker |
continuous_delivery |
Darby Frey |
Deployments::FinishedWorker |
continuous_delivery |
Darby Frey |
Deployments::SuccessWorker |
continuous_delivery |
Darby Frey |
PipelineNotificationWorker |
continuous_integration |
Elliot Rushton |
StuckCiJobsWorker |
continuous_integration |
Elliot Rushton |
PipelineSuccessWorker |
continuous_integration |
Elliot Rushton |
BuildCoverageWorker |
continuous_integration |
Elliot Rushton |
StoreSecurityReportsWorker |
continuous_integration |
Elliot Rushton |
ClearSharedRunnersMinutesWorker |
continuous_integration |
Elliot Rushton |
BuildFinishedWorker |
continuous_integration |
Elliot Rushton |
ExpirePipelineCacheWorker |
continuous_integration |
Elliot Rushton |
CreatePipelineWorker |
continuous_integration |
Elliot Rushton |
UpdateHeadPipelineForMergeRequestWorker |
continuous_integration |
Elliot Rushton |
Ci::ArchiveTracesCronWorker |
continuous_integration |
Elliot Rushton |
Ci::BuildPrepareWorker |
continuous_integration |
Elliot Rushton |
Ci::BuildTraceChunkFlushWorker |
continuous_integration |
Elliot Rushton |
Ci::BuildScheduleWorker |
continuous_integration |
Elliot Rushton |
Ci::CreateCrossProjectPipelineWorker |
continuous_integration |
Elliot Rushton |
Ci::PipelineBridgeStatusWorker |
continuous_integration |
Elliot Rushton |
BuildSuccessWorker |
continuous_integration |
Elliot Rushton |
RunPipelineScheduleWorker |
continuous_integration |
Elliot Rushton |
BuildTraceSectionsWorker |
continuous_integration |
Elliot Rushton |
BuildQueueWorker |
continuous_integration |
Elliot Rushton |
PipelineScheduleWorker |
continuous_integration |
Elliot Rushton |
ArchiveTraceWorker |
continuous_integration |
Elliot Rushton |
ExpireBuildInstanceArtifactsWorker |
continuous_integration |
Elliot Rushton |
StageUpdateWorker |
continuous_integration |
Elliot Rushton |
PipelineUpdateWorker |
continuous_integration |
Elliot Rushton |
ExpireBuildArtifactsWorker |
continuous_integration |
Elliot Rushton |
PipelineMetricsWorker |
continuous_integration |
Elliot Rushton |
ExpireJobCacheWorker |
continuous_integration |
Elliot Rushton |
PipelineHooksWorker |
continuous_integration |
Elliot Rushton |
PipelineProcessWorker |
continuous_integration |
Elliot Rushton |
BuildHooksWorker |
continuous_integration |
Elliot Rushton |
DesignManagement::NewVersionWorker |
design_management |
Darva Satcher |
Geo::FileDownloadWorker |
geo_replication |
Rachel Nienaber |
Geo::HashedStorageMigrationWorker |
geo_replication |
Rachel Nienaber |
Geo::RepositoryVerification::Secondary::SingleWorker |
geo_replication |
Rachel Nienaber |
Geo::Secondary::RepositoryBackfillWorker |
geo_replication |
Rachel Nienaber |
GeoRepositoryDestroyWorker |
geo_replication |
Rachel Nienaber |
Geo::RepositoryVerification::Secondary::ShardWorker |
geo_replication |
Rachel Nienaber |
Geo::DesignRepositorySyncWorker |
geo_replication |
Rachel Nienaber |
Geo::RepositoryVerification::Secondary::SchedulerWorker |
geo_replication |
Rachel Nienaber |
Geo::ContainerRepositorySyncDispatchWorker |
geo_replication |
Rachel Nienaber |
Geo::Batch::ProjectRegistryWorker |
geo_replication |
Rachel Nienaber |
Geo::Batch::ProjectRegistrySchedulerWorker |
geo_replication |
Rachel Nienaber |
Geo::RepositoryShardSyncWorker |
geo_replication |
Rachel Nienaber |
Geo::MetricsUpdateWorker |
geo_replication |
Rachel Nienaber |
Geo::RepositoryVerification::Primary::ShardWorker |
geo_replication |
Rachel Nienaber |
Geo::RepositoryVerification::Primary::SingleWorker |
geo_replication |
Rachel Nienaber |
Geo::MigratedLocalFilesCleanUpWorker |
geo_replication |
Rachel Nienaber |
Geo::ContainerRepositorySyncWorker |
geo_replication |
Rachel Nienaber |
Geo::RenameRepositoryWorker |
geo_replication |
Rachel Nienaber |
Geo::ProjectSyncWorker |
geo_replication |
Rachel Nienaber |
Geo::RepositorySyncWorker |
geo_replication |
Rachel Nienaber |
Geo::RepositoryCleanupWorker |
geo_replication |
Rachel Nienaber |
Geo::RepositoriesCleanUpWorker |
geo_replication |
Rachel Nienaber |
Geo::FileDownloadDispatchWorker |
geo_replication |
Rachel Nienaber |
Geo::SidekiqCronConfigWorker |
geo_replication |
Rachel Nienaber |
Geo::FileRegistryRemovalWorker |
geo_replication |
Rachel Nienaber |
Geo::HashedStorageAttachmentsMigrationWorker |
geo_replication |
Rachel Nienaber |
UploadChecksumWorker |
geo_replication |
Rachel Nienaber |
Geo::Scheduler::Secondary::PerShardSchedulerWorker |
geo_replication |
Rachel Nienaber |
Geo::Scheduler::Secondary::SchedulerWorker |
geo_replication |
Rachel Nienaber |
Geo::Scheduler::Primary::PerShardSchedulerWorker |
geo_replication |
Rachel Nienaber |
Geo::Scheduler::Primary::SchedulerWorker |
geo_replication |
Rachel Nienaber |
Geo::Scheduler::PerShardSchedulerWorker |
geo_replication |
Rachel Nienaber |
Geo::Scheduler::SchedulerWorker |
geo_replication |
Rachel Nienaber |
Geo::PruneEventLogWorker |
geo_replication |
Rachel Nienaber |
Geo::RepositoryVerification::Primary::BatchWorker |
geo_replication |
Rachel Nienaber |
Geo::FileRemovalWorker |
geo_replication |
Rachel Nienaber |
ObjectPool::JoinWorker |
gitaly |
ZJ (Interim) |
ObjectPool::ScheduleJoinWorker |
gitaly |
ZJ (Interim) |
ObjectPool::DestroyWorker |
gitaly |
ZJ (Interim) |
ObjectPool::CreateWorker |
gitaly |
ZJ (Interim) |
GitGarbageCollectWorker |
gitaly |
ZJ (Interim) |
GroupDestroyWorker |
groups |
Liam McAndrew |
ProjectImportScheduleWorker |
importers |
Liam McAndrew |
RepositoryImportWorker |
importers |
Liam McAndrew |
RepositoryUpdateMirrorWorker |
importers |
Liam McAndrew |
ImportExportProjectCleanupWorker |
importers |
Liam McAndrew |
StuckImportJobsWorker |
importers |
Liam McAndrew |
Gitlab::GithubImport::ImportDiffNoteWorker |
importers |
Liam McAndrew |
Gitlab::GithubImport::Stage::ImportLfsObjectsWorker |
importers |
Liam McAndrew |
Gitlab::GithubImport::Stage::ImportRepositoryWorker |
importers |
Liam McAndrew |
Gitlab::GithubImport::Stage::ImportIssuesAndDiffNotesWorker |
importers |
Liam McAndrew |
Gitlab::GithubImport::Stage::ImportNotesWorker |
importers |
Liam McAndrew |
Gitlab::GithubImport::Stage::FinishImportWorker |
importers |
Liam McAndrew |
Gitlab::GithubImport::Stage::ImportBaseDataWorker |
importers |
Liam McAndrew |
Gitlab::GithubImport::Stage::ImportPullRequestsWorker |
importers |
Liam McAndrew |
Gitlab::GithubImport::RefreshImportJidWorker |
importers |
Liam McAndrew |
Gitlab::GithubImport::ImportIssueWorker |
importers |
Liam McAndrew |
Gitlab::GithubImport::ImportPullRequestWorker |
importers |
Liam McAndrew |
Gitlab::GithubImport::AdvanceStageWorker |
importers |
Liam McAndrew |
Gitlab::GithubImport::ImportLfsObjectWorker |
importers |
Liam McAndrew |
Gitlab::GithubImport::ImportNoteWorker |
importers |
Liam McAndrew |
IncidentManagement::ProcessAlertWorker |
incident_management |
Seth Engelhard |
IncidentManagement::ProcessPrometheusAlertWorker |
incident_management |
Seth Engelhard |
PruneWebHookLogsWorker |
integrations |
Nick Nguyen |
ProjectServiceWorker |
integrations |
Nick Nguyen |
PseudonymizerWorker |
integrations |
Nick Nguyen |
JiraConnect::SyncBranchWorker |
integrations |
Nick Nguyen |
IrkerWorker |
integrations |
Nick Nguyen |
CreateGithubWebhookWorker |
integrations |
Nick Nguyen |
WebHookWorker |
integrations |
Nick Nguyen |
PluginWorker |
integrations |
Nick Nguyen |
JiraConnect::SyncMergeRequestWorker |
integrations |
Nick Nguyen |
TodosDestroyer::GroupPrivateWorker |
issue_tracking |
Sean McGivern |
MailScheduler::IssueDueWorker |
issue_tracking |
Sean McGivern |
MailScheduler::NotificationServiceWorker |
issue_tracking |
Sean McGivern |
NewIssueWorker |
issue_tracking |
Sean McGivern |
ExportCsvWorker |
issue_tracking |
Sean McGivern |
EmailReceiverWorker |
issue_tracking |
Sean McGivern |
ImportIssuesCsvWorker |
issue_tracking |
Sean McGivern |
NewNoteWorker |
issue_tracking |
Sean McGivern |
TodosDestroyer::EntityLeaveWorker |
issue_tracking |
Sean McGivern |
IssueDueSchedulerWorker |
issue_tracking |
Sean McGivern |
TodosDestroyer::PrivateFeaturesWorker |
issue_tracking |
Sean McGivern |
TodosDestroyer::ConfidentialIssueWorker |
issue_tracking |
Sean McGivern |
TodosDestroyer::ProjectPrivateWorker |
issue_tracking |
Sean McGivern |
ClusterWaitForIngressIpAddressWorker |
kubernetes_configuration |
Seth Engelhard (Interim) |
Clusters::Applications::UninstallWorker |
kubernetes_configuration |
Seth Engelhard (Interim) |
ClusterWaitForAppInstallationWorker |
kubernetes_configuration |
Seth Engelhard (Interim) |
ClusterProvisionWorker |
kubernetes_configuration |
Seth Engelhard (Interim) |
ClusterPatchAppWorker |
kubernetes_configuration |
Seth Engelhard (Interim) |
ClusterWaitForAppUpdateWorker |
kubernetes_configuration |
Seth Engelhard (Interim) |
ClusterProjectConfigureWorker |
kubernetes_configuration |
Seth Engelhard (Interim) |
ClusterConfigureWorker |
kubernetes_configuration |
Seth Engelhard (Interim) |
Clusters::Applications::WaitForUninstallAppWorker |
kubernetes_configuration |
Seth Engelhard (Interim) |
WaitForClusterCreationWorker |
kubernetes_configuration |
Seth Engelhard (Interim) |
ClusterInstallAppWorker |
kubernetes_configuration |
Seth Engelhard (Interim) |
ClusterUpdateAppWorker |
kubernetes_configuration |
Seth Engelhard (Interim) |
ClusterUpgradeAppWorker |
kubernetes_configuration |
Seth Engelhard (Interim) |
HistoricalDataWorker |
license_compliance |
Olivier Gonzalez (Interim) |
ImportSoftwareLicensesWorker |
license_compliance |
Olivier Gonzalez (Interim) |
UpdateMaxSeatsUsedForGitlabComSubscriptionsWorker |
license_compliance |
Olivier Gonzalez (Interim) |
RefreshLicenseComplianceChecksWorker |
license_compliance |
Olivier Gonzalez (Interim) |
PruneOldEventsWorker |
not_owned |
|
ReactiveCachingWorker |
not_owned |
|
AdminEmailsWorker |
not_owned |
|
ObjectStorage::BackgroundMoveWorker |
not_owned |
|
ObjectStorage::MigrateUploadsWorker |
not_owned |
|
AdminEmailWorker |
not_owned |
|
GitlabUsagePingWorker |
not_owned |
|
DeleteStoredFilesWorker |
not_owned |
|
BackgroundMigrationWorker |
not_owned |
|
PagesDomainSslRenewalWorker |
pages |
Darby Frey |
PagesDomainSslRenewalCronWorker |
pages |
Darby Frey |
PagesDomainVerificationWorker |
pages |
Darby Frey |
PagesDomainRemovalCronWorker |
pages |
Darby Frey |
PagesWorker |
pages |
Darby Frey |
PagesDomainVerificationCronWorker |
pages |
Darby Frey |
ElasticCommitIndexerWorker |
search |
Craig Gomes (Interim) |
ElasticBatchProjectIndexerWorker |
search |
Craig Gomes (Interim) |
ElasticNamespaceIndexerWorker |
search |
Craig Gomes (Interim) |
ElasticFullIndexWorker |
search |
Craig Gomes (Interim) |
ElasticIndexerWorker |
search |
Craig Gomes (Interim) |
DetectRepositoryLanguagesWorker |
source_code_management |
Michelle Gill |
CreateNoteDiffFileWorker |
source_code_management |
Michelle Gill |
NewMergeRequestWorker |
source_code_management |
Michelle Gill |
RepositoryForkWorker |
source_code_management |
Michelle Gill |
ScheduleMigrateExternalDiffsWorker |
source_code_management |
Michelle Gill |
PostReceive |
source_code_management |
Michelle Gill |
CreateGpgSignatureWorker |
source_code_management |
Michelle Gill |
RemoteMirrorNotificationWorker |
source_code_management |
Michelle Gill |
Namespaces::RootStatisticsWorker |
source_code_management |
Michelle Gill |
TrendingProjectsWorker |
source_code_management |
Michelle Gill |
ProjectCacheWorker |
source_code_management |
Michelle Gill |
Namespaces::PruneAggregationSchedulesWorker |
source_code_management |
Michelle Gill |
Namespaces::ScheduleAggregationWorker |
source_code_management |
Michelle Gill |
MigrateExternalDiffsWorker |
source_code_management |
Michelle Gill |
EmailsOnPushWorker |
source_code_management |
Michelle Gill |
ProjectExportWorker |
source_code_management |
Michelle Gill |
StuckMergeJobsWorker |
source_code_management |
Michelle Gill |
UpdateMergeRequestsWorker |
source_code_management |
Michelle Gill |
RepositoryCheck::BatchWorker |
source_code_management |
Michelle Gill |
RepositoryCheck::SingleRepositoryWorker |
source_code_management |
Michelle Gill |
RepositoryCheck::DispatchWorker |
source_code_management |
Michelle Gill |
DeleteMergedBranchesWorker |
source_code_management |
Michelle Gill |
RepositoryCheck::ClearWorker |
source_code_management |
Michelle Gill |
RequestsProfilesWorker |
source_code_management |
Michelle Gill |
UpdateProjectStatisticsWorker |
source_code_management |
Michelle Gill |
RepositoryCleanupWorker |
source_code_management |
Michelle Gill |
ProjectDestroyWorker |
source_code_management |
Michelle Gill |
RepositoryUpdateRemoteMirrorWorker |
source_code_management |
Michelle Gill |
MergeWorker |
source_code_management |
Michelle Gill |
RepositoryArchiveCacheWorker |
source_code_management |
Michelle Gill |
RemoveUnreferencedLfsObjectsWorker |
source_code_management |
Michelle Gill |
HashedStorage::ProjectMigrateWorker |
source_code_management |
Michelle Gill |
HashedStorage::ProjectRollbackWorker |
source_code_management |
Michelle Gill |
HashedStorage::MigratorWorker |
source_code_management |
Michelle Gill |
HashedStorage::RollbackerWorker |
source_code_management |
Michelle Gill |
DeleteDiffFilesWorker |
source_code_management |
Michelle Gill |
ProjectDailyStatisticsWorker |
source_code_management |
Michelle Gill |
PropagateServiceTemplateWorker |
source_code_management |
Michelle Gill |
ProcessCommitWorker |
source_code_management |
Michelle Gill |
RepositoryRemoveRemoteWorker |
source_code_management |
Michelle Gill |
UpdateAllMirrorsWorker |
source_code_management |
Michelle Gill |
SystemHookPushWorker |
source_code_management |
Michelle Gill |
GitlabShellWorker |
source_code_management |
Michelle Gill |
RebaseWorker |
source_code_management |
Michelle Gill |
UpdateExternalPullRequestsWorker |
source_code_management |
Michelle Gill |
ProjectUpdateRepositoryStorageWorker |
source_code_management |
Michelle Gill |
InvalidGpgSignatureUpdateWorker |
source_code_management |
Michelle Gill |
SyncSecurityReportsToReportApprovalRulesWorker |
static_application_security_testing |
Thomas Woodham |
Generated with
require 'uri'
require 'net/http'
require 'yaml'
url = URI("https://gitlab.com/gitlab-com/www-gitlab-com/raw/master/data/stages.yml")
http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true
request = Net::HTTP::Get.new(url)
response = http.request(request)
stages_doc = YAML.load(response.read_body)
feature_managers = stages_doc["stages"].values.flat_map { |stage| stage["groups"].values.flat_map { |group| group["categories"]&.map { |c| [c, group["backend_engineering_manager"] ] } } }.select(&:itself).to_h
Gitlab::SidekiqConfig.workers.map { |klass| { k: klass.name, category: klass.get_feature_category } }.sort_by { |x| x[:category] }.each do |x|
puts "| \`#{x[:k]}\` | \`#{x[:category]}\` | #{feature_managers[x[:category].to_s]} |"
end
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation created/updated or follow-up review issue created -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Performance and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team