Skip to content

Attribute Sidekiq workers to feature categories

Andrew Newdigate requested to merge an-sidekiq-job-feature-attribution into master

What does this MR do?

Merge sequence: !18462 (merged) 🚋 !18066 (merged) 🚋

Adds feature_category attribution for each Sidekiq worker in the GitLab application.

Closes gitlab-com/gl-infra/scalability#32 (closed)

cc @marin

Tasks

  • Attribute each worker
  • Provide documentation in doc/development/sidekiq_style_guide.md
  • Add CI checks to validate that all workers have declared feature category attributes see spec/workers/every_sidekiq_worker_spec.rb
  • Add CI checks to validate that all feature category attributes match values from https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml  see spec/workers/every_sidekiq_worker_spec.rb
  • Provide a script to re-import the list of feature mappings from https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/stages.yml: see bin/update-feature-categories
  • Notify all backend engineering managers via a ping on this MR about this change, so that they has first right of refusal on any of the assignments

Mapping

Sidekiq Class Feature Category Backend Engineering Manager for Feature
NewEpicWorker agile_portfolio_management John Hope
LdapGroupSyncWorker authentication_and_authorization Liam McAndrew
RepositoryPushAuditEventWorker authentication_and_authorization Liam McAndrew
DeleteUserWorker authentication_and_authorization Liam McAndrew
RemoveExpiredGroupLinksWorker authentication_and_authorization Liam McAndrew
AuthorizedProjectsWorker authentication_and_authorization Liam McAndrew
LdapSyncWorker authentication_and_authorization Liam McAndrew
RemoveExpiredMembersWorker authentication_and_authorization Liam McAndrew
LdapAllGroupsSyncWorker authentication_and_authorization Liam McAndrew
NamespacelessProjectDestroyWorker authentication_and_authorization Liam McAndrew
AutoDevops::DisableWorker auto_devops Seth Engelhard (Interim)
Chaos::LeakMemWorker chaos_engineering Seth Engelhard (Interim)
Chaos::SleepWorker chaos_engineering Seth Engelhard (Interim)
Chaos::DbSpinWorker chaos_engineering Seth Engelhard (Interim)
Chaos::KillWorker chaos_engineering Seth Engelhard (Interim)
Chaos::CpuSpinWorker chaos_engineering Seth Engelhard (Interim)
ChatNotificationWorker chatops Nicholas Klick
CleanupContainerRepositoryWorker container_registry Daniel Croft
DeleteContainerRepositoryWorker container_registry Daniel Croft
AutoMergeProcessWorker continuous_delivery Darby Frey
Deployments::FinishedWorker continuous_delivery Darby Frey
Deployments::SuccessWorker continuous_delivery Darby Frey
PipelineNotificationWorker continuous_integration Elliot Rushton
StuckCiJobsWorker continuous_integration Elliot Rushton
PipelineSuccessWorker continuous_integration Elliot Rushton
BuildCoverageWorker continuous_integration Elliot Rushton
StoreSecurityReportsWorker continuous_integration Elliot Rushton
ClearSharedRunnersMinutesWorker continuous_integration Elliot Rushton
BuildFinishedWorker continuous_integration Elliot Rushton
ExpirePipelineCacheWorker continuous_integration Elliot Rushton
CreatePipelineWorker continuous_integration Elliot Rushton
UpdateHeadPipelineForMergeRequestWorker continuous_integration Elliot Rushton
Ci::ArchiveTracesCronWorker continuous_integration Elliot Rushton
Ci::BuildPrepareWorker continuous_integration Elliot Rushton
Ci::BuildTraceChunkFlushWorker continuous_integration Elliot Rushton
Ci::BuildScheduleWorker continuous_integration Elliot Rushton
Ci::CreateCrossProjectPipelineWorker continuous_integration Elliot Rushton
Ci::PipelineBridgeStatusWorker continuous_integration Elliot Rushton
BuildSuccessWorker continuous_integration Elliot Rushton
RunPipelineScheduleWorker continuous_integration Elliot Rushton
BuildTraceSectionsWorker continuous_integration Elliot Rushton
BuildQueueWorker continuous_integration Elliot Rushton
PipelineScheduleWorker continuous_integration Elliot Rushton
ArchiveTraceWorker continuous_integration Elliot Rushton
ExpireBuildInstanceArtifactsWorker continuous_integration Elliot Rushton
StageUpdateWorker continuous_integration Elliot Rushton
PipelineUpdateWorker continuous_integration Elliot Rushton
ExpireBuildArtifactsWorker continuous_integration Elliot Rushton
PipelineMetricsWorker continuous_integration Elliot Rushton
ExpireJobCacheWorker continuous_integration Elliot Rushton
PipelineHooksWorker continuous_integration Elliot Rushton
PipelineProcessWorker continuous_integration Elliot Rushton
BuildHooksWorker continuous_integration Elliot Rushton
DesignManagement::NewVersionWorker design_management Darva Satcher
Geo::FileDownloadWorker geo_replication Rachel Nienaber
Geo::HashedStorageMigrationWorker geo_replication Rachel Nienaber
Geo::RepositoryVerification::Secondary::SingleWorker geo_replication Rachel Nienaber
Geo::Secondary::RepositoryBackfillWorker geo_replication Rachel Nienaber
GeoRepositoryDestroyWorker geo_replication Rachel Nienaber
Geo::RepositoryVerification::Secondary::ShardWorker geo_replication Rachel Nienaber
Geo::DesignRepositorySyncWorker geo_replication Rachel Nienaber
Geo::RepositoryVerification::Secondary::SchedulerWorker geo_replication Rachel Nienaber
Geo::ContainerRepositorySyncDispatchWorker geo_replication Rachel Nienaber
Geo::Batch::ProjectRegistryWorker geo_replication Rachel Nienaber
Geo::Batch::ProjectRegistrySchedulerWorker geo_replication Rachel Nienaber
Geo::RepositoryShardSyncWorker geo_replication Rachel Nienaber
Geo::MetricsUpdateWorker geo_replication Rachel Nienaber
Geo::RepositoryVerification::Primary::ShardWorker geo_replication Rachel Nienaber
Geo::RepositoryVerification::Primary::SingleWorker geo_replication Rachel Nienaber
Geo::MigratedLocalFilesCleanUpWorker geo_replication Rachel Nienaber
Geo::ContainerRepositorySyncWorker geo_replication Rachel Nienaber
Geo::RenameRepositoryWorker geo_replication Rachel Nienaber
Geo::ProjectSyncWorker geo_replication Rachel Nienaber
Geo::RepositorySyncWorker geo_replication Rachel Nienaber
Geo::RepositoryCleanupWorker geo_replication Rachel Nienaber
Geo::RepositoriesCleanUpWorker geo_replication Rachel Nienaber
Geo::FileDownloadDispatchWorker geo_replication Rachel Nienaber
Geo::SidekiqCronConfigWorker geo_replication Rachel Nienaber
Geo::FileRegistryRemovalWorker geo_replication Rachel Nienaber
Geo::HashedStorageAttachmentsMigrationWorker geo_replication Rachel Nienaber
UploadChecksumWorker geo_replication Rachel Nienaber
Geo::Scheduler::Secondary::PerShardSchedulerWorker geo_replication Rachel Nienaber
Geo::Scheduler::Secondary::SchedulerWorker geo_replication Rachel Nienaber
Geo::Scheduler::Primary::PerShardSchedulerWorker geo_replication Rachel Nienaber
Geo::Scheduler::Primary::SchedulerWorker geo_replication Rachel Nienaber
Geo::Scheduler::PerShardSchedulerWorker geo_replication Rachel Nienaber
Geo::Scheduler::SchedulerWorker geo_replication Rachel Nienaber
Geo::PruneEventLogWorker geo_replication Rachel Nienaber
Geo::RepositoryVerification::Primary::BatchWorker geo_replication Rachel Nienaber
Geo::FileRemovalWorker geo_replication Rachel Nienaber
ObjectPool::JoinWorker gitaly ZJ (Interim)
ObjectPool::ScheduleJoinWorker gitaly ZJ (Interim)
ObjectPool::DestroyWorker gitaly ZJ (Interim)
ObjectPool::CreateWorker gitaly ZJ (Interim)
GitGarbageCollectWorker gitaly ZJ (Interim)
GroupDestroyWorker groups Liam McAndrew
ProjectImportScheduleWorker importers Liam McAndrew
RepositoryImportWorker importers Liam McAndrew
RepositoryUpdateMirrorWorker importers Liam McAndrew
ImportExportProjectCleanupWorker importers Liam McAndrew
StuckImportJobsWorker importers Liam McAndrew
Gitlab::GithubImport::ImportDiffNoteWorker importers Liam McAndrew
Gitlab::GithubImport::Stage::ImportLfsObjectsWorker importers Liam McAndrew
Gitlab::GithubImport::Stage::ImportRepositoryWorker importers Liam McAndrew
Gitlab::GithubImport::Stage::ImportIssuesAndDiffNotesWorker importers Liam McAndrew
Gitlab::GithubImport::Stage::ImportNotesWorker importers Liam McAndrew
Gitlab::GithubImport::Stage::FinishImportWorker importers Liam McAndrew
Gitlab::GithubImport::Stage::ImportBaseDataWorker importers Liam McAndrew
Gitlab::GithubImport::Stage::ImportPullRequestsWorker importers Liam McAndrew
Gitlab::GithubImport::RefreshImportJidWorker importers Liam McAndrew
Gitlab::GithubImport::ImportIssueWorker importers Liam McAndrew
Gitlab::GithubImport::ImportPullRequestWorker importers Liam McAndrew
Gitlab::GithubImport::AdvanceStageWorker importers Liam McAndrew
Gitlab::GithubImport::ImportLfsObjectWorker importers Liam McAndrew
Gitlab::GithubImport::ImportNoteWorker importers Liam McAndrew
IncidentManagement::ProcessAlertWorker incident_management Seth Engelhard
IncidentManagement::ProcessPrometheusAlertWorker incident_management Seth Engelhard
PruneWebHookLogsWorker integrations Nick Nguyen
ProjectServiceWorker integrations Nick Nguyen
PseudonymizerWorker integrations Nick Nguyen
JiraConnect::SyncBranchWorker integrations Nick Nguyen
IrkerWorker integrations Nick Nguyen
CreateGithubWebhookWorker integrations Nick Nguyen
WebHookWorker integrations Nick Nguyen
PluginWorker integrations Nick Nguyen
JiraConnect::SyncMergeRequestWorker integrations Nick Nguyen
TodosDestroyer::GroupPrivateWorker issue_tracking Sean McGivern
MailScheduler::IssueDueWorker issue_tracking Sean McGivern
MailScheduler::NotificationServiceWorker issue_tracking Sean McGivern
NewIssueWorker issue_tracking Sean McGivern
ExportCsvWorker issue_tracking Sean McGivern
EmailReceiverWorker issue_tracking Sean McGivern
ImportIssuesCsvWorker issue_tracking Sean McGivern
NewNoteWorker issue_tracking Sean McGivern
TodosDestroyer::EntityLeaveWorker issue_tracking Sean McGivern
IssueDueSchedulerWorker issue_tracking Sean McGivern
TodosDestroyer::PrivateFeaturesWorker issue_tracking Sean McGivern
TodosDestroyer::ConfidentialIssueWorker issue_tracking Sean McGivern
TodosDestroyer::ProjectPrivateWorker issue_tracking Sean McGivern
ClusterWaitForIngressIpAddressWorker kubernetes_configuration Seth Engelhard (Interim)
Clusters::Applications::UninstallWorker kubernetes_configuration Seth Engelhard (Interim)
ClusterWaitForAppInstallationWorker kubernetes_configuration Seth Engelhard (Interim)
ClusterProvisionWorker kubernetes_configuration Seth Engelhard (Interim)
ClusterPatchAppWorker kubernetes_configuration Seth Engelhard (Interim)
ClusterWaitForAppUpdateWorker kubernetes_configuration Seth Engelhard (Interim)
ClusterProjectConfigureWorker kubernetes_configuration Seth Engelhard (Interim)
ClusterConfigureWorker kubernetes_configuration Seth Engelhard (Interim)
Clusters::Applications::WaitForUninstallAppWorker kubernetes_configuration Seth Engelhard (Interim)
WaitForClusterCreationWorker kubernetes_configuration Seth Engelhard (Interim)
ClusterInstallAppWorker kubernetes_configuration Seth Engelhard (Interim)
ClusterUpdateAppWorker kubernetes_configuration Seth Engelhard (Interim)
ClusterUpgradeAppWorker kubernetes_configuration Seth Engelhard (Interim)
HistoricalDataWorker license_compliance Olivier Gonzalez (Interim)
ImportSoftwareLicensesWorker license_compliance Olivier Gonzalez (Interim)
UpdateMaxSeatsUsedForGitlabComSubscriptionsWorker license_compliance Olivier Gonzalez (Interim)
RefreshLicenseComplianceChecksWorker license_compliance Olivier Gonzalez (Interim)
PruneOldEventsWorker not_owned
ReactiveCachingWorker not_owned
AdminEmailsWorker not_owned
ObjectStorage::BackgroundMoveWorker not_owned
ObjectStorage::MigrateUploadsWorker not_owned
AdminEmailWorker not_owned
GitlabUsagePingWorker not_owned
DeleteStoredFilesWorker not_owned
BackgroundMigrationWorker not_owned
PagesDomainSslRenewalWorker pages Darby Frey
PagesDomainSslRenewalCronWorker pages Darby Frey
PagesDomainVerificationWorker pages Darby Frey
PagesDomainRemovalCronWorker pages Darby Frey
PagesWorker pages Darby Frey
PagesDomainVerificationCronWorker pages Darby Frey
ElasticCommitIndexerWorker search Craig Gomes (Interim)
ElasticBatchProjectIndexerWorker search Craig Gomes (Interim)
ElasticNamespaceIndexerWorker search Craig Gomes (Interim)
ElasticFullIndexWorker search Craig Gomes (Interim)
ElasticIndexerWorker search Craig Gomes (Interim)
DetectRepositoryLanguagesWorker source_code_management Michelle Gill
CreateNoteDiffFileWorker source_code_management Michelle Gill
NewMergeRequestWorker source_code_management Michelle Gill
RepositoryForkWorker source_code_management Michelle Gill
ScheduleMigrateExternalDiffsWorker source_code_management Michelle Gill
PostReceive source_code_management Michelle Gill
CreateGpgSignatureWorker source_code_management Michelle Gill
RemoteMirrorNotificationWorker source_code_management Michelle Gill
Namespaces::RootStatisticsWorker source_code_management Michelle Gill
TrendingProjectsWorker source_code_management Michelle Gill
ProjectCacheWorker source_code_management Michelle Gill
Namespaces::PruneAggregationSchedulesWorker source_code_management Michelle Gill
Namespaces::ScheduleAggregationWorker source_code_management Michelle Gill
MigrateExternalDiffsWorker source_code_management Michelle Gill
EmailsOnPushWorker source_code_management Michelle Gill
ProjectExportWorker source_code_management Michelle Gill
StuckMergeJobsWorker source_code_management Michelle Gill
UpdateMergeRequestsWorker source_code_management Michelle Gill
RepositoryCheck::BatchWorker source_code_management Michelle Gill
RepositoryCheck::SingleRepositoryWorker source_code_management Michelle Gill
RepositoryCheck::DispatchWorker source_code_management Michelle Gill
DeleteMergedBranchesWorker source_code_management Michelle Gill
RepositoryCheck::ClearWorker source_code_management Michelle Gill
RequestsProfilesWorker source_code_management Michelle Gill
UpdateProjectStatisticsWorker source_code_management Michelle Gill
RepositoryCleanupWorker source_code_management Michelle Gill
ProjectDestroyWorker source_code_management Michelle Gill
RepositoryUpdateRemoteMirrorWorker source_code_management Michelle Gill
MergeWorker source_code_management Michelle Gill
RepositoryArchiveCacheWorker source_code_management Michelle Gill
RemoveUnreferencedLfsObjectsWorker source_code_management Michelle Gill
HashedStorage::ProjectMigrateWorker source_code_management Michelle Gill
HashedStorage::ProjectRollbackWorker source_code_management Michelle Gill
HashedStorage::MigratorWorker source_code_management Michelle Gill
HashedStorage::RollbackerWorker source_code_management Michelle Gill
DeleteDiffFilesWorker source_code_management Michelle Gill
ProjectDailyStatisticsWorker source_code_management Michelle Gill
PropagateServiceTemplateWorker source_code_management Michelle Gill
ProcessCommitWorker source_code_management Michelle Gill
RepositoryRemoveRemoteWorker source_code_management Michelle Gill
UpdateAllMirrorsWorker source_code_management Michelle Gill
SystemHookPushWorker source_code_management Michelle Gill
GitlabShellWorker source_code_management Michelle Gill
RebaseWorker source_code_management Michelle Gill
UpdateExternalPullRequestsWorker source_code_management Michelle Gill
ProjectUpdateRepositoryStorageWorker source_code_management Michelle Gill
InvalidGpgSignatureUpdateWorker source_code_management Michelle Gill
SyncSecurityReportsToReportApprovalRulesWorker static_application_security_testing Thomas Woodham

Generated with

require 'uri'
require 'net/http'
require 'yaml'

url = URI("https://gitlab.com/gitlab-com/www-gitlab-com/raw/master/data/stages.yml")

http = Net::HTTP.new(url.host, url.port)
http.use_ssl = true

request = Net::HTTP::Get.new(url)

response = http.request(request)

stages_doc = YAML.load(response.read_body)

feature_managers = stages_doc["stages"].values.flat_map { |stage| stage["groups"].values.flat_map { |group| group["categories"]&.map { |c| [c, group["backend_engineering_manager"] ] } } }.select(&:itself).to_h

Gitlab::SidekiqConfig.workers.map { |klass| { k: klass.name, category: klass.get_feature_category } }.sort_by { |x| x[:category] }.each do |x|
  puts "| \`#{x[:k]}\` | \`#{x[:category]}\` | #{feature_managers[x[:category].to_s]} |"
end

Screenshots

Does this MR meet the acceptance criteria?

Conformity

Performance and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Kamil Trzciński

Merge request reports