Sidekiq pods never become ready, killed by failing liveness checks
Summary
The gitlab-sidekiq-all-in-1-v2
pod never listens on the metrics / health check port (3807), so the liveness and readiness checks fail.
Workaround
Ensure Enable health and performance metrics endpoint
from Settings > Metrics and Profiling > Metrics - Prometheus
(/admin/application_settings/metrics_and_profiling#js-prometheus-settings
) is ENABLED and restart any affected pods.
Steps to reproduce
I could not reproduce this on a different deployment, I just have one deployment where this happens. They are very similar.
The failing cluster was upgraded directly from chart 5.8.2 (GitLab 14.8.2) to 5.9.0 (GitLab 14.9.0). The problem still occurs with chart 5.9.1 (GL 14.9.1). I also tried removing the Helm release entirely and re-deploying with no changes in behaviour, so this is clearly something that's persisting in PostgreSQL or Redis.
/edit: added by GitLab
- Deploy GitLab
- Login to instance as administrator
- Disable
Enable health and performance metrics endpoint
fromSettings > Metrics and Profiling > Metrics - Prometheus
(/admin/application_settings/metrics_and_profiling#js-prometheus-settings
) - Restart Sidekiq Pods.
-
💥 See that no process is listening on port3807
Configuration used
global:
# Use GitLab CE, not EE
edition: ce
hosts:
domain: not-set.example.com # placeholder
https: true
# DNS names need to point at the Nginx-Ingress LB endpoint
gitlab:
name: gitlab.example.com
registry:
name: registry.gitlab.example.com
ingress:
# Use cluster-wide cert-manager and issuer
configureCertmanager: false
class: nginx
tls:
secretName: FIXME-placeholder-tls
psql:
host: gitlab-postgres-12
port: 5432
database: gitlab
username: gitlab
password:
secret: gitlab.gitlab-postgres-12.credentials
key: password
minio:
# We are using S3
enabled: false
redis:
host: gitlab
sentinels:
- host: gitlab-redis-ha-announce-0
port: 26379
- host: gitlab-redis-ha-announce-1
port: 26379
- host: gitlab-redis-ha-announce-2
port: 26379
password:
enabled: false
appConfig:
defaultProjectsFeatures:
# Disable features we don't generally use by default
snippets: false
wiki: false
object_store:
enabled: true
proxy_download: false
connection:
secret: gitlab-s3
key: connection
lfs:
proxy_download: false
bucket: git-lfs.gitlab.example.com
artifacts:
proxy_download: false
bucket: artifacts.gitlab.example.com
uploads:
proxy_download: false
bucket: uploads.gitlab.example.com
packages:
proxy_download: false
bucket: packages.gitlab.example.com
externalDiffs:
enabled: true
when: outdated
bucket: externaldiffs.gitlab.example.com
proxy_download: false
terraformState:
enabled: true
proxy_download: false
bucket: terraformstate.gitlab.example.com
dependencyProxy:
enabled: true
bucket: dependencyproxy.gitlab.example.com
proxy_download: false
pseudonymizer:
proxy_download: false
bucket: pseudonymizer.gitlab.example.com
backups:
bucket: backups.gitlab.example.com
tmpBucket: backups-temp.gitlab.example.com
omniauth:
enabled: true
autoSignInWithProvider: saml
syncProfileFromProvider: [saml]
syncProfileAttributes: [name, email]
allowSingleSignOn: [saml]
blockAutoCreatedUsers: true
autoLinkSamlUser: true
providers:
- secret: gitlab-omniauth-saml
shell:
hostKeys:
secret: gitlab-shell-host-keys
registry:
bucket: registry.gitlab.example.com
smtp:
enabled: true
address: email-smtp.eu-west-1.amazonaws.com
port: 587
user_name: REDACTED
password:
secret: gitlab-smtp
starttls_auto: true
email:
from: gitlab@gitlab.example.com
display_name: GitLab
reply_to: noreply@gitlab.example.com
time_zone: Europe/London
antiAffinity: soft
gitlab:
gitaly:
annotations:
# Avoid the auto-scaler evicting this pod, it's very disruptive
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
persistence:
storageClass: efs
gitlab-shell:
tcpExternalConfig: true
sidekiq:
annotations:
iam.amazonaws.com/role: GitLabRole
toolbox:
annotations:
iam.amazonaws.com/role: GitLabRole
backups:
cron:
enabled: true
schedule: "28 0 * * *"
extraArgs: "--skip registry,artifacts"
objectStorage:
config:
secret: gitlab-s3
key: s3cmd-config
webservice:
annotations:
iam.amazonaws.com/role: GitLabRole
antiAffinity: soft
ingress:
annotations:
alb.ingress.kubernetes.io/ip-address-type: dualstack
cert-manager.io/cluster-issuer: letsencrypt-prod
external-dns.alpha.kubernetes.io/hostname: gitlab.example.com
tls:
secretName: gitlab-unicorn-tls
resources:
limits:
memory: 1.8G
requests:
memory: 1.2G
certmanager:
# Use the cluster-wide cert-manager
install: false
prometheus:
# Use the cluster-wide Prometheus
install: false
nginx-ingress:
# Use the cluster-side nginx-ingress
enabled: false
gitlab-runner:
# We'll deploy our own gitlab-runner
install: false
postgresql:
install: false
redis:
install: false
registry:
annotations:
iam.amazonaws.com/role: GitLabRole
antiAffinity: soft
ingress:
annotations:
alb.ingress.kubernetes.io/ip-address-type: dualstack
cert-manager.io/cluster-issuer: letsencrypt-prod
tls:
secretName: gitlab-registry-tls
storage:
secret: gitlab-s3
key: registry-config
validation:
disabled: false # we do not use Windows containers
Current behavior
The Sidekiq pod starts and appears to be working correctly, processing jobs, connects to Redis and PostgreSQL fine - it just never listens on its metrics port so K8s never considers it ready/working.
Expected behavior
It should listen on port 3807.
Versions
- Chart: 5.9.1
- Platform:
- Cloud: EKS
- Kubernetes: (
kubectl version
)- Client: v1.23.5
- Server: v1.21.5-eks-bc4871b
- Helm: (
helm version
)- Client: 3.7.2 (Flux v2 helm-controller 0.15.0)
- Server: n/a
Relevant logs
+ /scripts/set-config /var/opt/gitlab/templates /srv/gitlab/config
Begin parsing .erb templates from /var/opt/gitlab/templates
Writing /srv/gitlab/config/cable.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/resque.yml
Begin parsing .tpl templates from /var/opt/gitlab/templates
Copying other config files found in /var/opt/gitlab/templates to /srv/gitlab/config
Copying smtp_settings.rb into /srv/gitlab/config
+ exec /scripts/exec-env /bin/sh -c /scripts/process-wrapper
{"severity":"INFO","time":"2022-03-23T17:39:29.143Z","message":"Starting cluster with 1 processes"}
*** /var/log/gitlab/application.log ***
2022-03-23T17:39:41.275Z: {:message=>"Syncing dynamic postgres partitions"}
2022-03-23T17:39:41.279Z: {:message=>"Switched database connection", :connection_name=>"main"}
2022-03-23T17:39:41.341Z: {:message=>"Checking state of dynamic postgres partitions", :table_name=>"audit_events", :connection_name=>"main"}
2022-03-23T17:39:41.580Z: {:message=>"Switched database connection", :connection_name=>"main"}
2022-03-23T17:39:41.581Z: {:message=>"Checking state of dynamic postgres partitions", :table_name=>"web_hook_logs", :connection_name=>"main"}
2022-03-23T17:39:41.646Z: {:message=>"Switched database connection", :connection_name=>"main"}
2022-03-23T17:39:41.647Z: {:message=>"Checking state of dynamic postgres partitions", :table_name=>"loose_foreign_keys_deleted_records", :connection_name=>"main"}
2022-03-23T17:39:41.718Z: {:message=>"Switched database connection", :connection_name=>"main"}
2022-03-23T17:39:41.718Z: {:message=>"Checking state of dynamic postgres partitions", :table_name=>"batched_background_migration_job_transition_logs", :connection_name=>"main"}
2022-03-23T17:39:41.764Z: {:message=>"Switched database connection", :connection_name=>"main"}
2022-03-23T17:39:41.765Z: {:message=>"Checking state of dynamic postgres partitions", :table_name=>"incident_management_pending_alert_escalations", :connection_name=>"main"}
2022-03-23T17:39:41.822Z: {:message=>"Switched database connection", :connection_name=>"main"}
2022-03-23T17:39:41.822Z: {:message=>"Checking state of dynamic postgres partitions", :table_name=>"incident_management_pending_issue_escalations", :connection_name=>"main"}
*** /var/log/gitlab/application_json.log ***
{"severity":"INFO","time":"2022-03-23T17:39:41.276Z","correlation_id":null,"message":"Syncing dynamic postgres partitions"}
{"severity":"DEBUG","time":"2022-03-23T17:39:41.280Z","correlation_id":null,"message":"Switched database connection","connection_name":"main"}
{"severity":"INFO","time":"2022-03-23T17:39:41.342Z","correlation_id":null,"message":"Checking state of dynamic postgres partitions","table_name":"audit_events","connection_name":"main"}
{"severity":"DEBUG","time":"2022-03-23T17:39:41.581Z","correlation_id":null,"message":"Switched database connection","connection_name":"main"}
{"severity":"INFO","time":"2022-03-23T17:39:41.581Z","correlation_id":null,"message":"Checking state of dynamic postgres partitions","table_name":"web_hook_logs","connection_name":"main"}
{"severity":"DEBUG","time":"2022-03-23T17:39:41.646Z","correlation_id":null,"message":"Switched database connection","connection_name":"main"}
{"severity":"INFO","time":"2022-03-23T17:39:41.647Z","correlation_id":null,"message":"Checking state of dynamic postgres partitions","table_name":"loose_foreign_keys_deleted_records","connection_name":"main"}
{"severity":"DEBUG","time":"2022-03-23T17:39:41.718Z","correlation_id":null,"message":"Switched database connection","connection_name":"main"}
{"severity":"INFO","time":"2022-03-23T17:39:41.718Z","correlation_id":null,"message":"Checking state of dynamic postgres partitions","table_name":"batched_background_migration_job_transition_logs","connection_name":"main"}
{"severity":"DEBUG","time":"2022-03-23T17:39:41.765Z","correlation_id":null,"message":"Switched database connection","connection_name":"main"}
{"severity":"INFO","time":"2022-03-23T17:39:41.765Z","correlation_id":null,"message":"Checking state of dynamic postgres partitions","table_name":"incident_management_pending_alert_escalations","connection_name":"main"}
{"severity":"DEBUG","time":"2022-03-23T17:39:41.822Z","correlation_id":null,"message":"Switched database connection","connection_name":"main"}
{"severity":"INFO","time":"2022-03-23T17:39:41.822Z","correlation_id":null,"message":"Checking state of dynamic postgres partitions","table_name":"incident_management_pending_issue_escalations","connection_name":"main"}
2022-03-23T17:39:42.026Z pid=23 tid=dzf INFO: Listening on queues ["analytics_usage_trends_counter_job", "approve_blocked_pending_approval_users", "authorized_keys", "authorized_project_update:authorized_project_update_project_create", "authorized_project_update:authorized_project_update_project_group_link_create", "authorized_project_update:authorized_project_update_project_recalculate", "authorized_project_update:authorized_project_update_project_recalculate_per_user", "authorized_project_update:authorized_project_update_user_refresh_from_replica", "authorized_project_update:authorized_project_update_user_refresh_over_user_range", "authorized_project_update:authorized_project_update_user_refresh_with_low_urgency", "authorized_projects", "auto_devops:auto_devops_disable", "auto_merge:auto_merge_process", "background_migration", "background_migration_ci_database", "bulk_import", "bulk_imports_entity", "bulk_imports_export_request", "bulk_imports_pipeline", "bulk_imports_relation_export", "chaos:chaos_cpu_spin", "chaos:chaos_db_spin", "chaos:chaos_kill", "chaos:chaos_leak_mem", "chaos:chaos_sleep", "chat_notification", "ci_delete_objects", "ci_job_artifacts_expire_project_build_artifacts", "cluster_agent:clusters_agents_delete_expired_events", "container_repository:cleanup_container_repository", "container_repository:container_expiration_policies_cleanup_container_repository", "container_repository:delete_container_repository", "create_commit_signature", "create_note_diff_file", "cronjob:admin_email", "cronjob:analytics_usage_trends_count_job_trigger", "cronjob:authorized_project_update_periodic_recalculate", "cronjob:ci_archive_traces_cron", "cronjob:ci_delete_unit_tests", "cronjob:ci_pipeline_artifacts_expire_artifacts", "cronjob:ci_platform_metrics_update_cron", "cronjob:ci_schedule_delete_objects_cron", "cronjob:ci_stuck_builds_drop_running", "cronjob:ci_stuck_builds_drop_scheduled", "cronjob:clusters_integrations_check_prometheus_health", "cronjob:container_expiration_policy", "cronjob:container_registry_migration_enqueuer", "cronjob:container_registry_migration_guard", "cronjob:container_registry_migration_observer", "cronjob:database_batched_background_migration", "cronjob:database_batched_background_migration_ci_database", "cronjob:database_drop_detached_partitions", "cronjob:database_partition_management", "cronjob:dependency_proxy_cleanup_dependency_proxy", "cronjob:dependency_proxy_image_ttl_group_policy", "cronjob:environments_auto_delete_cron", "cronjob:environments_auto_stop_cron", "cronjob:expire_build_artifacts", "cronjob:gitlab_service_ping", "cronjob:import_export_project_cleanup", "cronjob:import_stuck_project_import_jobs", "cronjob:issue_due_scheduler", "cronjob:issues_reschedule_stuck_issue_rebalances", "cronjob:jira_import_stuck_jira_import_jobs", "cronjob:loose_foreign_keys_cleanup", "cronjob:member_invitation_reminder_emails", "cronjob:metrics_dashboard_schedule_annotations_prune", "cronjob:namespaces_in_product_marketing_emails", "cronjob:namespaces_prune_aggregation_schedules", "cronjob:packages_cleanup_package_registry", "cronjob:packages_composer_cache_cleanup", "cronjob:pages_domain_removal_cron", "cronjob:pages_domain_ssl_renewal_cron", "cronjob:pages_domain_verification_cron", "cronjob:partition_creation", "cronjob:personal_access_tokens_expired_notification", "cronjob:personal_access_tokens_expiring", "cronjob:pipeline_schedule", "cronjob:projects_schedule_refresh_build_artifacts_size_statistics", "cronjob:prune_old_events", "cronjob:quality_test_data_cleanup", "cronjob:releases_manage_evidence", "cronjob:remove_expired_group_links", "cronjob:remove_expired_members", "cronjob:remove_unaccepted_member_invites", "cronjob:remove_unreferenced_lfs_objects", "cronjob:repository_archive_cache", "cronjob:repository_check_dispatch", "cronjob:requests_profiles", "cronjob:schedule_merge_request_cleanup_refs", "cronjob:schedule_migrate_external_diffs", "cronjob:ssh_keys_expired_notification", "cronjob:ssh_keys_expiring_soon_notification", "cronjob:stuck_ci_jobs", "cronjob:stuck_export_jobs", "cronjob:stuck_merge_jobs", "cronjob:trending_projects", "cronjob:update_container_registry_info", "cronjob:user_status_cleanup_batch", "cronjob:users_create_statistics", "cronjob:users_deactivate_dormant_users", "cronjob:x509_issuer_crl_check", "default", "delete_diff_files", "delete_merged_branches", "delete_stored_files", "delete_user", "dependency_proxy:purge_dependency_proxy_cache", "dependency_proxy_blob:dependency_proxy_cleanup_blob", "dependency_proxy_manifest:dependency_proxy_cleanup_manifest", "deployment:deployments_archive_in_project", "deployment:deployments_drop_older_deployments", "deployment:deployments_hooks", "deployment:deployments_link_merge_request", "deployment:deployments_update_environment", "design_management_copy_design_collection", "design_management_new_version", "destroy_pages_deployments", "detect_repository_languages", "disallow_two_factor_for_group", "disallow_two_factor_for_subgroups", "email_receiver", "emails_on_push", "environments_auto_stop", "environments_canary_ingress_update", "error_tracking_issue_link", "experiments_record_conversion_event", "expire_build_instance_artifacts", "export_csv", "external_service_reactive_caching", "file_hook", "flush_counter_increments", "gcp_cluster:cluster_configure_istio", "gcp_cluster:cluster_install_app", "gcp_cluster:cluster_patch_app", "gcp_cluster:cluster_provision", "gcp_cluster:cluster_update_app", "gcp_cluster:cluster_upgrade_app", "gcp_cluster:cluster_wait_for_app_installation", "gcp_cluster:cluster_wait_for_app_update", "gcp_cluster:cluster_wait_for_ingress_ip_address", "gcp_cluster:clusters_applications_activate_service", "gcp_cluster:clusters_applications_deactivate_service", "gcp_cluster:clusters_applications_uninstall", "gcp_cluster:clusters_applications_wait_for_uninstall_app", "gcp_cluster:clusters_cleanup_project_namespace", "gcp_cluster:clusters_cleanup_service_account", "gcp_cluster:wait_for_cluster_creation", "github_import_advance_stage", "github_importer:github_import_import_diff_note", "github_importer:github_import_import_issue", "github_importer:github_import_import_lfs_object", "github_importer:github_import_import_note", "github_importer:github_import_import_pull_request", "github_importer:github_import_import_pull_request_merged_by", "github_importer:github_import_import_pull_request_review", "github_importer:github_import_refresh_import_jid", "github_importer:github_import_stage_finish_import", "github_importer:github_import_stage_import_base_data", "github_importer:github_import_stage_import_issues_and_diff_notes", "github_importer:github_import_stage_import_lfs_objects", "github_importer:github_import_stage_import_notes", "github_importer:github_import_stage_import_pull_requests", "github_importer:github_import_stage_import_pull_requests_merged_by", "github_importer:github_import_stage_import_pull_requests_reviews", "github_importer:github_import_stage_import_repository", "gitlab_performance_bar_stats", "gitlab_shell", "group_destroy", "group_export", "group_import", "groups_update_statistics", "hashed_storage:hashed_storage_migrator", "hashed_storage:hashed_storage_project_migrate", "hashed_storage:hashed_storage_project_rollback", "hashed_storage:hashed_storage_rollbacker", "import_issues_csv", "incident_management:incident_management_add_severity_system_note", "incident_management:incident_management_pager_duty_process_incident", "incident_management:incident_management_process_alert_worker_v2", "integrations_create_external_cross_reference", "invalid_gpg_signature_update", "irker", "issuable_export_csv", "issuable_label_links_destroy", "issuables_clear_groups_issue_counter", "issue_placement", "issue_rebalancing", "issues_placement", "issues_rebalancing", "jira_connect:jira_connect_forward_event", "jira_connect:jira_connect_retry_request", "jira_connect:jira_connect_sync_branch", "jira_connect:jira_connect_sync_builds", "jira_connect:jira_connect_sync_deployments", "jira_connect:jira_connect_sync_feature_flags", "jira_connect:jira_connect_sync_merge_request", "jira_connect:jira_connect_sync_project", "jira_importer:jira_import_advance_stage", "jira_importer:jira_import_import_issue", "jira_importer:jira_import_stage_finish_import", "jira_importer:jira_import_stage_import_attachments", "jira_importer:jira_import_stage_import_issues", "jira_importer:jira_import_stage_import_labels", "jira_importer:jira_import_stage_import_notes", "jira_importer:jira_import_stage_start_import", "mail_scheduler:mail_scheduler_issue_due", "mail_scheduler:mail_scheduler_notification_service", "mailers", "merge", "merge_request_cleanup_refs", "merge_request_mergeability_check", "merge_requests_delete_source_branch", "merge_requests_handle_assignees_change", "merge_requests_resolve_todos", "merge_requests_update_head_pipeline", "metrics_dashboard_prune_old_annotations", "metrics_dashboard_sync_dashboards", "migrate_external_diffs", "namespaceless_project_destroy", "namespaces_invite_team_email", "namespaces_onboarding_issue_created", "namespaces_onboarding_pipeline_created", "namespaces_onboarding_progress", "namespaces_onboarding_user_added", "namespaces_process_sync_events", "namespaces_update_root_statistics", "new_issue", "new_merge_request", "new_note", "object_pool:object_pool_create", "object_pool:object_pool_destroy", "object_pool:object_pool_join", "object_pool:object_pool_schedule_join", "object_storage:object_storage_background_move", "object_storage:object_storage_migrate_uploads", "package_cleanup:packages_cleanup_package_file", "package_cleanup:packages_mark_package_files_for_destruction", "package_repositories:packages_debian_generate_distribution", "package_repositories:packages_debian_process_changes", "package_repositories:packages_go_sync_packages", "package_repositories:packages_helm_extraction", "package_repositories:packages_maven_metadata_sync", "package_repositories:packages_nuget_extraction", "package_repositories:packages_rubygems_extraction", "packages_composer_cache_update", "pages", "pages_domain_ssl_renewal", "pages_domain_verification", "pages_transfer", "phabricator_import_import_tasks", "pipeline_background:archive_trace", "pipeline_background:ci_archive_trace", "pipeline_background:ci_build_trace_chunk_flush", "pipeline_background:ci_daily_build_group_report_results", "pipeline_background:ci_pending_builds_update_group", "pipeline_background:ci_pending_builds_update_project", "pipeline_background:ci_pipeline_artifacts_coverage_report", "pipeline_background:ci_pipeline_artifacts_create_quality_report", "pipeline_background:ci_pipeline_success_unlock_artifacts", "pipeline_background:ci_ref_delete_unlock_artifacts", "pipeline_background:ci_test_failure_history", "pipeline_cache:expire_job_cache", "pipeline_cache:expire_pipeline_cache", "pipeline_creation:ci_external_pull_requests_create_pipeline", "pipeline_creation:create_pipeline", "pipeline_creation:merge_requests_create_pipeline", "pipeline_creation:run_pipeline_schedule", "pipeline_default:ci_create_cross_project_pipeline", "pipeline_default:ci_create_downstream_pipeline", "pipeline_default:ci_drop_pipeline", "pipeline_default:ci_merge_requests_add_todo_when_build_fails", "pipeline_default:ci_pipeline_bridge_status", "pipeline_default:ci_retry_pipeline", "pipeline_default:pipeline_metrics", "pipeline_default:pipeline_notification", "pipeline_hooks:build_hooks", "pipeline_hooks:pipeline_hooks", "pipeline_processing:build_finished", "pipeline_processing:build_queue", "pipeline_processing:build_success", "pipeline_processing:ci_build_finished", "pipeline_processing:ci_build_prepare", "pipeline_processing:ci_build_schedule", "pipeline_processing:ci_initial_pipeline_process", "pipeline_processing:ci_resource_groups_assign_resource_from_resource_group", "pipeline_processing:pipeline_process", "pipeline_processing:stage_update", "pipeline_processing:update_head_pipeline_for_merge_request", "post_receive", "process_commit", "project_cache", "project_daily_statistics", "project_destroy", "project_export", "project_service", "projects_git_garbage_collect", "projects_post_creation", "projects_process_sync_events", "projects_refresh_build_artifacts_size_statistics", "projects_schedule_bulk_repository_shard_moves", "projects_update_repository_storage", "prometheus_create_default_alerts", "propagate_integration", "propagate_integration_group", "propagate_integration_inherit", "propagate_integration_inherit_descendant", "propagate_integration_project", "reactive_caching", "rebase", "releases_create_evidence", "remote_mirror_notification", "repository_check:repository_check_batch", "repository_check:repository_check_clear", "repository_check:repository_check_single_repository", "repository_cleanup", "repository_fork", "repository_import", "repository_remove_remote", "repository_update_remote_mirror", "self_monitoring_project_create", "self_monitoring_project_delete", "service_desk_email_receiver", "snippets_schedule_bulk_repository_shard_moves", "snippets_update_repository_storage", "system_hook_push", "tasks_to_be_done_create", "todos_destroyer:todos_destroyer_confidential_issue", "todos_destroyer:todos_destroyer_destroyed_designs", "todos_destroyer:todos_destroyer_destroyed_issuable", "todos_destroyer:todos_destroyer_entity_leave", "todos_destroyer:todos_destroyer_group_private", "todos_destroyer:todos_destroyer_private_features", "todos_destroyer:todos_destroyer_project_private", "unassign_issuables:members_destroyer_unassign_issuables", "update_external_pull_requests", "update_highest_role", "update_merge_requests", "update_namespace_statistics:namespaces_root_statistics", "update_namespace_statistics:namespaces_schedule_aggregation", "update_project_statistics", "upload_checksum", "web_hook", "web_hooks_destroy", "web_hooks_log_execution", "wikis_git_garbage_collect", "x509_certificate_revoke"]
2022-03-23T17:39:42.026Z pid=23 tid=dzf INFO: Booting Sidekiq 6.4.0 with redis options {:scheme=>"redis", :host=>"gitlab", :port=>6379, :password=>nil, :sentinels=>[{:host=>"gitlab-redis-ha-announce-0", :port=>26379}, {:host=>"gitlab-redis-ha-announce-1", :port=>26379}, {:host=>"gitlab-redis-ha-announce-2", :port=>26379}], :id=>nil, :instrumentation_class=>Gitlab::Instrumentation::Redis::Queues, :namespace=>"resque:gitlab"}
*** /var/log/gitlab/application.log ***
2022-03-23T17:39:41.870Z: {:message=>"Switched database connection", :connection_name=>"main"}
2022-03-23T17:39:41.871Z: {:message=>"Checking state of dynamic postgres partitions", :table_name=>"verification_codes", :connection_name=>"main"}
2022-03-23T17:39:41.920Z: {:message=>"Finished sync of dynamic postgres partitions"}
*** /var/log/gitlab/application_json.log ***
{"severity":"DEBUG","time":"2022-03-23T17:39:41.871Z","correlation_id":null,"message":"Switched database connection","connection_name":"main"}
{"severity":"INFO","time":"2022-03-23T17:39:41.871Z","correlation_id":null,"message":"Checking state of dynamic postgres partitions","table_name":"verification_codes","connection_name":"main"}
{"severity":"INFO","time":"2022-03-23T17:39:41.920Z","correlation_id":null,"message":"Finished sync of dynamic postgres partitions"}
*** /var/log/gitlab/production.log ***
Raven 3.1.2 configured not to capture errors: DSN not set
*** /var/log/gitlab/auth.log ***
{"severity":"INFO","time":"2022-03-23T17:39:41.927Z","correlation_id":null,"gitlab_throttle_user_allowlist":[]}
2022-03-23T17:39:42.982Z pid=23 tid=dzf INFO: GitLab reliable fetch activated!
2022-03-23T17:39:43.027Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: stuck_ci_jobs_worker
2022-03-23T17:39:43.033Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: pipeline_schedule_worker
2022-03-23T17:39:43.039Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: expire_build_artifacts_worker
2022-03-23T17:39:43.045Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: ci_pipelines_expire_artifacts_worker
2022-03-23T17:39:43.051Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: ci_schedule_delete_objects_worker
2022-03-23T17:39:43.057Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: environments_auto_stop_cron_worker
2022-03-23T17:39:43.063Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: environments_auto_delete_cron_worker
2022-03-23T17:39:43.069Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: repository_check_worker
2022-03-23T17:39:43.075Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: admin_email_worker
2022-03-23T17:39:43.082Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: personal_access_tokens_expiring_worker
2022-03-23T17:39:43.088Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: personal_access_tokens_expired_notification_worker
2022-03-23T17:39:43.093Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: repository_archive_cache_worker
2022-03-23T17:39:43.099Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: import_export_project_cleanup_worker
2022-03-23T17:39:43.105Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: ci_archive_traces_cron_worker
2022-03-23T17:39:43.110Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: requests_profiles_worker
2022-03-23T17:39:43.116Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: remove_expired_members_worker
2022-03-23T17:39:43.122Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: remove_expired_group_links_worker
2022-03-23T17:39:43.156Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: remove_unaccepted_member_invites_worker
2022-03-23T17:39:43.163Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: prune_old_events_worker
2022-03-23T17:39:43.170Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: trending_projects_worker
2022-03-23T17:39:43.176Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: remove_unreferenced_lfs_objects_worker
2022-03-23T17:39:43.183Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: import_stuck_project_import_jobs
2022-03-23T17:39:43.189Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: jira_import_stuck_jira_import_jobs
2022-03-23T17:39:43.195Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: stuck_export_jobs_worker
2022-03-23T17:39:43.200Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: gitlab_service_ping_worker
2022-03-23T17:39:43.205Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: stuck_merge_jobs_worker
2022-03-23T17:39:43.211Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: pages_domain_verification_cron_worker
2022-03-23T17:39:43.216Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: pages_domain_removal_cron_worker
2022-03-23T17:39:43.221Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: pages_domain_ssl_renewal_cron_worker
2022-03-23T17:39:43.229Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: issue_due_scheduler_worker
2022-03-23T17:39:43.235Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: metrics_dashboard_schedule_annotations_prune_worker
2022-03-23T17:39:43.241Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: schedule_migrate_external_diffs_worker
2022-03-23T17:39:43.246Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: namespaces_prune_aggregation_schedules_worker
2022-03-23T17:39:43.251Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: container_expiration_policy_worker
2022-03-23T17:39:43.258Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: container_registry_migration_guard_worker
2022-03-23T17:39:43.264Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: container_registry_migration_observer_worker
2022-03-23T17:39:43.269Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: container_registry_migration_enqueuer_worker
2022-03-23T17:39:43.275Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: image_ttl_group_policy_worker
2022-03-23T17:39:43.281Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: cleanup_dependency_proxy_worker
2022-03-23T17:39:43.287Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: cleanup_package_registry_worker
2022-03-23T17:39:43.293Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: x509_issuer_crl_check_worker
2022-03-23T17:39:43.299Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: users_create_statistics_worker
2022-03-23T17:39:43.306Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: authorized_project_update_periodic_recalculate_worker
2022-03-23T17:39:43.313Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: update_container_registry_info_worker
2022-03-23T17:39:43.319Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: postgres_dynamic_partitions_manager
2022-03-23T17:39:43.325Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: postgres_dynamic_partitions_dropper
2022-03-23T17:39:43.332Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: ci_platform_metrics_update_cron_worker
2022-03-23T17:39:43.338Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: analytics_usage_trends_count_job_trigger_worker
2022-03-23T17:39:43.344Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: member_invitation_reminder_emails_worker
2022-03-23T17:39:43.349Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: schedule_merge_request_cleanup_refs_worker
2022-03-23T17:39:43.355Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: manage_evidence_worker
2022-03-23T17:39:43.360Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: user_status_cleanup_batch_worker
2022-03-23T17:39:43.365Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: ssh_keys_expired_notification_worker
2022-03-23T17:39:43.372Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: namespaces_in_product_marketing_emails_worker
2022-03-23T17:39:43.377Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: ssh_keys_expiring_soon_notification_worker
2022-03-23T17:39:43.383Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: users_deactivate_dormant_users_worker
2022-03-23T17:39:43.388Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: ci_delete_unit_tests_worker
2022-03-23T17:39:43.394Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: batched_background_migrations_worker
2022-03-23T17:39:43.399Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: batched_background_migration_worker_ci_database
2022-03-23T17:39:43.405Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: issues_reschedule_stuck_issue_rebalances
2022-03-23T17:39:43.410Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: clusters_integrations_check_prometheus_health_worker
2022-03-23T17:39:43.415Z pid=23 tid=dzf INFO: Cron Jobs - add job with name: projects_schedule_refresh_build_artifacts_size_statistics_worker
2022-03-23T17:39:44.732Z pid=23 tid=dzf WARN: Creating scope :without_statuses. Overwriting existing method CommitStatus.without_statuses.
2022-03-23T17:39:44.775Z pid=23 tid=dzf WARN: Creating scope :without_statuses. Overwriting existing method Ci::Pipeline.without_statuses.
*** /var/log/gitlab/production.log ***
Creating scope :without_statuses. Overwriting existing method CommitStatus.without_statuses.
Creating scope :without_statuses. Overwriting existing method Ci::Pipeline.without_statuses.
2022-03-23T17:39:45.027Z pid=23 tid=dzf WARN: Creating scope :in_pipelines. Overwriting existing method Ci::Build.in_pipelines.
2022-03-23T17:39:45.286Z pid=23 tid=dzf WARN: Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects.
2022-03-23T17:39:45.287Z pid=23 tid=dzf WARN: Creating scope :join_project. Overwriting existing method MergeRequest.join_project.
2022-03-23T17:39:45.288Z pid=23 tid=dzf WARN: Creating scope :references_project. Overwriting existing method MergeRequest.references_project.
2022-03-23T17:39:45.307Z pid=23 tid=dzf WARN: Creating scope :system. Overwriting existing method Note.system.
2022-03-23T17:39:45.831Z pid=23 tid=dzf WARN: Creating scope :order_created_desc. Overwriting existing method Packages::Package.order_created_desc.
2022-03-23T17:39:45.831Z pid=23 tid=dzf WARN: Creating scope :order_name_desc. Overwriting existing method Packages::Package.order_name_desc.
Creating scope :in_pipelines. Overwriting existing method Ci::Build.in_pipelines.
Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects.
Creating scope :join_project. Overwriting existing method MergeRequest.join_project.
Creating scope :references_project. Overwriting existing method MergeRequest.references_project.
Creating scope :system. Overwriting existing method Note.system.
Creating scope :order_created_desc. Overwriting existing method Packages::Package.order_created_desc.
Creating scope :order_name_desc. Overwriting existing method Packages::Package.order_name_desc.
2022-03-23T17:39:47.595Z pid=23 tid=dzf WARN: Creating scope :open. Overwriting existing method AlertManagement::Alert.open.
2022-03-23T17:39:47.641Z pid=23 tid=dzf WARN: Creating scope :open. Overwriting existing method IncidentManagement::IssuableEscalationStatus.open.
Creating scope :open. Overwriting existing method AlertManagement::Alert.open.
Creating scope :open. Overwriting existing method IncidentManagement::IssuableEscalationStatus.open.
2022-03-23T17:39:48.166Z pid=23 tid=dzf WARN: Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc.
2022-03-23T17:39:48.166Z pid=23 tid=dzf WARN: Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc.
Creating scope :order_name_asc. Overwriting existing method Label.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Label.order_name_desc.
2022-03-23T17:39:50.003Z pid=23 tid=dzf WARN: Creating scope :open. Overwriting existing method ExternalPullRequest.open.
2022-03-23T17:39:50.119Z pid=23 tid=dzf WARN: Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc.
2022-03-23T17:39:50.119Z pid=23 tid=dzf WARN: Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc.
2022-03-23T17:39:50.343Z pid=23 tid=dzf WARN: Creating scope :system. Overwriting existing method PrometheusMetric.system.
2022-03-23T17:39:50.604Z pid=23 tid=dzf WARN: Creating scope :with_user. Overwriting existing method GroupMember.with_user.
Creating scope :open. Overwriting existing method ExternalPullRequest.open.
Creating scope :order_name_asc. Overwriting existing method Member.order_name_asc.
Creating scope :order_name_desc. Overwriting existing method Member.order_name_desc.
Creating scope :system. Overwriting existing method PrometheusMetric.system.
Creating scope :with_user. Overwriting existing method GroupMember.with_user.
2022-03-23T17:39:51.508Z pid=23 tid=dzf WARN: Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses.
2022-03-23T17:39:51.509Z pid=23 tid=dzf WARN: Creating scope :created. Overwriting existing method Ci::Stage.created.
2022-03-23T17:39:51.509Z pid=23 tid=dzf WARN: Creating scope :pending. Overwriting existing method Ci::Stage.pending.
2022-03-23T17:39:51.510Z pid=23 tid=dzf WARN: Creating scope :running. Overwriting existing method Ci::Stage.running.
2022-03-23T17:39:51.510Z pid=23 tid=dzf WARN: Creating scope :success. Overwriting existing method Ci::Stage.success.
2022-03-23T17:39:51.510Z pid=23 tid=dzf WARN: Creating scope :failed. Overwriting existing method Ci::Stage.failed.
2022-03-23T17:39:51.511Z pid=23 tid=dzf WARN: Creating scope :canceled. Overwriting existing method Ci::Stage.canceled.
2022-03-23T17:39:51.511Z pid=23 tid=dzf WARN: Creating scope :skipped. Overwriting existing method Ci::Stage.skipped.
2022-03-23T17:39:51.511Z pid=23 tid=dzf WARN: Creating scope :manual. Overwriting existing method Ci::Stage.manual.
2022-03-23T17:39:51.512Z pid=23 tid=dzf WARN: Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled.
2022-03-23T17:39:51.512Z pid=23 tid=dzf WARN: Creating scope :preparing. Overwriting existing method Ci::Stage.preparing.
2022-03-23T17:39:51.512Z pid=23 tid=dzf WARN: Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource.
2022-03-23T17:39:51.555Z pid=23 tid=dzf WARN: Creating scope :database. Overwriting existing method Ci::BuildTraceChunk.database.
Creating scope :without_statuses. Overwriting existing method Ci::Stage.without_statuses.
Creating scope :created. Overwriting existing method Ci::Stage.created.
Creating scope :pending. Overwriting existing method Ci::Stage.pending.
Creating scope :running. Overwriting existing method Ci::Stage.running.
Creating scope :success. Overwriting existing method Ci::Stage.success.
Creating scope :failed. Overwriting existing method Ci::Stage.failed.
Creating scope :canceled. Overwriting existing method Ci::Stage.canceled.
Creating scope :skipped. Overwriting existing method Ci::Stage.skipped.
Creating scope :manual. Overwriting existing method Ci::Stage.manual.
Creating scope :scheduled. Overwriting existing method Ci::Stage.scheduled.
Creating scope :preparing. Overwriting existing method Ci::Stage.preparing.
Creating scope :waiting_for_resource. Overwriting existing method Ci::Stage.waiting_for_resource.
Creating scope :database. Overwriting existing method Ci::BuildTraceChunk.database.
2022-03-23T17:39:58.080Z pid=23 tid=dzf WARN: Creating scope :system. Overwriting existing method Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system.
Creating scope :system. Overwriting existing method Gitlab::DatabaseImporters::CommonMetrics::PrometheusMetric.system.
*** /var/log/gitlab/application_json.log ***
{"severity":"INFO","time":"2022-03-23T17:39:58.772Z","correlation_id":null,"message":"Application boot finished","runtime":"sidekiq","duration_s":29.632850574329495}
2022-03-23T17:40:09.397Z pid=23 tid=dzf INFO: Booted Rails 6.1.4.6 application in production environment
2022-03-23T17:40:09.397Z pid=23 tid=dzf INFO: Running in ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
2022-03-23T17:40:09.397Z pid=23 tid=dzf INFO: See LICENSE and the LGPL-3.0 for licensing details.
2022-03-23T17:40:09.397Z pid=23 tid=dzf INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org
2022-03-23T17:40:09.406Z pid=23 tid=5se3 INFO: {:class=>"Gitlab::SidekiqDaemon::MemoryKiller", :action=>"start", :pid=>23, :message=>"Starting Gitlab::SidekiqDaemon::MemoryKiller Daemon"}
2022-03-23T17:40:23.347Z pid=23 tid=5ssr class=DeleteDiffFilesWorker jid=e564d7330c3d8c9d48b496d5 INFO: start
2022-03-23T17:40:23.353Z pid=23 tid=5stj class=Ci::StuckBuilds::DropScheduledWorker jid=0f200dd70ed20e51eb3afe82 INFO: start
2022-03-23T17:40:23.356Z pid=23 tid=5ssr class=DeleteDiffFilesWorker jid=e564d7330c3d8c9d48b496d5 INFO: arguments: [22174]
2022-03-23T17:40:23.362Z pid=23 tid=5stj class=Ci::StuckBuilds::DropScheduledWorker jid=0f200dd70ed20e51eb3afe82 INFO: arguments: []
2022-03-23T17:40:23.418Z pid=23 tid=5stj class=Ci::StuckBuilds::DropScheduledWorker jid=0f200dd70ed20e51eb3afe82 elapsed=0.065 INFO: done
2022-03-23T17:40:23.463Z pid=23 tid=5ssr class=DeleteDiffFilesWorker jid=e564d7330c3d8c9d48b496d5 elapsed=0.115 INFO: done
{"severity":"INFO","time":"2022-03-23T17:40:23.395Z","correlation_id":"187227ee637e83cc8ed0e5fc96144c66","message":"Ci::StuckBuilds::DropScheduledService: Cleaning scheduled, timed-out builds"}
*** /var/log/gitlab/application.log ***
2022-03-23T17:40:23.395Z: Ci::StuckBuilds::DropScheduledService: Cleaning scheduled, timed-out builds
2022-03-23T17:40:24.237Z pid=23 tid=5stj class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=84a10baa6e2e5ce2b00f0d58 INFO: start
2022-03-23T17:40:24.239Z pid=23 tid=5stj class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=84a10baa6e2e5ce2b00f0d58 INFO: arguments: []
2022-03-23T17:40:24.241Z pid=23 tid=5stj class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=84a10baa6e2e5ce2b00f0d58 INFO: {:class=>"Database::BatchedBackgroundMigration::CiDatabaseWorker", :database=>"ci", :message=>"skipping migration execution for unconfigured database"}
2022-03-23T17:40:24.285Z pid=23 tid=5stj class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=84a10baa6e2e5ce2b00f0d58 elapsed=0.048 INFO: done
2022-03-23T17:40:24.292Z pid=23 tid=5ssr class=ContainerRegistry::Migration::GuardWorker jid=a3ea5578669511f8446acff2 INFO: start
2022-03-23T17:40:24.294Z pid=23 tid=5ssr class=ContainerRegistry::Migration::GuardWorker jid=a3ea5578669511f8446acff2 INFO: arguments: []
2022-03-23T17:40:24.315Z pid=23 tid=5ssr class=ContainerRegistry::Migration::GuardWorker jid=a3ea5578669511f8446acff2 elapsed=0.023 INFO: done
2022-03-23T17:40:24.318Z pid=23 tid=5t27 class=Database::BatchedBackgroundMigrationWorker jid=ab0603067fb280ac54ae0147 INFO: start
2022-03-23T17:40:24.320Z pid=23 tid=5t27 class=Database::BatchedBackgroundMigrationWorker jid=ab0603067fb280ac54ae0147 INFO: arguments: []
2022-03-23T17:40:24.337Z pid=23 tid=5t2f class=ScheduleMergeRequestCleanupRefsWorker jid=a91836f42bc5087a1b339b90 INFO: start
2022-03-23T17:40:24.338Z pid=23 tid=5t27 class=Database::BatchedBackgroundMigrationWorker jid=ab0603067fb280ac54ae0147 elapsed=0.021 INFO: done
2022-03-23T17:40:24.339Z pid=23 tid=5t2f class=ScheduleMergeRequestCleanupRefsWorker jid=a91836f42bc5087a1b339b90 INFO: arguments: []
2022-03-23T17:40:24.363Z pid=23 tid=5t2f class=ScheduleMergeRequestCleanupRefsWorker jid=a91836f42bc5087a1b339b90 elapsed=0.026 INFO: done
2022-03-23T17:40:24.363Z pid=23 tid=5t3n class=UserStatusCleanup::BatchWorker jid=9da0cfd89590149bc6eec5db INFO: start
2022-03-23T17:40:24.367Z pid=23 tid=5t3n class=UserStatusCleanup::BatchWorker jid=9da0cfd89590149bc6eec5db INFO: arguments: []
2022-03-23T17:40:24.377Z pid=23 tid=5t4z class=PagesDomainSslRenewalCronWorker jid=1b08185f478a9b7313c6d9b6 INFO: start
2022-03-23T17:40:24.378Z pid=23 tid=5t3n class=UserStatusCleanup::BatchWorker jid=9da0cfd89590149bc6eec5db elapsed=0.015 INFO: done
2022-03-23T17:40:24.379Z pid=23 tid=5t4z class=PagesDomainSslRenewalCronWorker jid=1b08185f478a9b7313c6d9b6 INFO: arguments: []
2022-03-23T17:40:24.499Z pid=23 tid=5t4z class=PagesDomainSslRenewalCronWorker jid=1b08185f478a9b7313c6d9b6 elapsed=0.123 INFO: done
2022-03-23T17:40:52.572Z pid=23 tid=5t4z class=Ci::DailyBuildGroupReportResultsWorker jid=80f197ed602e809817db9c4d INFO: start
2022-03-23T17:40:52.578Z pid=23 tid=5t4z class=Ci::DailyBuildGroupReportResultsWorker jid=80f197ed602e809817db9c4d INFO: arguments: [35451]
2022-03-23T17:40:52.928Z pid=23 tid=5t4z class=Ci::DailyBuildGroupReportResultsWorker jid=80f197ed602e809817db9c4d elapsed=0.356 INFO: done
2022-03-23T17:41:22.253Z pid=23 tid=5t2f class=ExternalServiceReactiveCachingWorker jid=468705db10f9d525052575df INFO: start
2022-03-23T17:41:22.254Z pid=23 tid=5t2f class=ExternalServiceReactiveCachingWorker jid=468705db10f9d525052575df INFO: arguments: ["VersionCheck","14.9.1"]
2022-03-23T17:41:22.256Z pid=23 tid=5stj class=ExternalServiceReactiveCachingWorker jid=9da0b7c9c4b8085357c3c853 INFO: start
2022-03-23T17:41:22.258Z pid=23 tid=5stj class=ExternalServiceReactiveCachingWorker jid=9da0b7c9c4b8085357c3c853 INFO: arguments: ["VersionCheck","14.9.1"]
2022-03-23T17:41:22.271Z pid=23 tid=5stj class=ExternalServiceReactiveCachingWorker jid=9da0b7c9c4b8085357c3c853 elapsed=0.015 INFO: done
2022-03-23T17:41:22.710Z pid=23 tid=5t2f class=ExternalServiceReactiveCachingWorker jid=468705db10f9d525052575df WARN: Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in
Sidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today
by calling `Sidekiq.strict_args!` during Sidekiq initialization.
2022-03-23T17:41:22.719Z pid=23 tid=5t2f class=ExternalServiceReactiveCachingWorker jid=468705db10f9d525052575df elapsed=0.466 INFO: done
2022-03-23T17:41:25.033Z pid=23 tid=5t3n class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=c8151ba4d996f0fd06326065 INFO: start
2022-03-23T17:41:25.034Z pid=23 tid=5t3n class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=c8151ba4d996f0fd06326065 INFO: arguments: []
2022-03-23T17:41:25.094Z pid=23 tid=5t3n class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=c8151ba4d996f0fd06326065 INFO: {:class=>"Database::BatchedBackgroundMigration::CiDatabaseWorker", :database=>"ci", :message=>"skipping migration execution for unconfigured database"}
2022-03-23T17:41:25.096Z pid=23 tid=5t3n class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=c8151ba4d996f0fd06326065 elapsed=0.063 INFO: done
2022-03-23T17:41:25.099Z pid=23 tid=5t4z class=Database::BatchedBackgroundMigrationWorker jid=d900874cca194e6b813c955d INFO: start
2022-03-23T17:41:25.101Z pid=23 tid=5t4z class=Database::BatchedBackgroundMigrationWorker jid=d900874cca194e6b813c955d INFO: arguments: []
2022-03-23T17:41:25.122Z pid=23 tid=5t4z class=Database::BatchedBackgroundMigrationWorker jid=d900874cca194e6b813c955d elapsed=0.023 INFO: done
2022-03-23T17:41:25.124Z pid=23 tid=5t9z class=ScheduleMergeRequestCleanupRefsWorker jid=2c659caf21ec488af377f71d INFO: start
2022-03-23T17:41:25.126Z pid=23 tid=5t9z class=ScheduleMergeRequestCleanupRefsWorker jid=2c659caf21ec488af377f71d INFO: arguments: []
2022-03-23T17:41:25.149Z pid=23 tid=5t9z class=ScheduleMergeRequestCleanupRefsWorker jid=2c659caf21ec488af377f71d elapsed=0.024 INFO: done
2022-03-23T17:41:25.149Z pid=23 tid=5t9f class=UserStatusCleanup::BatchWorker jid=6952fc9dfd2b5768305b9b3a INFO: start
2022-03-23T17:41:25.152Z pid=23 tid=5t9f class=UserStatusCleanup::BatchWorker jid=6952fc9dfd2b5768305b9b3a INFO: arguments: []
2022-03-23T17:41:25.266Z pid=23 tid=5t9f class=UserStatusCleanup::BatchWorker jid=6952fc9dfd2b5768305b9b3a elapsed=0.117 INFO: done
2022-03-23T17:42:04.522Z pid=23 tid=5stj class=ExpireBuildArtifactsWorker jid=428f9c49799e11ddf3a0fcdf INFO: start
2022-03-23T17:42:04.523Z pid=23 tid=5stj class=ExpireBuildArtifactsWorker jid=428f9c49799e11ddf3a0fcdf INFO: arguments: []
2022-03-23T17:42:04.548Z pid=23 tid=5ssr class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=5ed3fd9e01fb74eddceb6fd1 INFO: start
2022-03-23T17:42:04.552Z pid=23 tid=5ssr class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=5ed3fd9e01fb74eddceb6fd1 INFO: arguments: []
2022-03-23T17:42:04.611Z pid=23 tid=5ssr class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=5ed3fd9e01fb74eddceb6fd1 INFO: {:class=>"Database::BatchedBackgroundMigration::CiDatabaseWorker", :database=>"ci", :message=>"skipping migration execution for unconfigured database"}
2022-03-23T17:42:04.615Z pid=23 tid=5ssr class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=5ed3fd9e01fb74eddceb6fd1 elapsed=0.067 INFO: done
2022-03-23T17:42:04.618Z pid=23 tid=5t27 class=Database::BatchedBackgroundMigrationWorker jid=b6fac8589f0644878601625b INFO: start
2022-03-23T17:42:04.620Z pid=23 tid=5t27 class=Database::BatchedBackgroundMigrationWorker jid=b6fac8589f0644878601625b INFO: arguments: []
2022-03-23T17:42:04.646Z pid=23 tid=5t27 class=Database::BatchedBackgroundMigrationWorker jid=b6fac8589f0644878601625b elapsed=0.028 INFO: done
2022-03-23T17:42:04.649Z pid=23 tid=5t2f class=ScheduleMergeRequestCleanupRefsWorker jid=627f4c00096bf57241162691 INFO: start
2022-03-23T17:42:04.654Z pid=23 tid=5t2f class=ScheduleMergeRequestCleanupRefsWorker jid=627f4c00096bf57241162691 INFO: arguments: []
2022-03-23T17:42:04.674Z pid=23 tid=5t2f class=ScheduleMergeRequestCleanupRefsWorker jid=627f4c00096bf57241162691 elapsed=0.025 INFO: done
2022-03-23T17:42:04.675Z pid=23 tid=5t3n class=UserStatusCleanup::BatchWorker jid=612223fc97e66c1b749aa9df INFO: start
2022-03-23T17:42:04.678Z pid=23 tid=5t3n class=UserStatusCleanup::BatchWorker jid=612223fc97e66c1b749aa9df INFO: arguments: []
2022-03-23T17:42:04.832Z pid=23 tid=5t3n class=UserStatusCleanup::BatchWorker jid=612223fc97e66c1b749aa9df elapsed=0.157 INFO: done
*** /var/log/gitlab/application_json.log ***
{"severity":"INFO","time":"2022-03-23T17:42:04.836Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.844Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.851Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.858Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.866Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.872Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
*** /var/log/gitlab/application.log ***
2022-03-23T17:42:04.836Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.844Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.851Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.858Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.865Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.872Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.662Z pid=23 tid=5stj class=ExpireBuildArtifactsWorker jid=428f9c49799e11ddf3a0fcdf elapsed=1.141 INFO: done
*** /var/log/gitlab/application_json.log ***
{"severity":"INFO","time":"2022-03-23T17:42:04.880Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.887Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.894Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.901Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.908Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.916Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.928Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.936Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.944Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.953Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.969Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.977Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.986Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:04.993Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.003Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.013Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.023Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.033Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.042Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.052Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.060Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.068Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.076Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.084Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.092Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.108Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.120Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.127Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.135Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.144Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.152Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.159Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.167Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.175Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.183Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.195Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.202Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.210Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.219Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.229Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.237Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.244Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.252Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.260Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.267Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.274Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.284Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.291Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.298Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.306Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.314Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.321Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.328Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.335Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.343Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.350Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.357Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.367Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.376Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.383Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.392Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.399Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.407Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.415Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.423Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.432Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.442Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.450Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.458Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.465Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.474Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.481Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.488Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.495Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.503Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.511Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.519Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.526Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.533Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.540Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.548Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.559Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.569Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.578Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.585Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.592Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.600Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.615Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.623Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.631Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.638Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.646Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.652Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
{"severity":"INFO","time":"2022-03-23T17:42:05.660Z","correlation_id":"53a043bc901d176e8b30caf14971544b","message":"Fixed expire_at from artifacts.","fixed_artifacts_expire_at_count":0}
*** /var/log/gitlab/application.log ***
2022-03-23T17:42:04.880Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.886Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.893Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.901Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.908Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.916Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.927Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.936Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.944Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.953Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.969Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.977Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.986Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:04.993Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.003Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.012Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.023Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.033Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.042Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.052Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.060Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.068Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.076Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.084Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.092Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.108Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.120Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.127Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.135Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.144Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.151Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.159Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.166Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.175Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.183Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.195Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.202Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.210Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.219Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.229Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.237Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.244Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.252Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.260Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.267Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.274Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.284Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.291Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.298Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.306Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.314Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.321Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.328Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.335Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.343Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.350Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.357Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.367Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.376Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.383Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.392Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.399Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.407Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.414Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.423Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.431Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.442Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.450Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.458Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.465Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.473Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.481Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.488Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.495Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.503Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.511Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.519Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.526Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.533Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.540Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.548Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.559Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.569Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.578Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.585Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.592Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.600Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.615Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.623Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.630Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.638Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.645Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.652Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:05.659Z: {:message=>"Fixed expire_at from artifacts.", :fixed_artifacts_expire_at_count=>0}
2022-03-23T17:42:25.024Z pid=23 tid=5t9z class=ExternalServiceReactiveCachingWorker jid=45bf75a6724fd0f059eddd72 INFO: start
2022-03-23T17:42:25.026Z pid=23 tid=5t9z class=ExternalServiceReactiveCachingWorker jid=45bf75a6724fd0f059eddd72 INFO: arguments: ["VersionCheck","14.9.1"]
2022-03-23T17:42:25.459Z pid=23 tid=5t9z class=ExternalServiceReactiveCachingWorker jid=45bf75a6724fd0f059eddd72 WARN: Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in
Sidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today
by calling `Sidekiq.strict_args!` during Sidekiq initialization.
2022-03-23T17:42:25.463Z pid=23 tid=5t9z class=ExternalServiceReactiveCachingWorker jid=45bf75a6724fd0f059eddd72 elapsed=0.439 INFO: done
2022-03-23T17:42:42.808Z pid=23 tid=5ssr class=FlushCounterIncrementsWorker jid=408b9e529226b9f5bbed2f15 INFO: start
2022-03-23T17:42:42.810Z pid=23 tid=5ssr class=FlushCounterIncrementsWorker jid=408b9e529226b9f5bbed2f15 INFO: arguments: ["ProjectStatistics",11,"build_artifacts_size"]
2022-03-23T17:42:42.814Z pid=23 tid=5t27 class=FlushCounterIncrementsWorker jid=e514a68534bb2a557c59db41 INFO: start
2022-03-23T17:42:42.815Z pid=23 tid=5t27 class=FlushCounterIncrementsWorker jid=e514a68534bb2a557c59db41 INFO: arguments: ["ProjectStatistics",11,"storage_size"]
2022-03-23T17:42:42.870Z pid=23 tid=5t2f class=Namespaces::ScheduleAggregationWorker jid=e3a9e26b7c7966fb62eb9c34 INFO: start
2022-03-23T17:42:42.871Z pid=23 tid=5t27 class=FlushCounterIncrementsWorker jid=e514a68534bb2a557c59db41 INFO: {"retry"=>3, "queue"=>"update_namespace_statistics:namespaces_schedule_aggregation", "version"=>0, "queue_namespace"=>:update_namespace_statistics, "args"=>[3], "class"=>"Namespaces::ScheduleAggregationWorker", "jid"=>"d97423a2fcc9d10378e8f5c2", "created_at"=>1648057362.8695774, "correlation_id"=>"fec89fd6960038b273fa26447ff17df2", "meta.user"=>"crb", "meta.project"=>"tcl/puppet", "meta.root_namespace"=>"tcl", "meta.client_id"=>"user/7", "meta.caller_id"=>"FlushCounterIncrementsWorker", "meta.remote_ip"=>"18.133.253.30", "meta.feature_category"=>"source_code_management", "worker_data_consistency"=>:always, "idempotency_key"=>"resque:gitlab:duplicate:update_namespace_statistics:namespaces_schedule_aggregation:8d4335eec5dff1ae5e5d90cd6cea009e7ffec34a5d30af1132a51bca2b897ce4", "duplicate-of"=>"e3a9e26b7c7966fb62eb9c34", "job_size_bytes"=>3, "pid"=>23, "job_status"=>"deduplicated", "message"=>"Namespaces::ScheduleAggregationWorker JID-d97423a2fcc9d10378e8f5c2: deduplicated: dropped until executing", "deduplication.type"=>"dropped until executing"}
2022-03-23T17:42:42.872Z pid=23 tid=5t2f class=Namespaces::ScheduleAggregationWorker jid=e3a9e26b7c7966fb62eb9c34 INFO: arguments: [3]
2022-03-23T17:42:42.872Z pid=23 tid=5ssr class=FlushCounterIncrementsWorker jid=408b9e529226b9f5bbed2f15 elapsed=0.064 INFO: done
2022-03-23T17:42:42.876Z pid=23 tid=5t27 class=FlushCounterIncrementsWorker jid=e514a68534bb2a557c59db41 elapsed=0.062 INFO: done
2022-03-23T17:42:42.890Z pid=23 tid=5t2f class=Namespaces::ScheduleAggregationWorker jid=e3a9e26b7c7966fb62eb9c34 elapsed=0.02 INFO: done
2022-03-23T17:43:23.993Z pid=23 tid=5t9f class=PipelineScheduleWorker jid=9c75f444c26d6a02d5cff698 INFO: start
2022-03-23T17:43:23.995Z pid=23 tid=5t9f class=PipelineScheduleWorker jid=9c75f444c26d6a02d5cff698 INFO: arguments: []
2022-03-23T17:43:24.017Z pid=23 tid=5stj class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=8df22a9e508a667fa7f9d047 INFO: start
2022-03-23T17:43:24.019Z pid=23 tid=5stj class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=8df22a9e508a667fa7f9d047 INFO: arguments: []
2022-03-23T17:43:24.082Z pid=23 tid=5stj class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=8df22a9e508a667fa7f9d047 INFO: {:class=>"Database::BatchedBackgroundMigration::CiDatabaseWorker", :database=>"ci", :message=>"skipping migration execution for unconfigured database"}
2022-03-23T17:43:24.085Z pid=23 tid=5stj class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=8df22a9e508a667fa7f9d047 elapsed=0.068 INFO: done
2022-03-23T17:43:24.089Z pid=23 tid=5t9f class=PipelineScheduleWorker jid=9c75f444c26d6a02d5cff698 elapsed=0.096 INFO: done
2022-03-23T17:43:24.090Z pid=23 tid=5tjz class=Database::BatchedBackgroundMigrationWorker jid=1bdf7990df6951c03f8e9690 INFO: start
2022-03-23T17:43:24.092Z pid=23 tid=5tjz class=Database::BatchedBackgroundMigrationWorker jid=1bdf7990df6951c03f8e9690 INFO: arguments: []
2022-03-23T17:43:24.106Z pid=23 tid=5tjz class=Database::BatchedBackgroundMigrationWorker jid=1bdf7990df6951c03f8e9690 elapsed=0.016 INFO: done
2022-03-23T17:43:24.108Z pid=23 tid=5tk7 class=ScheduleMergeRequestCleanupRefsWorker jid=2d2c1bae3b162ba393b4865b INFO: start
2022-03-23T17:43:24.110Z pid=23 tid=5tk7 class=ScheduleMergeRequestCleanupRefsWorker jid=2d2c1bae3b162ba393b4865b INFO: arguments: []
2022-03-23T17:43:24.129Z pid=23 tid=5tk7 class=ScheduleMergeRequestCleanupRefsWorker jid=2d2c1bae3b162ba393b4865b elapsed=0.021 INFO: done
2022-03-23T17:43:24.132Z pid=23 tid=5tlf class=UserStatusCleanup::BatchWorker jid=c0fcf023f65ea38a7bd85a30 INFO: start
2022-03-23T17:43:24.134Z pid=23 tid=5tlf class=UserStatusCleanup::BatchWorker jid=c0fcf023f65ea38a7bd85a30 INFO: arguments: []
2022-03-23T17:43:24.248Z pid=23 tid=5tlf class=UserStatusCleanup::BatchWorker jid=c0fcf023f65ea38a7bd85a30 elapsed=0.115 INFO: done
2022-03-23T17:43:27.408Z pid=23 tid=5t9z class=ExternalServiceReactiveCachingWorker jid=35e4fbefcfe267f77dd75c8d INFO: start
2022-03-23T17:43:27.410Z pid=23 tid=5t9z class=ExternalServiceReactiveCachingWorker jid=35e4fbefcfe267f77dd75c8d INFO: arguments: ["VersionCheck","14.9.1"]
2022-03-23T17:43:27.847Z pid=23 tid=5t9z class=ExternalServiceReactiveCachingWorker jid=35e4fbefcfe267f77dd75c8d WARN: Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in
Sidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today
by calling `Sidekiq.strict_args!` during Sidekiq initialization.
2022-03-23T17:43:27.851Z pid=23 tid=5t9z class=ExternalServiceReactiveCachingWorker jid=35e4fbefcfe267f77dd75c8d elapsed=0.443 INFO: done
2022-03-23T17:44:26.678Z pid=23 tid=5tlf class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=8aecbd77c2a684ce6e57cdb3 INFO: start
2022-03-23T17:44:26.679Z pid=23 tid=5tlf class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=8aecbd77c2a684ce6e57cdb3 INFO: arguments: []
2022-03-23T17:44:26.741Z pid=23 tid=5tlf class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=8aecbd77c2a684ce6e57cdb3 INFO: {:class=>"Database::BatchedBackgroundMigration::CiDatabaseWorker", :database=>"ci", :message=>"skipping migration execution for unconfigured database"}
2022-03-23T17:44:26.742Z pid=23 tid=5tlf class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=8aecbd77c2a684ce6e57cdb3 elapsed=0.064 INFO: done
2022-03-23T17:44:26.745Z pid=23 tid=5tmj class=Database::BatchedBackgroundMigrationWorker jid=11e5a86b7987e11916d08c6a INFO: start
2022-03-23T17:44:26.746Z pid=23 tid=5tmj class=Database::BatchedBackgroundMigrationWorker jid=11e5a86b7987e11916d08c6a INFO: arguments: []
2022-03-23T17:44:26.761Z pid=23 tid=5tmj class=Database::BatchedBackgroundMigrationWorker jid=11e5a86b7987e11916d08c6a elapsed=0.016 INFO: done
2022-03-23T17:44:26.763Z pid=23 tid=5tmr class=ScheduleMergeRequestCleanupRefsWorker jid=dded67c539e46546b5c918ad INFO: start
2022-03-23T17:44:26.781Z pid=23 tid=5tmr class=ScheduleMergeRequestCleanupRefsWorker jid=dded67c539e46546b5c918ad INFO: arguments: []
2022-03-23T17:44:26.786Z pid=23 tid=5tof class=UserStatusCleanup::BatchWorker jid=5413e7a17d08d1650bf9b915 INFO: start
2022-03-23T17:44:26.787Z pid=23 tid=5tmr class=ScheduleMergeRequestCleanupRefsWorker jid=dded67c539e46546b5c918ad elapsed=0.024 INFO: done
2022-03-23T17:44:26.788Z pid=23 tid=5tof class=UserStatusCleanup::BatchWorker jid=5413e7a17d08d1650bf9b915 INFO: arguments: []
2022-03-23T17:44:26.900Z pid=23 tid=5tof class=UserStatusCleanup::BatchWorker jid=5413e7a17d08d1650bf9b915 elapsed=0.114 INFO: done
2022-03-23T17:44:29.457Z pid=23 tid=5ton class=ExternalServiceReactiveCachingWorker jid=bbc31d509e994c0d926e3323 INFO: start
2022-03-23T17:44:29.458Z pid=23 tid=5ton class=ExternalServiceReactiveCachingWorker jid=bbc31d509e994c0d926e3323 INFO: arguments: ["VersionCheck","14.9.1"]
2022-03-23T17:44:29.881Z pid=23 tid=5ton class=ExternalServiceReactiveCachingWorker jid=bbc31d509e994c0d926e3323 WARN: Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in
Sidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today
by calling `Sidekiq.strict_args!` during Sidekiq initialization.
2022-03-23T17:44:29.885Z pid=23 tid=5ton class=ExternalServiceReactiveCachingWorker jid=bbc31d509e994c0d926e3323 elapsed=0.428 INFO: done
2022-03-23T17:45:18.133Z pid=23 tid=5ton class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=ebe5179536833472615d9076 INFO: start
2022-03-23T17:45:18.135Z pid=23 tid=5ton class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=ebe5179536833472615d9076 INFO: arguments: []
2022-03-23T17:45:18.165Z pid=23 tid=5ton class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=ebe5179536833472615d9076 INFO: {:class=>"Database::BatchedBackgroundMigration::CiDatabaseWorker", :database=>"ci", :message=>"skipping migration execution for unconfigured database"}
2022-03-23T17:45:18.168Z pid=23 tid=5ton class=Database::BatchedBackgroundMigration::CiDatabaseWorker jid=ebe5179536833472615d9076 elapsed=0.035 INFO: done
2022-03-23T17:45:18.174Z pid=23 tid=5t4z class=PagesDomainVerificationCronWorker jid=f1536a5c79acc1435e1ef56a INFO: start
2022-03-23T17:45:18.176Z pid=23 tid=5t4z class=PagesDomainVerificationCronWorker jid=f1536a5c79acc1435e1ef56a INFO: arguments: []
2022-03-23T17:45:18.198Z pid=23 tid=5t4z class=PagesDomainVerificationCronWorker jid=f1536a5c79acc1435e1ef56a elapsed=0.023 INFO: done
2022-03-23T17:45:18.198Z pid=23 tid=5t9z class=Issues::RescheduleStuckIssueRebalancesWorker jid=e69dac27ff78cff2ab15e188 INFO: start
2022-03-23T17:45:18.200Z pid=23 tid=5t9z class=Issues::RescheduleStuckIssueRebalancesWorker jid=e69dac27ff78cff2ab15e188 INFO: arguments: []
2022-03-23T17:45:18.227Z pid=23 tid=5t9z class=Issues::RescheduleStuckIssueRebalancesWorker jid=e69dac27ff78cff2ab15e188 elapsed=0.029 INFO: done
2022-03-23T17:45:18.229Z pid=23 tid=5tk7 class=Database::BatchedBackgroundMigrationWorker jid=3ca6b1e62d82045b2d632f02 INFO: start
2022-03-23T17:45:18.231Z pid=23 tid=5tk7 class=Database::BatchedBackgroundMigrationWorker jid=3ca6b1e62d82045b2d632f02 INFO: arguments: []
2022-03-23T17:45:18.244Z pid=23 tid=5tk7 class=Database::BatchedBackgroundMigrationWorker jid=3ca6b1e62d82045b2d632f02 elapsed=0.015 INFO: done
2022-03-23T17:45:18.245Z pid=23 tid=5tmr class=ScheduleMergeRequestCleanupRefsWorker jid=b7c419346bf99c0fbefe4158 INFO: start
2022-03-23T17:45:18.247Z pid=23 tid=5tmr class=ScheduleMergeRequestCleanupRefsWorker jid=b7c419346bf99c0fbefe4158 INFO: arguments: []
2022-03-23T17:45:18.265Z pid=23 tid=5tmr class=ScheduleMergeRequestCleanupRefsWorker jid=b7c419346bf99c0fbefe4158 elapsed=0.019 INFO: done
2022-03-23T17:45:18.283Z pid=23 tid=5stj class=UserStatusCleanup::BatchWorker jid=5f831baa002ad39fbd856d91 INFO: start
2022-03-23T17:45:18.285Z pid=23 tid=5stj class=UserStatusCleanup::BatchWorker jid=5f831baa002ad39fbd856d91 INFO: arguments: []
2022-03-23T17:45:18.294Z pid=23 tid=5stj class=UserStatusCleanup::BatchWorker jid=5f831baa002ad39fbd856d91 elapsed=0.011 INFO: done
2022-03-23T17:45:18.296Z pid=23 tid=5t9f class=ContainerRegistry::Migration::EnqueuerWorker jid=4c57ac9abaf0e8d3edc3116a INFO: start
2022-03-23T17:45:18.298Z pid=23 tid=5t9f class=ContainerRegistry::Migration::EnqueuerWorker jid=4c57ac9abaf0e8d3edc3116a INFO: arguments: []
2022-03-23T17:45:18.422Z pid=23 tid=5t9f class=ContainerRegistry::Migration::EnqueuerWorker jid=4c57ac9abaf0e8d3edc3116a elapsed=0.125 INFO: done
2022-03-23T17:45:27.009Z pid=23 tid=5tlf class=UpdateProjectStatisticsWorker jid=c5abc6f75dba1158c932f31f INFO: start
2022-03-23T17:45:27.011Z pid=23 tid=5tlf class=UpdateProjectStatisticsWorker jid=c5abc6f75dba1158c932f31f INFO: arguments: [11,["repository_size","commit_count"]]
2022-03-23T17:45:27.138Z pid=23 tid=5tmj class=Namespaces::ScheduleAggregationWorker jid=e7d73bd9c67c324d76237fe0 INFO: start
2022-03-23T17:45:27.139Z pid=23 tid=5tlf class=UpdateProjectStatisticsWorker jid=c5abc6f75dba1158c932f31f elapsed=0.13 INFO: done
2022-03-23T17:45:27.140Z pid=23 tid=5tmj class=Namespaces::ScheduleAggregationWorker jid=e7d73bd9c67c324d76237fe0 INFO: arguments: [3]
2022-03-23T17:45:27.150Z pid=23 tid=5tmj class=Namespaces::ScheduleAggregationWorker jid=e7d73bd9c67c324d76237fe0 elapsed=0.012 INFO: done
*** /var/log/gitlab/application_json.log ***
{"severity":"INFO","time":"2022-03-23T17:45:27.021Z","correlation_id":"c9d4410898daf83d7e6d50783225688b","message":"Updating statistics for project 11"}
*** /var/log/gitlab/application.log ***
2022-03-23T17:45:27.021Z: Updating statistics for project 11
2022-03-23T17:45:32.144Z pid=23 tid=5t3n class=UpdateProjectStatisticsWorker jid=d071f8312c3f25cfe4fa3281 INFO: start
2022-03-23T17:45:32.145Z pid=23 tid=5t3n class=UpdateProjectStatisticsWorker jid=d071f8312c3f25cfe4fa3281 INFO: arguments: [11,["repository_size"]]
2022-03-23T17:45:32.148Z pid=23 tid=5t4z class=ExternalServiceReactiveCachingWorker jid=4302576581b7c0ab10d0eebe INFO: start
2022-03-23T17:45:32.151Z pid=23 tid=5t4z class=ExternalServiceReactiveCachingWorker jid=4302576581b7c0ab10d0eebe INFO: arguments: ["VersionCheck","14.9.1"]
2022-03-23T17:45:32.213Z pid=23 tid=5ton class=Namespaces::ScheduleAggregationWorker jid=9a428038d5b004565f6265ad INFO: start
2022-03-23T17:45:32.215Z pid=23 tid=5t3n class=UpdateProjectStatisticsWorker jid=d071f8312c3f25cfe4fa3281 elapsed=0.071 INFO: done
2022-03-23T17:45:32.215Z pid=23 tid=5ton class=Namespaces::ScheduleAggregationWorker jid=9a428038d5b004565f6265ad INFO: arguments: [3]
2022-03-23T17:45:32.225Z pid=23 tid=5ton class=Namespaces::ScheduleAggregationWorker jid=9a428038d5b004565f6265ad elapsed=0.012 INFO: done
2022-03-23T17:45:32.602Z pid=23 tid=5t4z class=ExternalServiceReactiveCachingWorker jid=4302576581b7c0ab10d0eebe WARN: Job arguments to ExternalServiceReactiveCachingWorker do not serialize to JSON safely. This will raise an error in
Sidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today
by calling `Sidekiq.strict_args!` during Sidekiq initialization.
2022-03-23T17:45:32.606Z pid=23 tid=5t4z class=ExternalServiceReactiveCachingWorker jid=4302576581b7c0ab10d0eebe elapsed=0.458 INFO: done
*** /var/log/gitlab/application_json.log ***
{"severity":"INFO","time":"2022-03-23T17:45:32.152Z","correlation_id":"292b96e72d398c29c0591072b2c2a012","message":"Updating statistics for project 11"}
Name: gitlab-sidekiq-all-in-1-v2-74468d844b-vmw4n
Namespace: gitlab
Priority: 0
Node: ip-10-195-26-142.eu-west-2.compute.internal/10.195.26.142
Start Time: Wed, 23 Mar 2022 17:39:07 +0000
Labels: app=sidekiq
chart=sidekiq-5.9.1
heritage=Helm
pod-template-hash=74468d844b
queue-pod-name=all-in-1
release=gitlab
Annotations: checksum/configmap: ad8e31196773705cfa779b846798452191f8566c887049921e847e78bc873679
cluster-autoscaler.kubernetes.io/safe-to-evict: true
gitlab.com/prometheus_path: /metrics
gitlab.com/prometheus_port: 3807
gitlab.com/prometheus_scrape: true
iam.amazonaws.com/role: cfn-prod-gitlab-resources-GitLabRole-LJ8LT3HOGSLA
kubernetes.io/psp: eks.privileged
prometheus.io/path: /metrics
prometheus.io/port: 3807
prometheus.io/scrape: true
Status: Running
IP: 10.195.24.102
IPs:
IP: 10.195.24.102
Controlled By: ReplicaSet/gitlab-sidekiq-all-in-1-v2-74468d844b
Init Containers:
certificates:
Container ID: docker://e66d7028a6276b9244987b767a871ee658fbad33613e45b2a4e45c4ed2383797
Image: registry.gitlab.com/gitlab-org/build/cng/alpine-certificates:20191127-r2@sha256:4678ac2a66f126b20362faddd333be907d4eded47a7fb8ea2653c1522ddbed49
Image ID: docker-pullable://registry.gitlab.com/gitlab-org/build/cng/alpine-certificates@sha256:4678ac2a66f126b20362faddd333be907d4eded47a7fb8ea2653c1522ddbed49
Port: <none>
Host Port: <none>
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 23 Mar 2022 17:39:10 +0000
Finished: Wed, 23 Mar 2022 17:39:11 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 50m
Environment: <none>
Mounts:
/etc/ssl/certs from etc-ssl-certs (rw)
configure:
Container ID: docker://c9536efc482d434bb258e1ce26507baba718c28b9f874bc4a4ff4a9c28bfdfcb
Image: registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox:latest
Image ID: docker-pullable://registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox@sha256:139abcf41943b8bcd4bc5c42ee71ddc9402c7ad69ad9e177b0a9bc4541f14924
Port: <none>
Host Port: <none>
Command:
sh
/config/configure
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 23 Mar 2022 17:39:13 +0000
Finished: Wed, 23 Mar 2022 17:39:13 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 50m
Environment: <none>
Mounts:
/config from sidekiq-config (ro)
/init-config from init-sidekiq-secrets (ro)
/init-secrets from sidekiq-secrets (rw)
dependencies:
Container ID: docker://bb6537128af5c336502f809648caef8521f7331013c8beaf8d4a348fd7cd0efe
Image: registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce:v14.9.1
Image ID: docker-pullable://registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce@sha256:bb584787e0059849d07ae5a11d758725676c71896554c95cd3027244d39585a3
Port: <none>
Host Port: <none>
Args:
/scripts/wait-for-deps
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 23 Mar 2022 17:39:14 +0000
Finished: Wed, 23 Mar 2022 17:39:21 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 50m
Environment:
GITALY_FEATURE_DEFAULT_ON: 1
CONFIG_TEMPLATE_DIRECTORY: /var/opt/gitlab/templates
CONFIG_DIRECTORY: /srv/gitlab/config
SIDEKIQ_CONCURRENCY: 25
SIDEKIQ_TIMEOUT: 25
ENABLE_BOOTSNAP: 1
Mounts:
/etc/gitlab from sidekiq-secrets (ro)
/etc/ssl/certs/ from etc-ssl-certs (ro)
/srv/gitlab/config/secrets.yml from sidekiq-secrets (ro,path="rails-secrets/secrets.yml")
/var/opt/gitlab/templates from sidekiq-config (ro)
Containers:
sidekiq:
Container ID: docker://10f5ab95d312ad3e085497c2350781065f5012b7b097c76ade3e1c95210dfc3a
Image: registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce:v14.9.1
Image ID: docker-pullable://registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce@sha256:bb584787e0059849d07ae5a11d758725676c71896554c95cd3027244d39585a3
Port: 3807/TCP
Host Port: 0/TCP
State: Running
Started: Wed, 23 Mar 2022 17:39:23 +0000
Ready: False
Restart Count: 0
Requests:
cpu: 900m
memory: 2G
Liveness: http-get http://:3807/liveness delay=30s timeout=30s period=15s #success=1 #failure=120
Readiness: http-get http://:3807/readiness delay=0s timeout=2s period=10s #success=1 #failure=3
Environment:
prometheus_multiproc_dir: /metrics
GITALY_FEATURE_DEFAULT_ON: 1
CONFIG_TEMPLATE_DIRECTORY: /var/opt/gitlab/templates
CONFIG_DIRECTORY: /srv/gitlab/config
SIDEKIQ_QUEUE_SELECTOR:
SIDEKIQ_CONCURRENCY: 25
SIDEKIQ_TIMEOUT: 25
SIDEKIQ_QUEUES:
SIDEKIQ_NEGATE_QUEUES:
SIDEKIQ_DAEMON_MEMORY_KILLER: 1
SIDEKIQ_MEMORY_KILLER_CHECK_INTERVAL: 3
SIDEKIQ_MEMORY_KILLER_MAX_RSS: 2000000
SIDEKIQ_MEMORY_KILLER_GRACE_TIME: 900
SIDEKIQ_MEMORY_KILLER_SHUTDOWN_WAIT: 30
ENABLE_BOOTSNAP: 1
Mounts:
/etc/gitlab from sidekiq-secrets (ro)
/etc/ssl/certs/ from etc-ssl-certs (ro)
/metrics from sidekiq-metrics (rw)
/srv/gitlab/INSTALLATION_TYPE from sidekiq-config (rw,path="installation_type")
/srv/gitlab/config/initializers/smtp_settings.rb from sidekiq-config (rw,path="smtp_settings.rb")
/srv/gitlab/config/secrets.yml from sidekiq-secrets (rw,path="rails-secrets/secrets.yml")
/var/opt/gitlab/templates from sidekiq-config (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
sidekiq-metrics:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
sidekiq-config:
Type: Projected (a volume that contains injected data from multiple sources)
ConfigMapName: gitlab-sidekiq
ConfigMapOptional: <nil>
init-sidekiq-secrets:
Type: Projected (a volume that contains injected data from multiple sources)
SecretName: gitlab-rails-secret
SecretOptionalName: <nil>
SecretName: gitlab-gitaly-secret
SecretOptionalName: <nil>
SecretName: gitlab.gitlab-postgres-12.credentials
SecretOptionalName: <nil>
SecretName: gitlab-registry-secret
SecretOptionalName: <nil>
SecretName: gitlab-registry-notification
SecretOptionalName: <nil>
SecretName: gitlab-s3
SecretOptionalName: <nil>
SecretName: gitlab-omniauth-saml
SecretOptionalName: <nil>
SecretName: gitlab-smtp
SecretOptionalName: <nil>
sidekiq-secrets:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
etc-ssl-certs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned gitlab/gitlab-sidekiq-all-in-1-v2-74468d844b-vmw4n to ip-10-195-26-142.eu-west-2.compute.internal
Warning FailedMount 23m kubelet MountVolume.SetUp failed for volume "sidekiq-config" : failed to sync configmap cache: timed out waiting for the condition
Warning FailedMount 23m kubelet MountVolume.SetUp failed for volume "init-sidekiq-secrets" : failed to sync secret cache: timed out waiting for the condition
Normal Pulled 23m kubelet Container image "registry.gitlab.com/gitlab-org/build/cng/alpine-certificates:20191127-r2@sha256:4678ac2a66f126b20362faddd333be907d4eded47a7fb8ea2653c1522ddbed49" already present on machine
Normal Created 23m kubelet Created container certificates
Normal Started 23m kubelet Started container certificates
Normal Pulling 23m kubelet Pulling image "registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox:latest"
Normal Pulled 23m kubelet Successfully pulled image "registry.gitlab.com/gitlab-org/cloud-native/mirror/images/busybox:latest" in 965.731758ms
Normal Created 23m kubelet Created container configure
Normal Started 23m kubelet Started container configure
Normal Pulled 23m kubelet Container image "registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce:v14.9.1" already present on machine
Normal Created 23m kubelet Created container dependencies
Normal Started 23m kubelet Started container dependencies
Normal Pulled 23m kubelet Container image "registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce:v14.9.1" already present on machine
Normal Created 23m kubelet Created container sidekiq
Normal Started 23m kubelet Started container sidekiq
Warning Unhealthy 22m (x3 over 22m) kubelet Liveness probe failed: Get "http://10.195.24.102:3807/liveness": dial tcp 10.195.24.102:3807: connect: connection refused
Warning Unhealthy 3m30s (x120 over 23m) kubelet Readiness probe failed: Get "http://10.195.24.102:3807/readiness": dial tcp 10.195.24.102:3807: connect: connection refused