CI job will indicate it is running, but I don't suspect it actually is
Summary
GitLab.com maintains an Ops instance in order to sync various items and provide operations should .com go down. SOMETIMES a CI jobs will go into a state of Running, but after a long time has passed, no details whatsoever of said job ever show up, such as the logs for the CI job. Utilize this issue to help troubleshoot why this happens. Historically when I personally have run into this, I shrug it off and restart/retry the job that has no details. This happens often enough we should attempt to learn why.
Steps to reproduce
This happens sporadically when a CI job is picked up. Unsure how to reproduce this consistently.
What is the current bug behavior?
What is the expected correct behavior?
We should see details and the job show the appropriate status code.
Relevant logs and/or screenshots
Job json
{
"id": 4924912,
"name": "gstg:upgrade",
"started": "2021-09-20T15:43:05.629Z",
"complete": false,
"archived": false,
"build_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/4924912",
"cancel_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/4924912/cancel?continue%5Bto%5D=%2Fgitlab-com%2Fgl-infra%2Fk8s-workloads%2Fgitlab-com%2F-%2Fjobs%2F4924912",
"playable": false,
"scheduled": false,
"created_at": "2021-09-20T15:40:23.818Z",
"updated_at": "2021-09-20T15:43:05.630Z",
"status": {
"icon": "status_running",
"text": "running",
"label": "running",
"group": "running",
"tooltip": "running",
"has_details": true,
"details_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/4924912",
"illustration": {
"image": "/assets/illustrations/skipped-job_empty-29a8a37d8a61d1b6f68cf3484f9024e53cd6eb95e28eae3554f8011a1146bf27.svg",
"size": "svg-430",
"title": "This job does not have a trace."
},
"favicon": "/assets/ci_favicons/favicon_status_running-9c635b2419a8e1ec991c993061b89cc5aefc0743bb238ecd0c381e7741a70e8c.png",
"action": {
"icon": "cancel",
"title": "Cancel",
"path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/4924912/cancel",
"method": "post",
"button_title": "Cancel this job"
}
},
"coverage": null,
"erased_at": null,
"duration": 3961.874594184,
"tags": [
"k8s-workloads"
],
"has_trace": false,
"stage": "non-prod:deploy",
"stuck": false,
"user": {
"id": 30,
"name": "ops-gitlab-net",
"username": "ops-gitlab-net",
"state": "active",
"avatar_url": "https://secure.gravatar.com/avatar/68056dafa382a8ed653fa1a8f97425f4?s=80&d=identicon",
"web_url": "https://ops.gitlab.net/ops-gitlab-net",
"show_status": false,
"path": "/ops-gitlab-net"
},
"runner": {
"id": 14,
"description": "runner-01-inf-ops",
"short_sha": "538fe8a5"
},
"metadata": {
"timeout_human_readable": "2h",
"timeout_source": "project"
},
"pipeline": {
"id": 802342,
"user": {
"id": 30,
"name": "ops-gitlab-net",
"username": "ops-gitlab-net",
"state": "active",
"avatar_url": "https://secure.gravatar.com/avatar/68056dafa382a8ed653fa1a8f97425f4?s=80&d=identicon",
"web_url": "https://ops.gitlab.net/ops-gitlab-net",
"show_status": false,
"path": "/ops-gitlab-net"
},
"active": true,
"coverage": null,
"source": "push",
"created_at": "2021-09-20T15:40:23.413Z",
"updated_at": "2021-09-20T15:43:04.595Z",
"path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342",
"flags": {
"stuck": false,
"auto_devops": false,
"merge_request": false,
"yaml_errors": false,
"retryable": false,
"cancelable": true,
"failure_reason": false,
"detached_merge_request_pipeline": false,
"merge_request_pipeline": false,
"merge_train_pipeline": false
},
"details": {
"status": {
"icon": "status_running",
"text": "running",
"label": "running",
"group": "running",
"tooltip": "running",
"has_details": true,
"details_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342",
"illustration": null,
"favicon": "/assets/ci_favicons/favicon_status_running-9c635b2419a8e1ec991c993061b89cc5aefc0743bb238ecd0c381e7741a70e8c.png"
},
"stages": [
{
"name": "check",
"title": "check: passed",
"status": {
"icon": "status_success",
"text": "passed",
"label": "passed",
"group": "success",
"tooltip": "passed",
"has_details": true,
"details_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342#check",
"illustration": null,
"favicon": "/assets/ci_favicons/favicon_status_success-8451333011eee8ce9f2ab25dc487fe24a8758c694827a582f17f42b0a90446a2.png"
},
"path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342#check",
"dropdown_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342/stage.json?stage=check"
},
{
"name": "dryrun",
"title": "dryrun: passed",
"status": {
"icon": "status_success",
"text": "passed",
"label": "passed",
"group": "success",
"tooltip": "passed",
"has_details": true,
"details_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342#dryrun",
"illustration": null,
"favicon": "/assets/ci_favicons/favicon_status_success-8451333011eee8ce9f2ab25dc487fe24a8758c694827a582f17f42b0a90446a2.png"
},
"path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342#dryrun",
"dropdown_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342/stage.json?stage=dryrun"
},
{
"name": "non-prod-cny:deploy",
"title": "non-prod-cny:deploy: passed",
"status": {
"icon": "status_success",
"text": "passed",
"label": "passed",
"group": "success",
"tooltip": "passed",
"has_details": true,
"details_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342#non-prod-cny:deploy",
"illustration": null,
"favicon": "/assets/ci_favicons/favicon_status_success-8451333011eee8ce9f2ab25dc487fe24a8758c694827a582f17f42b0a90446a2.png"
},
"path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342#non-prod-cny:deploy",
"dropdown_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342/stage.json?stage=non-prod-cny%3Adeploy"
},
{
"name": "non-prod:deploy",
"title": "non-prod:deploy: running",
"status": {
"icon": "status_running",
"text": "running",
"label": "running",
"group": "running",
"tooltip": "running",
"has_details": true,
"details_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342#non-prod:deploy",
"illustration": null,
"favicon": "/assets/ci_favicons/favicon_status_running-9c635b2419a8e1ec991c993061b89cc5aefc0743bb238ecd0c381e7741a70e8c.png"
},
"path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342#non-prod:deploy",
"dropdown_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342/stage.json?stage=non-prod%3Adeploy"
},
{
"name": "gprd-cny:deploy",
"title": "gprd-cny:deploy: created",
"status": {
"icon": "status_created",
"text": "created",
"label": "created",
"group": "created",
"tooltip": "created",
"has_details": true,
"details_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342#gprd-cny:deploy",
"illustration": null,
"favicon": "/assets/ci_favicons/favicon_status_created-4b975aa976d24e5a3ea7cd9a5713e6ce2cd9afd08b910415e96675de35f64955.png"
},
"path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342#gprd-cny:deploy",
"dropdown_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342/stage.json?stage=gprd-cny%3Adeploy"
},
{
"name": "gprd:deploy:alpha",
"title": "gprd:deploy:alpha: created",
"status": {
"icon": "status_created",
"text": "created",
"label": "created",
"group": "created",
"tooltip": "created",
"has_details": true,
"details_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342#gprd:deploy:alpha",
"illustration": null,
"favicon": "/assets/ci_favicons/favicon_status_created-4b975aa976d24e5a3ea7cd9a5713e6ce2cd9afd08b910415e96675de35f64955.png"
},
"path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342#gprd:deploy:alpha",
"dropdown_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342/stage.json?stage=gprd%3Adeploy%3Aalpha"
},
{
"name": "gprd:deploy:beta",
"title": "gprd:deploy:beta: created",
"status": {
"icon": "status_created",
"text": "created",
"label": "created",
"group": "created",
"tooltip": "created",
"has_details": true,
"details_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342#gprd:deploy:beta",
"illustration": null,
"favicon": "/assets/ci_favicons/favicon_status_created-4b975aa976d24e5a3ea7cd9a5713e6ce2cd9afd08b910415e96675de35f64955.png"
},
"path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342#gprd:deploy:beta",
"dropdown_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342/stage.json?stage=gprd%3Adeploy%3Abeta"
}
],
"duration": null,
"finished_at": null,
"name": "Pipeline"
},
"ref": {
"name": "master",
"path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/commits/master",
"tag": false,
"branch": true,
"merge_request": false
},
"commit": {
"id": "0fe370a48cb802ab501cf8a7c709a6ee759617f1",
"short_id": "0fe370a4",
"created_at": "2021-09-20T15:40:13.000+00:00",
"parent_ids": [
"14d26ea68455ea0118a441847b8668bf97c55c59",
"aca475f1df40bb7df1bd7830b22725720d39acda"
],
"title": "Merge branch 'jts/bump-chart-6' into 'master'",
"message": "Merge branch 'jts/bump-chart-6' into 'master'\n\nBumps Chart version in preprod\n\nSee merge request gitlab-com/gl-infra/k8s-workloads/gitlab-com!1244",
"author_name": "John Skarbek",
"author_email": "jskarbek@gitlab.com",
"authored_date": "2021-09-20T15:40:13.000+00:00",
"committer_name": "John Skarbek",
"committer_email": "jskarbek@gitlab.com",
"committed_date": "2021-09-20T15:40:13.000+00:00",
"web_url": "https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/commit/0fe370a48cb802ab501cf8a7c709a6ee759617f1",
"author": {
"id": 25,
"name": "John Skarbek",
"username": "skarbek",
"state": "active",
"avatar_url": "https://secure.gravatar.com/avatar/c6455ce4726023aac6dd7fb0acd1f236?s=80&d=identicon",
"web_url": "https://ops.gitlab.net/skarbek",
"show_status": false,
"path": "/skarbek"
},
"author_gravatar_url": "https://secure.gravatar.com/avatar/c6455ce4726023aac6dd7fb0acd1f236?s=80&d=identicon",
"commit_url": "https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/commit/0fe370a48cb802ab501cf8a7c709a6ee759617f1",
"commit_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/commit/0fe370a48cb802ab501cf8a7c709a6ee759617f1"
},
"cancel_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342/cancel",
"delete_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/pipelines/802342"
},
"deployment_status": {
"status": "creating",
"environment": {
"id": 90,
"global_id": "gid://gitlab/Environment/90",
"name": "gstg",
"state": "available",
"external_url": "https://staging.gitlab.com",
"environment_type": null,
"name_without_type": "gstg",
"last_deployment": {
"id": 179451,
"iid": 67726,
"sha": "0fe370a48cb802ab501cf8a7c709a6ee759617f1",
"ref": {
"name": "master",
"ref_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/tree/master"
},
"status": "success",
"created_at": "2021-09-20T15:40:23.764Z",
"deployed_at": "2021-09-20T15:43:03.260Z",
"tag": false,
"last?": true,
"user": {
"id": 30,
"name": "ops-gitlab-net",
"username": "ops-gitlab-net",
"state": "active",
"avatar_url": "https://secure.gravatar.com/avatar/68056dafa382a8ed653fa1a8f97425f4?s=80&d=identicon",
"web_url": "https://ops.gitlab.net/ops-gitlab-net",
"show_status": false,
"path": "/ops-gitlab-net"
},
"deployable": {
"name": "gstg-cny:upgrade",
"build_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/4924910"
},
"cluster": {
"name": "gstg-gitlab-gke",
"path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/clusters/21",
"kubernetes_namespace": "gitlab"
}
},
"has_stop_action": false,
"rollout_status": {
"status": "found",
"has_legacy_app_label": false,
"instances": [
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
},
{
"status": "pending",
"pod_name": "Not provided",
"tooltip": "Not provided (Pending)",
"track": "stable",
"stable": true
}
],
"completion": 0,
"is_completed": false
},
"upcoming_deployment": {
"id": 179453,
"iid": 67728,
"sha": "0fe370a48cb802ab501cf8a7c709a6ee759617f1",
"ref": {
"name": "master",
"ref_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/tree/master"
},
"status": "running",
"created_at": "2021-09-20T15:40:23.830Z",
"deployed_at": null,
"tag": false,
"last?": false,
"user": {
"id": 30,
"name": "ops-gitlab-net",
"username": "ops-gitlab-net",
"state": "active",
"avatar_url": "https://secure.gravatar.com/avatar/68056dafa382a8ed653fa1a8f97425f4?s=80&d=identicon",
"web_url": "https://ops.gitlab.net/ops-gitlab-net",
"show_status": false,
"path": "/ops-gitlab-net"
},
"deployable": {
"name": "gstg:upgrade",
"build_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/4924912"
}
},
"metrics_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/environments/90/metrics",
"environment_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/environments/90",
"stop_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/environments/90/stop",
"cancel_auto_stop_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/environments/90/cancel_auto_stop",
"delete_path": "/api/v4/projects/221/environments/90",
"cluster_type": "project_type",
"terminal_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/environments/90/terminal",
"folder_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/environments/folders/gstg",
"created_at": "2019-06-26T16:31:02.640Z",
"updated_at": "2021-03-23T03:31:41.079Z",
"can_stop": true,
"logs_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/logs?environment_name=gstg",
"logs_api_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/logs/k8s.json?environment_name=gstg",
"enable_advanced_logs_querying": false,
"can_delete": false,
"has_opened_alert": false
}
},
"deployment_cluster": {
"name": "gstg-gitlab-gke",
"path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/clusters/21",
"kubernetes_namespace": "gitlab"
},
"artifact": {
"locked": true
},
"reports": [],
"raw_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/jobs/4924912/raw",
"runners": {
"online": true,
"available": true,
"settings_path": "/gitlab-com/gl-infra/k8s-workloads/gitlab-com/runners"
}
}
Runner Logs
Sep 20 15:40:39 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924899 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:39 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924899 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:40 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924900 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:40 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924900 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:41 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924901 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:41 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924901 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:41 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924902 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:41 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924902 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:42 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924903 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:42 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924903 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:42 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924905 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:42 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924905 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:42 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924904 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:42 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924904 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:43 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924906 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:43 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924906 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:43 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924907 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:43 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924907 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:44 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924908 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:44 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924908 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:44 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924909 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:40:44 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924909 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:41:41 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m2.081635692s job=4924899 project=221 runner=538fe8a5
Sep 20 15:41:41 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m2.081635692s job=4924899 project=221 runner=538fe8a5
Sep 20 15:41:44 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m4.507275795s job=4924900 project=221 runner=538fe8a5
Sep 20 15:41:44 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m4.507275795s job=4924900 project=221 runner=538fe8a5
Sep 20 15:41:46 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m4.512293106s job=4924903 project=221 runner=538fe8a5
Sep 20 15:41:46 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m4.512293106s job=4924903 project=221 runner=538fe8a5
Sep 20 15:41:47 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m5.769034381s job=4924902 project=221 runner=538fe8a5
Sep 20 15:41:47 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m5.769034381s job=4924902 project=221 runner=538fe8a5
Sep 20 15:41:49 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m6.689482595s job=4924904 project=221 runner=538fe8a5
Sep 20 15:41:49 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m6.689482595s job=4924904 project=221 runner=538fe8a5
Sep 20 15:41:50 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m9.555939288s job=4924901 project=221 runner=538fe8a5
Sep 20 15:41:50 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m9.555939288s job=4924901 project=221 runner=538fe8a5
Sep 20 15:41:59 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m15.287961167s job=4924908 project=221 runner=538fe8a5
Sep 20 15:41:59 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m15.287961167s job=4924908 project=221 runner=538fe8a5
Sep 20 15:41:59 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m17.093801468s job=4924905 project=221 runner=538fe8a5
Sep 20 15:41:59 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m17.093801468s job=4924905 project=221 runner=538fe8a5
Sep 20 15:41:59 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m16.274871242s job=4924907 project=221 runner=538fe8a5
Sep 20 15:41:59 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m16.274871242s job=4924907 project=221 runner=538fe8a5
Sep 20 15:42:00 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m16.171219967s job=4924909 project=221 runner=538fe8a5
Sep 20 15:42:00 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m16.171219967s job=4924909 project=221 runner=538fe8a5
Sep 20 15:42:02 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m19.385605373s job=4924906 project=221 runner=538fe8a5
Sep 20 15:42:02 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m19.385605373s job=4924906 project=221 runner=538fe8a5
Sep 20 15:42:06 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924910 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:42:06 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924910 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:43:03 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=57.011087644s job=4924910 project=221 runner=538fe8a5
Sep 20 15:43:03 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=57.011087644s job=4924910 project=221 runner=538fe8a5
Sep 20 15:43:05 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924911 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:43:05 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924911 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:43:06 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=Error decoding json payload read tcp 10.250.7.2:38264->172.65.19.90:443: read: connection reset by peer
Sep 20 15:43:06 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=Error decoding json payload read tcp 10.250.7.2:38264->172.65.19.90:443: read: connection reset by peer
Sep 20 15:43:06 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:38256->172.65.19.90:443: read: connection reset by peer
Sep 20 15:43:06 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:38256->172.65.19.90:443: read: connection reset by peer
Sep 20 15:43:09 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924913 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:43:09 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924913 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:43:10 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924914 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:43:10 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924914 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:43:10 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924915 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:43:10 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924915 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/k8s-workloads/gitlab-com.git runner=538fe8a5
Sep 20 15:44:08 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=59.119789095s job=4924913 project=221 runner=538fe8a5
Sep 20 15:44:08 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=59.119789095s job=4924913 project=221 runner=538fe8a5
Sep 20 15:44:09 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=58.599139826s job=4924915 project=221 runner=538fe8a5
Sep 20 15:44:09 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=58.599139826s job=4924915 project=221 runner=538fe8a5
Sep 20 15:44:11 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m1.216764263s job=4924914 project=221 runner=538fe8a5
Sep 20 15:44:11 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m1.216764263s job=4924914 project=221 runner=538fe8a5
Sep 20 15:44:30 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m25.173434124s job=4924911 project=221 runner=538fe8a5
Sep 20 15:44:30 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m25.173434124s job=4924911 project=221 runner=538fe8a5
Sep 20 15:44:55 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:40566->172.65.19.90:443: read: connection reset by peer
Sep 20 15:44:55 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:40566->172.65.19.90:443: read: connection reset by peer
Sep 20 15:44:57 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:39936->172.65.19.90:443: read: connection reset by peer
Sep 20 15:44:57 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:39936->172.65.19.90:443: read: connection reset by peer
Sep 20 15:45:23 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924930 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/infradev-report.git runner=538fe8a5
Sep 20 15:45:23 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4924930 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/infradev-report.git runner=538fe8a5
Sep 20 15:46:50 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m26.993582066s job=4924930 project=500 runner=538fe8a5
Sep 20 15:46:50 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m26.993582066s job=4924930 project=500 runner=538fe8a5
Sep 20 15:49:15 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:40598->172.65.19.90:443: read: connection reset by peer
Sep 20 15:49:15 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:40598->172.65.19.90:443: read: connection reset by peer
Sep 20 15:51:09 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:40738->172.65.19.90:443: read: connection reset by peer
Sep 20 15:51:09 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:40738->172.65.19.90:443: read: connection reset by peer
Sep 20 15:55:23 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:40774->172.65.19.90:443: read: connection reset by peer
Sep 20 15:55:23 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:40774->172.65.19.90:443: read: connection reset by peer
Sep 20 15:57:29 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:40858->172.65.19.90:443: read: connection reset by peer
Sep 20 15:57:29 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:40858->172.65.19.90:443: read: connection reset by peer
Sep 20 15:59:37 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:40900->172.65.19.90:443: read: connection reset by peer
Sep 20 15:59:37 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:40900->172.65.19.90:443: read: connection reset by peer
Sep 20 16:05:54 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4925032 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/infradev-report.git runner=538fe8a5
Sep 20 16:05:54 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4925032 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/infradev-report.git runner=538fe8a5
Sep 20 16:07:24 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m30.382573322s job=4925032 project=500 runner=538fe8a5
Sep 20 16:07:24 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m30.382573322s job=4925032 project=500 runner=538fe8a5
Sep 20 16:07:30 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:40970->172.65.19.90:443: read: connection reset by peer
Sep 20 16:07:30 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:40970->172.65.19.90:443: read: connection reset by peer
Sep 20 16:12:10 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:41564->172.65.19.90:443: read: connection reset by peer
Sep 20 16:12:10 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:41564->172.65.19.90:443: read: connection reset by peer
Sep 20 16:16:10 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:41696->172.65.19.90:443: read: connection reset by peer
Sep 20 16:16:10 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:41696->172.65.19.90:443: read: connection reset by peer
Sep 20 16:24:12 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:41738->172.65.19.90:443: read: connection reset by peer
Sep 20 16:24:12 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:41738->172.65.19.90:443: read: connection reset by peer
Sep 20 16:25:22 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4925239 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/infradev-report.git runner=538fe8a5
Sep 20 16:25:22 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4925239 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/infradev-report.git runner=538fe8a5
Sep 20 16:26:06 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:41888->172.65.19.90:443: read: connection reset by peer
Sep 20 16:26:06 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:41888->172.65.19.90:443: read: connection reset by peer
Sep 20 16:26:57 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m34.732813323s job=4925239 project=500 runner=538fe8a5
Sep 20 16:26:57 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m34.732813323s job=4925239 project=500 runner=538fe8a5
Sep 20 16:36:29 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4925263 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/helicopter.git runner=538fe8a5
Sep 20 16:36:29 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4925263 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/helicopter.git runner=538fe8a5
Sep 20 16:36:46 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=16.424987764s job=4925263 project=383 runner=538fe8a5
Sep 20 16:36:46 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=16.424987764s job=4925263 project=383 runner=538fe8a5
Sep 20 16:38:17 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:41970->172.65.19.90:443: read: connection reset by peer
Sep 20 16:38:17 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:41970->172.65.19.90:443: read: connection reset by peer
Sep 20 16:40:31 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:42734->172.65.19.90:443: read: connection reset by peer
Sep 20 16:40:31 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:42734->172.65.19.90:443: read: connection reset by peer
Sep 20 16:45:12 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4925347 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/infradev-report.git runner=538fe8a5
Sep 20 16:45:12 runner-01-inf-ops gitlab-runner[27325]: Checking for jobs... received job=4925347 repo_url=https://ops.gitlab.net/gitlab-com/gl-infra/infradev-report.git runner=538fe8a5
Sep 20 16:46:37 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m24.398433235s job=4925347 project=500 runner=538fe8a5
Sep 20 16:46:37 runner-01-inf-ops gitlab-runner[27325]: Job succeeded duration=1m24.398433235s job=4925347 project=500 runner=538fe8a5
Sep 20 16:48:37 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:42780->172.65.19.90:443: read: connection reset by peer
Sep 20 16:48:37 runner-01-inf-ops gitlab-runner[27325]: WARNING: Checking for jobs... failed runner=538fe8a5 status=couldn't execute POST against https://ops.gitlab.net/api/v4/jobs/request: Post https://ops.gitlab.net/api/v4/jobs/request: read tcp 10.250.7.2:42780->172.65.19.90:443: read: connection reset by peer
Notice that the runner that is failing to make a TCP connection, is able to pick up other jobs surrounding it.
Output of checks
Ops Checks
skarbek@gitlab-01-inf-ops.c.gitlab-ops.internal:~$ sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.18.1 ? ... OK (13.18.1)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
16/3 ... yes
16/5 ... yes
33/12 ... yes
33/13 ... yes
33/14 ... yes
33/15 ... yes
33/16 ... yes
33/17 ... yes
33/18 ... yes
33/19 ... yes
33/20 ... yes
33/21 ... yes
33/22 ... yes
33/23 ... yes
33/24 ... yes
33/25 ... yes
33/26 ... yes
33/27 ... yes
33/28 ... yes
33/29 ... yes
33/30 ... yes
33/31 ... yes
33/32 ... yes
33/33 ... yes
33/34 ... yes
33/35 ... yes
33/36 ... yes
33/37 ... yes
33/38 ... yes
33/39 ... yes
33/40 ... yes
33/41 ... yes
33/42 ... yes
33/43 ... yes
33/44 ... yes
33/45 ... yes
33/46 ... yes
33/47 ... yes
33/48 ... yes
33/49 ... yes
33/50 ... yes
33/51 ... yes
33/52 ... yes
33/53 ... yes
33/54 ... yes
33/55 ... yes
33/56 ... yes
33/57 ... yes
33/58 ... yes
33/59 ... yes
33/60 ... yes
33/61 ... yes
33/62 ... yes
33/63 ... yes
33/64 ... yes
33/65 ... yes
33/66 ... yes
33/67 ... yes
33/68 ... yes
33/69 ... yes
33/70 ... yes
33/71 ... yes
33/72 ... yes
33/73 ... yes
33/74 ... yes
33/75 ... yes
33/76 ... yes
33/77 ... yes
33/78 ... yes
33/79 ... yes
33/80 ... yes
33/83 ... yes
37/86 ... yes
37/87 ... yes
37/88 ... yes
37/89 ... yes
37/90 ... yes
37/91 ... yes
37/92 ... yes
37/93 ... yes
37/94 ... yes
37/95 ... yes
37/96 ... yes
37/97 ... yes
37/98 ... yes
37/99 ... yes
37/100 ... yes
37/101 ... yes
37/102 ... yes
39/103 ... yes
39/104 ... yes
34/105 ... yes
39/106 ... yes
39/107 ... yes
39/108 ... yes
39/109 ... yes
39/110 ... yes
39/111 ... yes
39/112 ... yes
39/113 ... yes
39/114 ... yes
39/115 ... yes
39/116 ... yes
39/117 ... yes
39/118 ... yes
34/119 ... yes
34/120 ... yes
34/121 ... yes
34/122 ... yes
34/123 ... yes
40/124 ... yes
33/125 ... yes
23/126 ... yes
33/127 ... yes
677/128 ... yes
33/129 ... yes
61/130 ... yes
34/131 ... yes
33/132 ... yes
33/133 ... yes
34/134 ... yes
40/135 ... yes
33/136 ... yes
33/137 ... yes
34/138 ... yes
102/139 ... yes
34/140 ... yes
34/143 ... yes
102/144 ... yes
33/145 ... yes
33/146 ... yes
102/147 ... yes
16/148 ... yes
102/150 ... yes
102/151 ... yes
49/152 ... yes
102/153 ... yes
40/154 ... yes
33/155 ... yes
102/157 ... yes
102/158 ... yes
14/159 ... yes
33/160 ... yes
102/161 ... yes
102/162 ... yes
168/163 ... yes
168/164 ... yes
19/165 ... yes
168/166 ... yes
168/167 ... yes
168/168 ... yes
168/169 ... yes
39/170 ... yes
168/171 ... yes
168/172 ... yes
168/173 ... yes
23/174 ... yes
44/175 ... yes
179/176 ... yes
177/177 ... yes
177/178 ... yes
177/179 ... yes
177/180 ... yes
177/181 ... yes
177/182 ... yes
177/183 ... yes
177/184 ... yes
177/185 ... yes
177/186 ... yes
177/187 ... yes
177/188 ... yes
177/189 ... yes
177/190 ... yes
177/191 ... yes
177/192 ... yes
16/193 ... yes
177/194 ... yes
177/195 ... yes
34/196 ... yes
188/199 ... yes
102/200 ... yes
138/201 ... yes
102/204 ... yes
130/205 ... yes
22/206 ... yes
33/207 ... yes
23/208 ... yes
102/209 ... yes
168/210 ... yes
198/211 ... yes
33/212 ... yes
33/213 ... yes
49/214 ... yes
19/215 ... yes
237/217 ... yes
102/218 ... yes
244/219 ... yes
34/220 ... yes
244/221 ... yes
25/224 ... yes
189/225 ... yes
102/226 ... yes
34/227 ... yes
102/228 ... yes
244/229 ... yes
33/230 ... yes
177/231 ... yes
243/232 ... yes
33/233 ... yes
244/234 ... yes
33/235 ... yes
177/236 ... yes
33/237 ... yes
48/238 ... yes
102/239 ... yes
176/241 ... yes
40/243 ... yes
28/244 ... yes
314/245 ... yes
317/246 ... yes
74/247 ... yes
102/249 ... yes
83/250 ... yes
177/251 ... yes
321/252 ... yes
212/253 ... yes
177/254 ... yes
242/255 ... yes
65/256 ... yes
116/257 ... yes
223/258 ... yes
244/259 ... yes
243/262 ... yes
354/263 ... yes
358/264 ... yes
354/267 ... yes
354/268 ... yes
354/269 ... yes
354/270 ... yes
354/271 ... yes
354/272 ... yes
354/273 ... yes
354/274 ... yes
354/275 ... yes
354/276 ... yes
354/277 ... yes
354/278 ... yes
240/279 ... yes
40/280 ... yes
40/282 ... yes
293/283 ... yes
102/285 ... yes
33/286 ... yes
406/287 ... yes
176/288 ... yes
52/290 ... yes
191/291 ... yes
76/292 ... yes
102/293 ... yes
354/294 ... yes
470/295 ... yes
470/296 ... yes
99/299 ... yes
177/301 ... yes
102/302 ... yes
76/303 ... yes
102/304 ... yes
354/305 ... yes
470/306 ... yes
102/307 ... yes
219/308 ... yes
181/309 ... yes
353/311 ... yes
353/312 ... yes
284/313 ... yes
181/314 ... yes
102/315 ... yes
513/316 ... yes
273/317 ... yes
506/318 ... yes
176/319 ... yes
64/320 ... yes
102/321 ... yes
34/322 ... yes
102/323 ... yes
513/324 ... yes
235/325 ... yes
448/326 ... yes
61/327 ... yes
244/328 ... yes
358/329 ... yes
573/330 ... yes
399/331 ... yes
376/333 ... yes
244/334 ... yes
61/335 ... yes
585/336 ... yes
89/337 ... yes
102/338 ... yes
177/339 ... yes
102/340 ... yes
176/341 ... yes
177/343 ... yes
176/344 ... yes
585/345 ... yes
177/346 ... yes
102/347 ... yes
77/348 ... yes
177/349 ... yes
102/350 ... yes
626/351 ... yes
33/352 ... yes
14/353 ... yes
34/354 ... yes
102/355 ... yes
102/356 ... yes
654/357 ... yes
354/358 ... yes
76/359 ... yes
83/360 ... yes
653/361 ... yes
102/363 ... yes
463/364 ... yes
61/365 ... yes
77/366 ... yes
78/367 ... yes
662/368 ... yes
102/369 ... yes
141/371 ... yes
633/372 ... yes
102/374 ... yes
676/376 ... yes
631/377 ... yes
102/378 ... yes
633/379 ... yes
513/380 ... yes
633/381 ... yes
482/382 ... yes
102/383 ... yes
258/384 ... yes
54/385 ... yes
70/386 ... yes
102/387 ... yes
585/388 ... yes
102/389 ... yes
28/390 ... yes
631/391 ... yes
713/392 ... yes
713/393 ... yes
718/394 ... yes
718/395 ... yes
718/396 ... yes
717/397 ... yes
717/398 ... yes
717/399 ... yes
717/400 ... yes
244/401 ... yes
177/402 ... yes
102/403 ... yes
723/404 ... yes
585/405 ... yes
33/406 ... yes
102/407 ... yes
168/408 ... yes
128/409 ... yes
128/410 ... yes
128/411 ... yes
513/412 ... yes
699/413 ... yes
102/414 ... yes
33/416 ... yes
739/417 ... yes
718/418 ... yes
102/419 ... yes
718/420 ... yes
718/421 ... yes
739/422 ... yes
152/423 ... yes
152/424 ... yes
152/425 ... yes
518/426 ... yes
39/427 ... yes
33/428 ... yes
764/429 ... yes
84/430 ... yes
358/431 ... yes
102/432 ... yes
548/433 ... yes
102/434 ... yes
102/435 ... yes
33/436 ... yes
388/437 ... yes
394/438 ... yes
33/439 ... yes
598/441 ... yes
723/442 ... yes
723/443 ... yes
723/444 ... yes
723/445 ... yes
723/446 ... yes
723/447 ... yes
792/448 ... yes
792/449 ... yes
792/450 ... yes
792/451 ... yes
633/452 ... yes
102/456 ... yes
102/457 ... yes
680/461 ... yes
33/462 ... yes
628/463 ... yes
513/464 ... yes
854/465 ... yes
15/466 ... yes
696/467 ... yes
70/468 ... yes
141/470 ... yes
48/471 ... yes
48/472 ... yes
40/473 ... yes
50/474 ... yes
821/476 ... yes
821/477 ... yes
304/478 ... yes
176/479 ... yes
764/480 ... yes
52/481 ... yes
16/482 ... yes
513/483 ... yes
513/484 ... yes
830/486 ... yes
830/487 ... yes
96/488 ... yes
33/490 ... yes
350/491 ... yes
40/492 ... yes
102/495 ... yes
854/496 ... yes
48/498 ... yes
33/499 ... yes
102/500 ... yes
599/501 ... yes
870/502 ... yes
513/503 ... yes
495/506 ... yes
513/511 ... yes
406/512 ... yes
879/513 ... yes
879/514 ... yes
16/515 ... yes
177/516 ... yes
883/517 ... yes
885/518 ... yes
102/519 ... yes
96/523 ... yes
96/524 ... yes
96/525 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.31.0 ? ... yes (2.31.1)
Git user has default SSH configuration? ... yes
Active users: ... 642
Is authorized keys file accessible? ... skipped (authorized keys not enabled)
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 7.x (6.4 - 6.x deprecated to be removed in 13.8)? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Results of GitLab environment info
Expand for output related to GitLab environment info
skarbek@gitlab-01-inf-ops.c.gitlab-ops.internal:~$ sudo gitlab-rake gitlab:env:info
System information
System: Ubuntu 16.04
Proxy: no
Current User: git
Using RVM: no
Ruby Version: 2.7.2p137
Gem Version: 3.1.4
Bundler Version:2.1.4
Rake Version: 13.0.3
Redis Version: 6.0.14
Git Version: 2.31.1
Sidekiq Version:5.2.9
Go Version: unknown
GitLab information
Version: 13.12.9-ee
Revision: cdec15d2a57
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.7
URL: https://ops.gitlab.net
HTTP Clone URL: https://ops.gitlab.net/some-group/some-project.git
SSH Clone URL: git@ops.gitlab.net:some-group/some-project.git
Elasticsearch: no
Geo: no
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: google_oauth2, saml
GitLab Shell
Version: 13.18.1
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
root@runner-01-inf-ops.c.gitlab-ops.internal:/var/log# gitlab-runner health-check
Runtime platform arch=amd64 os=linux pid=26873 revision=6fbc7474 version=13.1.1
FATAL: No HOST or PORT found
root@runner-01-inf-ops.c.gitlab-ops.internal:/var/log# gitlab-runner status
Runtime platform arch=amd64 os=linux pid=26999 revision=6fbc7474 version=13.1.1
gitlab-runner: Service is running!