Failures for upgrade from 1.1.0 to main nightly CIs
It was noticed by @marc.bailly1 that last batches of upgrade from tag 1.1.0 to main nightly CIs for capo-rke2 have been failing at update-management-cluster stage, always with issues reported for harbor unit.
Last working job for the capo-rke2 stage update-management-cluster was in 2024-04-22T19:30:
2024/04/22 20:29:30.449682 HelmRelease/harbor state changed: InstallSucceeded - Helm install succeeded for release harbor/harbor.v1 with chart harbor@1.14.0while the next night one 2024-04-23T19:30 (and actually all others after that), have been failing with some form of harbor HelmRelease upgrade/rollback timeout:
2024/04/23 21:07:21.837151 Kustomization/harbor state changed: HealthCheckFailed - health check failed after 50.065829ms: failed early due to stalled resources: [HelmRelease/sylva-system/harbor status: 'Failed']
2024/04/23 21:07:52.668864 Command timeout exceeded
Timed-out waiting for the following resources to be ready:
IDENTIFIER STATUS REASON MESSAGE
Kustomization/sylva-system/harbor InProgress Kustomization generation is 3, but latest observed generation is 1
╰┄╴HelmRelease/sylva-system/harbor Failed Failed to upgrade after 4 attempt(s)
├┄╴┬┄┄[Conditions]
┆ ├┄╴Stalled True RetriesExceeded Failed to upgrade after 4 attempt(s)
┆ ├┄╴Ready False RollbackSucceeded Helm rollback to previous release harbor/harbor.v7 with chart harbor@1.14.0 succeeded
┆ ├┄╴Released False UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆ ╰┄╴Remediated True RollbackSucceeded Helm rollback to previous release harbor/harbor.v7 with chart harbor@1.14.0 succeeded
╰┄╴┬┄┄[Events]
├┄╴harbor.17c9041c4f2e4300 RollbackSucceeded Helm rollback to previous release harbor/harbor.v5 with chart harbor@1.14.0 succeeded
├┄╴harbor.17c90469ca46eaed RollbackSucceeded Helm rollback to previous release harbor/harbor.v7 with chart harbor@1.14.0 succeeded
├┄╴harbor.17c903295d292e60 HelmChartConfigured Configured HelmChart/sylva-system/sylva-system-harbor with SourceRef 'HelmRepository/sylva-system/unit-harbor'
├┄╴harbor.17c9041462e83cf1 UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-23T20:48:36.906537776Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-23T20:48:37.263570016Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-23T20:48:37.432052504Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-23T20:48:37.766947515Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-23T20:48:37.856128706Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-23T20:48:37.869440143Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-23T20:48:37.869452165Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-23T20:48:37.964884122Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-23T20:48:56.054667018Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (9 duplicate lines omitted)
┆ 2024-04-23T20:48:58.092902352Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-23T20:53:36.147427186Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (138 duplicate lines omitted)
┆ 2024-04-23T20:53:38.363191215Z: warning: Upgrade "harbor" failed: context deadline exceeded
├┄╴harbor.17c9037a8b915e91 RollbackSucceeded Helm rollback to previous release harbor/harbor.v1 with chart harbor@1.14.0 succeeded
├┄╴harbor.17c903c496b28fce UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-23T20:42:54.407620879Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-23T20:42:54.60309483Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-23T20:42:54.784377087Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-23T20:42:55.026491089Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-23T20:42:55.146714136Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-23T20:42:55.157281083Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-23T20:42:55.157289628Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-23T20:42:55.290613427Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-23T20:43:13.4530005Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (9 duplicate lines omitted)
┆ 2024-04-23T20:43:15.52112341Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-23T20:47:53.439260487Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (138 duplicate lines omitted)
┆ 2024-04-23T20:47:55.65037361Z: warning: Upgrade "harbor" failed: context deadline exceeded
├┄╴harbor.17c903ca39491c08 RollbackSucceeded Helm rollback to previous release harbor/harbor.v3 with chart harbor@1.14.0 succeeded
├┄╴harbor.17c90372a547da9a UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-23T20:37:01.882492577Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-23T20:37:02.137165581Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-23T20:37:02.384202931Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-23T20:37:02.712907215Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-23T20:37:02.908393039Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-23T20:37:02.979583636Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-23T20:37:02.979593554Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-23T20:37:03.090328293Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-23T20:37:31.243068557Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (14 duplicate lines omitted)
┆ 2024-04-23T20:37:33.267915006Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-23T20:42:01.256647208Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (133 duplicate lines omitted)
┆ 2024-04-23T20:42:03.607499681Z: warning: Upgrade "harbor" failed: context deadline exceeded
╰┄╴harbor.17c90466970d5ab6 UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
Last Helm logs:
2024-04-23T20:54:29.988485195Z: Patch Deployment "harbor-jobservice" in namespace harbor
2024-04-23T20:54:30.214511375Z: Patch Deployment "harbor-portal" in namespace harbor
2024-04-23T20:54:30.389265458Z: Patch Deployment "harbor-registry" in namespace harbor
2024-04-23T20:54:30.624848927Z: Patch StatefulSet "harbor-redis" in namespace harbor
2024-04-23T20:54:30.999609347Z: Looks like there are no changes for Ingress "harbor-ingress"
2024-04-23T20:54:31.036170103Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
2024-04-23T20:54:31.0361813Z: beginning wait for 24 resources with timeout of 5m0s
2024-04-23T20:54:31.157675931Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
2024-04-23T20:54:49.3350528Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (9 duplicate lines omitted)
2024-04-23T20:54:51.335732144Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
2024-04-23T20:59:29.327535776Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (138 duplicate lines omitted)
2024-04-23T20:59:31.47484744Z: warning: Upgrade "harbor" failed: context deadline exceededsubsequent CI failures (click to expand)
2024/04/28 20:56:59.104559 Kustomization/harbor state changed: HealthCheckFailed - health check failed after 30.025354371s: timeout waiting for: [HelmRelease/sylva-system/harbor status: 'InProgress']
2024/04/28 20:56:59.104622 ╭╴Waiting for the following resources to progress:
2024/04/28 20:56:59.104628 │ HelmRelease/harbor RollbackFailed - Helm rollback to previous release harbor/harbor.v1 with chart harbor@1.14.0 failed: release harbor failed: context deadline exceeded
2024/04/28 20:56:59.104633 │ Kustomization/harbor HealthCheckFailed - health check failed after 30.025354371s: timeout waiting for: [HelmRelease/sylva-system/harbor status: 'InProgress']
2024/04/28 20:56:59.104635 ╰╴╴╴┄
2024/04/28 20:57:45.028243 Command timeout exceeded
Timed-out waiting for the following resources to be ready:
IDENTIFIER STATUS REASON MESSAGE
Kustomization/sylva-system/harbor InProgress Kustomization generation is 3, but latest observed generation is 1
╰┄╴HelmRelease/sylva-system/harbor InProgress HelmRelease generation is 2, but latest observed generation is 1
╰┄╴Deployment/harbor/harbor-registry Failed Progress deadline exceeded
╰┄╴ReplicaSet/harbor/harbor-registry-64cdcc67c5 InProgress Available: 0/1
╰┄╴Pod/harbor/harbor-registry-64cdcc67c5-cnknc InProgress Pod is in the Pending phase
├┄╴┬┄┄[Conditions]
┆ ├┄╴Initialized True
┆ ├┄╴Ready False ContainersNotReady containers with unready status: [registry registryctl]
┆ ├┄╴ContainersReady False ContainersNotReady containers with unready status: [registry registryctl]
┆ ╰┄╴PodScheduled True
╰┄╴┬┄┄[Events]
├┄╴harbor-registry-64cdcc67c5-cnknc.17ca8bb32c927d76 Scheduled Successfully assigned harbor/harbor-registry-64cdcc67c5-cnknc to mgmt-1270939286-rke2-capo-md-md0-eadc10cd2d-dxxz2
╰┄╴harbor-registry-64cdcc67c5-cnknc.17ca8bb3312217ca FailedAttachVolume Multi-Attach error for volume "pvc-51090355-5593-4450-abe5-c749576f6753" Volume is already used by pod(s) harbor-registry-865d74556f-9htq42024/04/27 20:57:01.857922 Kustomization/harbor state changed: HealthCheckFailed - health check failed after 33.245687ms: failed early due to stalled resources: [HelmRelease/sylva-system/harbor status: 'Failed']
Timed-out waiting for the following resources to be ready:
2024/04/27 20:57:35.124395 Command timeout exceeded
IDENTIFIER STATUS REASON MESSAGE
Kustomization/sylva-system/harbor InProgress Kustomization generation is 3, but latest observed generation is 1
╰┄╴HelmRelease/sylva-system/harbor Failed Failed to upgrade after 4 attempt(s)
├┄╴┬┄┄[Conditions]
┆ ├┄╴Stalled True RetriesExceeded Failed to upgrade after 4 attempt(s)
┆ ├┄╴Ready False RollbackSucceeded Helm rollback to previous release harbor/harbor.v7 with chart harbor@1.14.0 succeeded
┆ ├┄╴Released False UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆ ╰┄╴Remediated True RollbackSucceeded Helm rollback to previous release harbor/harbor.v7 with chart harbor@1.14.0 succeeded
╰┄╴┬┄┄[Events]
├┄╴harbor.17ca3d6484d4d9d8 UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-27T20:30:08.732948217Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-27T20:30:08.920052826Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-27T20:30:09.063176789Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-27T20:30:09.269440119Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-27T20:30:09.368149667Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-27T20:30:09.371273473Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-27T20:30:09.371279972Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-27T20:30:09.464172778Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-27T20:30:27.540207062Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (9 duplicate lines omitted)
┆ 2024-04-27T20:30:29.549261715Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-27T20:35:07.542932526Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (138 duplicate lines omitted)
┆ 2024-04-27T20:35:09.73147674Z: warning: Upgrade "harbor" failed: context deadline exceeded
├┄╴harbor.17ca3ccd3683710b HelmChartConfigured Configured HelmChart/sylva-system/sylva-system-harbor with SourceRef 'HelmRepository/sylva-system/unit-harbor'
├┄╴harbor.17ca3db96241b129 RollbackSucceeded Helm rollback to previous release harbor/harbor.v5 with chart harbor@1.14.0 succeeded
├┄╴harbor.17ca3d1abd88822f RollbackSucceeded Helm rollback to previous release harbor/harbor.v1 with chart harbor@1.14.0 succeeded
├┄╴harbor.17ca3db4003748c6 UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-27T20:35:50.090206953Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-27T20:35:50.251123843Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-27T20:35:50.394708374Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-27T20:35:50.599190333Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-27T20:35:50.718690604Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-27T20:35:50.7215145Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-27T20:35:50.721520994Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-27T20:35:50.82844971Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-27T20:36:08.997280779Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (9 duplicate lines omitted)
┆ 2024-04-27T20:36:11.016764244Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-27T20:40:49.018868475Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (138 duplicate lines omitted)
┆ 2024-04-27T20:40:51.11895412Z: warning: Upgrade "harbor" failed: context deadline exceeded
├┄╴harbor.17ca3d156978b388 UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-27T20:24:28.516165109Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-27T20:24:28.941486044Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-27T20:24:29.119057594Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-27T20:24:29.366745513Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-27T20:24:29.568102393Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-27T20:24:29.587910147Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-27T20:24:29.587921866Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-27T20:24:29.859440835Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-27T20:25:07.985152492Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (19 duplicate lines omitted)
┆ 2024-04-27T20:25:10.008137538Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-27T20:29:27.991414773Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (128 duplicate lines omitted)
┆ 2024-04-27T20:29:29.949895392Z: warning: Upgrade "harbor" failed: context deadline exceeded
├┄╴harbor.17ca3e0343eefeb8 UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-27T20:41:30.493989259Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-27T20:41:30.683025318Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-27T20:41:30.848681678Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-27T20:41:31.037867311Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-27T20:41:31.165615289Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-27T20:41:31.179065932Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-27T20:41:31.179075169Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-27T20:41:31.253661548Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-27T20:41:59.325719347Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (14 duplicate lines omitted)
┆ 2024-04-27T20:42:01.338244211Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-27T20:46:29.351448325Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (133 duplicate lines omitted)
┆ 2024-04-27T20:46:31.562082192Z: warning: Upgrade "harbor" failed: context deadline exceeded
├┄╴harbor.17ca3e0899c439f2 RollbackSucceeded Helm rollback to previous release harbor/harbor.v7 with chart harbor@1.14.0 succeeded
╰┄╴harbor.17ca3d69d715b265 RollbackSucceeded Helm rollback to previous release harbor/harbor.v3 with chart harbor@1.14.0 succeeded2024/04/26 21:04:54.578942 Kustomization/harbor state changed: HealthCheckFailed - health check failed after 31.827972ms: failed early due to stalled resources: [HelmRelease/sylva-system/harbor status: 'Failed']
2024/04/26 21:05:16.816250 Command timeout exceeded
Timed-out waiting for the following resources to be ready:
IDENTIFIER STATUS REASON MESSAGE
Kustomization/sylva-system/harbor InProgress Kustomization generation is 3, but latest observed generation is 1
╰┄╴HelmRelease/sylva-system/harbor Failed Failed to upgrade after 4 attempt(s)
├┄╴┬┄┄[Conditions]
┆ ├┄╴Stalled True RetriesExceeded Failed to upgrade after 4 attempt(s)
┆ ├┄╴Ready False RollbackSucceeded Helm rollback to previous release harbor/harbor.v7 with chart harbor@1.14.0 succeeded
┆ ├┄╴Released False UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆ ╰┄╴Remediated True RollbackSucceeded Helm rollback to previous release harbor/harbor.v7 with chart harbor@1.14.0 succeeded
╰┄╴┬┄┄[Events]
├┄╴harbor.17c9ef0a303599e2 UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-26T20:34:18.681616257Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-26T20:34:18.838374994Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-26T20:34:18.995568749Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-26T20:34:19.231400688Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-26T20:34:19.387994653Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-26T20:34:19.393745854Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-26T20:34:19.393757819Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-26T20:34:19.486093516Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-26T20:34:47.636940565Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (14 duplicate lines omitted)
┆ 2024-04-26T20:34:49.640745281Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-26T20:39:17.636957099Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (133 duplicate lines omitted)
┆ 2024-04-26T20:39:19.860563865Z: warning: Upgrade "harbor" failed: context deadline exceeded
├┄╴harbor.17c9eec2acd96dae HelmChartConfigured Configured HelmChart/sylva-system/sylva-system-harbor with SourceRef 'HelmRepository/sylva-system/unit-harbor'
├┄╴harbor.17c9ef5962432511 UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-26T20:39:58.983771878Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-26T20:39:59.13610909Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-26T20:39:59.265781064Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-26T20:39:59.471759088Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-26T20:39:59.587405066Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-26T20:39:59.590974924Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-26T20:39:59.590994095Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-26T20:39:59.690476632Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-26T20:40:27.796971646Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (14 duplicate lines omitted)
┆ 2024-04-26T20:40:29.794452037Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-26T20:44:57.814430396Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (133 duplicate lines omitted)
┆ 2024-04-26T20:44:59.973331763Z: warning: Upgrade "harbor" failed: context deadline exceeded
├┄╴harbor.17c9ef0f962d82fa RollbackSucceeded Helm rollback to previous release harbor/harbor.v1 with chart harbor@1.14.0 succeeded
├┄╴harbor.17c9efa8abedd518 UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-26T20:45:39.366368437Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-26T20:45:39.593472989Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-26T20:45:39.80764568Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-26T20:45:40.0136109Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-26T20:45:40.135874489Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-26T20:45:40.145515258Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-26T20:45:40.145527477Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-26T20:45:40.20901248Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-26T20:45:58.272170551Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (9 duplicate lines omitted)
┆ 2024-04-26T20:46:00.293245626Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-26T20:50:38.294180876Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (138 duplicate lines omitted)
┆ 2024-04-26T20:50:40.513743192Z: warning: Upgrade "harbor" failed: context deadline exceeded
├┄╴harbor.17c9efae13ee1780 RollbackSucceeded Helm rollback to previous release harbor/harbor.v5 with chart harbor@1.14.0 succeeded
├┄╴harbor.17c9eff82f3ec998 UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-26T20:51:20.94428795Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-26T20:51:21.117539391Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-26T20:51:21.259111297Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-26T20:51:21.488753469Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-26T20:51:21.596895818Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-26T20:51:21.606756899Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-26T20:51:21.606769469Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-26T20:51:21.693256224Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-26T20:51:39.789221908Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (9 duplicate lines omitted)
┆ 2024-04-26T20:51:41.804737362Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-26T20:56:19.789323295Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (138 duplicate lines omitted)
┆ 2024-04-26T20:56:22.044074135Z: warning: Upgrade "harbor" failed: context deadline exceeded
├┄╴harbor.17c9effdb2cf3b38 RollbackSucceeded Helm rollback to previous release harbor/harbor.v7 with chart harbor@1.14.0 succeeded
╰┄╴harbor.17c9ef5ebcbee6ce RollbackSucceeded Helm rollback to previous release harbor/harbor.v3 with chart harbor@1.14.0 succeeded-
2024-04-25T19:30: runner error
2024/04/24 21:11:48.314339 Kustomization/harbor state changed: HealthCheckFailed - health check failed after 54.634914ms: failed early due to stalled resources: [HelmRelease/sylva-system/harbor status: 'Failed']
2024/04/24 21:12:23.046449 Command timeout exceeded
Timed-out waiting for the following resources to be ready:
IDENTIFIER STATUS REASON MESSAGE
Kustomization/sylva-system/harbor InProgress Kustomization generation is 3, but latest observed generation is 1
╰┄╴HelmRelease/sylva-system/harbor Failed Failed to upgrade after 4 attempt(s)
├┄╴┬┄┄[Conditions]
┆ ├┄╴Stalled True RetriesExceeded Failed to upgrade after 4 attempt(s)
┆ ├┄╴Ready False RollbackSucceeded Helm rollback to previous release harbor/harbor.v7 with chart harbor@1.14.0 succeeded
┆ ├┄╴Released False UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆ ╰┄╴Remediated True RollbackSucceeded Helm rollback to previous release harbor/harbor.v7 with chart harbor@1.14.0 succeeded
╰┄╴┬┄┄[Events]
├┄╴harbor.17c952f3a026fa3d RollbackSucceeded Helm rollback to previous release harbor/harbor.v5 with chart harbor@1.14.0 succeeded
├┄╴harbor.17c9529eb18238e9 UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-24T20:47:53.142584102Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-24T20:47:53.304812588Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-24T20:47:53.477370616Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-24T20:47:53.680344139Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-24T20:47:53.797317266Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-24T20:47:53.838389632Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-24T20:47:53.838400522Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-24T20:47:53.919802393Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-24T20:48:12.079896915Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (9 duplicate lines omitted)
┆ 2024-04-24T20:48:14.115371645Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-24T20:52:52.09365749Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (138 duplicate lines omitted)
┆ 2024-04-24T20:52:54.318788443Z: warning: Upgrade "harbor" failed: context deadline exceeded
├┄╴harbor.17c952077c4a2e9b HelmChartConfigured Configured HelmChart/sylva-system/sylva-system-harbor with SourceRef 'HelmRepository/sylva-system/unit-harbor'
├┄╴harbor.17c952ee347fb4b4 UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-24T20:53:34.80511677Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-24T20:53:34.976893706Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-24T20:53:35.093003468Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-24T20:53:35.273617368Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-24T20:53:35.374913918Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-24T20:53:35.395816636Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-24T20:53:35.39582559Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-24T20:53:35.468225491Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-24T20:53:53.616810409Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (9 duplicate lines omitted)
┆ 2024-04-24T20:53:55.611436573Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-24T20:58:33.60124458Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (138 duplicate lines omitted)
┆ 2024-04-24T20:58:35.792989124Z: warning: Upgrade "harbor" failed: context deadline exceeded
├┄╴harbor.17c9524f4149e68d UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-24T20:42:12.012767402Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-24T20:42:12.156877659Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-24T20:42:12.305127531Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-24T20:42:12.573859585Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-24T20:42:12.682705851Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-24T20:42:12.708353325Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-24T20:42:12.708393723Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-24T20:42:12.874842087Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-24T20:42:41.016024415Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (14 duplicate lines omitted)
┆ 2024-04-24T20:42:43.229510983Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-24T20:47:11.044018313Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (133 duplicate lines omitted)
┆ 2024-04-24T20:47:13.138876602Z: warning: Upgrade "harbor" failed: context deadline exceeded
├┄╴harbor.17c952a423b77651 RollbackSucceeded Helm rollback to previous release harbor/harbor.v3 with chart harbor@1.14.0 succeeded
├┄╴harbor.17c95343550e27fe RollbackSucceeded Helm rollback to previous release harbor/harbor.v7 with chart harbor@1.14.0 succeeded
├┄╴harbor.17c9533dd9b4cfe0 UpgradeFailed Helm upgrade failed for release harbor/harbor with chart harbor@1.14.2: context deadline exceeded
┆
┆ Last Helm logs:
┆
┆ 2024-04-24T20:59:16.602363908Z: Patch Deployment "harbor-jobservice" in namespace harbor
┆ 2024-04-24T20:59:16.772118334Z: Patch Deployment "harbor-portal" in namespace harbor
┆ 2024-04-24T20:59:16.933073454Z: Patch Deployment "harbor-registry" in namespace harbor
┆ 2024-04-24T20:59:17.165082828Z: Patch StatefulSet "harbor-redis" in namespace harbor
┆ 2024-04-24T20:59:17.344915357Z: Looks like there are no changes for Ingress "harbor-ingress"
┆ 2024-04-24T20:59:17.404499932Z: waiting for release harbor resources (created: 0 updated: 24 deleted: 0)
┆ 2024-04-24T20:59:17.404507697Z: beginning wait for 24 resources with timeout of 5m0s
┆ 2024-04-24T20:59:17.511064806Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready
┆ 2024-04-24T20:59:35.636672514Z: Deployment is not ready: harbor/harbor-core. 0 out of 1 expected pods are ready (9 duplicate lines omitted)
┆ 2024-04-24T20:59:37.669876043Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready
┆ 2024-04-24T21:04:15.653571401Z: Deployment is not ready: harbor/harbor-jobservice. 0 out of 1 expected pods are ready (138 duplicate lines omitted)
┆ 2024-04-24T21:04:17.815872595Z: warning: Upgrade "harbor" failed: context deadline exceeded
╰┄╴harbor.17c95254b8631ea8 RollbackSucceeded Helm rollback to previous release harbor/harbor.v1 with chart harbor@1.14.0 succeededThe timing and logs show it's related to the diff
-| **harbor** | installs Harbor | beta | | [Helm](https://helm.goharbor.io) | 1.14.0 |
+| **harbor** | installs Harbor | beta | | [Helm](https://helm.goharbor.io) | 1.14.2 |introduced by !2101 (merged), having the commit to main:
[git:main]root@caasdev:sylva-core# git log 45edc6e3a182c2564fcb5ce3480d058cbaf049d7
commit 45edc6e3a182c2564fcb5ce3480d058cbaf049d7
Author: renovate-bot <group_59800588_bot_1078c5eb1c95c7cadbb07de931bba3e4@noreply.gitlab.com>
Date: Tue Apr 23 08:42:49 2024 +0000
Update Helm release harbor to v1.14.2It's unclear yet what the problem is for the harbor chart (v1.14.0 to v1.14.2) bump and the pipelines/1271940733 was started to try reproduce, but we should probably rollback first this harbor tag bump to check if it unblocks such CI runs.
Summary
related references
Details
Edited by Bogdan-Adrian Burciu