Issue with enabling longhorn with apply.sh
Summary
During BM deployment longhorn was initially disabled and was enabled using apply.sh script after the bootstap.sh script was completed successfully. This resulted in changing the storage class of PVC for previously installed units by bootstrap script which resulted in error as it is an immutable field
related references
✓ Kustomization/calico - Resource is ready
✓ Kustomization/calico-ready - Resource is ready
✓ Kustomization/ingress-nginx - Resource is ready
✗ HelmRelease/metal3 - Stalled - RetriesExceeded: Failed to upgrade after 4 attempt(s)
✗ HelmRelease/os-image-server - Stalled - RetriesExceeded: Failed to upgrade after 4 attempt(s)
⠈⠱ Kustomization/metal3 - HealthCheckFailed - health check failed after 28.165693ms: failed early due to stalled resources: [HelmRelease/sylva-system/metal3 status: 'Failed']
⠈⠱ Kustomization/os-image-server - HealthCheckFailed - health check failed after 29.921588ms: failed early due to stalled resources: [HelmRelease/sylva-system/os-image-server status: 'Failed']
✗ Command timeout exceeded
Timed-out waiting for the following resources to be ready:
IDENTIFIER STATUS REASON MESSAGE
Kustomization/sylva-system/metal3 InProgress Kustomization generation is 3, but latest observed generation is 2
╰┄╴HelmRelease/sylva-system/metal3 Failed Failed to upgrade after 4 attempt(s)
├┄╴┬┄┄[Conditions]
┆ ├┄╴Stalled True RetriesExceeded Failed to upgrade after 4 attempt(s)
┆ ├┄╴Ready False UpgradeFailed Helm upgrade failed for release metal3-system/metal3 with chart metal3@0.8.3: cannot patch "ironic-shared-volume" with kind PersistentVolumeClaim: PersistentVolumeClaim "ironic-shared-volume" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
┆ ┆ core.PersistentVolumeClaimSpec{
┆ ┆ ... // 2 identical fields
┆ ┆ Resources: {Requests: {s"storage": {i: {...}, s: "10Gi", Format: "BinarySI"}}},
┆ ┆ VolumeName: "pvc-c638ad12-17f5-4492-b620-274e919a2c85",
┆ ┆ - StorageClassName: &"local-path",
┆ ┆ + StorageClassName: &"longhorn",
┆ ┆ VolumeMode: &"Filesystem",
┆ ┆ DataSource: nil,
┆ ┆ ... // 2 identical fields
┆ ┆ }
┆ ┆ && cannot patch "mariadb-volume-claim" with kind PersistentVolumeClaim: PersistentVolumeClaim "mariadb-volume-claim" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
┆ ┆ core.PersistentVolumeClaimSpec{
┆ ┆ ... // 2 identical fields
┆ ┆ Resources: {Requests: {s"storage": {i: {...}, s: "1Gi", Format: "BinarySI"}}},
┆ ┆ VolumeName: "pvc-ff4dc002-3bec-42f0-ae1f-702af7743977",
┆ ┆ - StorageClassName: &"local-path",
┆ ┆ + StorageClassName: &"longhorn",
┆ ┆ VolumeMode: &"Filesystem",
┆ ┆ DataSource: nil,
┆ ┆ ... // 2 identical fields
┆ ┆ }
┆ ╰┄╴Released False UpgradeFailed Helm upgrade failed for release metal3-system/metal3 with chart metal3@0.8.3: cannot patch "ironic-shared-volume" with kind PersistentVolumeClaim: PersistentVolumeClaim "ironic-shared-volume" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
┆ core.PersistentVolumeClaimSpec{
┆ ... // 2 identical fields
┆ Resources: {Requests: {s"storage": {i: {...}, s: "10Gi", Format: "BinarySI"}}},
┆ VolumeName: "pvc-c638ad12-17f5-4492-b620-274e919a2c85",
┆ - StorageClassName: &"local-path",
┆ + StorageClassName: &"longhorn",
┆ VolumeMode: &"Filesystem",
┆ DataSource: nil,
┆ ... // 2 identical fields
┆ }
┆ && cannot patch "mariadb-volume-claim" with kind PersistentVolumeClaim: PersistentVolumeClaim "mariadb-volume-claim" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
┆ core.PersistentVolumeClaimSpec{
┆ ... // 2 identical fields
┆ Resources: {Requests: {s"storage": {i: {...}, s: "1Gi", Format: "BinarySI"}}},
┆ VolumeName: "pvc-ff4dc002-3bec-42f0-ae1f-702af7743977",
┆ - StorageClassName: &"local-path",
┆ + StorageClassName: &"longhorn",
┆ VolumeMode: &"Filesystem",
┆ DataSource: nil,
┆ ... // 2 identical fields
┆ }
╰┄╴┬┄┄[Events]
├┄╴2025-05-05 10:13:27 Normal HelmChartCreated Created HelmChart/sylva-system/sylva-system-metal3 with SourceRef 'HelmRepository/sylva-system/unit-metal3'
├┄╴2025-05-05 10:17:19 Normal InstallSucceeded Helm install succeeded for release metal3-system/metal3.v1 with chart metal3@0.8.3
├┄╴2025-05-05 10:52:47 Warning UpgradeFailed Helm upgrade failed for release metal3-system/metal3 with chart metal3@0.8.3: cannot patch "ironic-shared-volume" with kind PersistentVolumeClaim: PersistentVolumeClaim "ironic-shared-volume" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
┆ core.PersistentVolumeClaimSpec{
┆ ... // 2 identical fields
┆ Resources: {Requests: {s"storage": {i: {...}, s: "10Gi", Format: "BinarySI"}}},
┆ VolumeName: "pvc-c638ad12-17f5-4492-b620-274e919a2c85",
┆ - StorageClassName: &"local-path",
┆ + StorageClassName: &"longhorn",
┆ VolumeMode: &"Filesystem",
┆ DataSource: nil,
┆ ... // 2 identical fields
┆ }
┆ && cannot patch "mariadb-volume-claim" with kind PersistentVolumeClaim: PersistentVolumeClaim "mariadb-volume-claim" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
┆ core.PersistentVolumeClaimSpec{
┆ ... // 2 identical fields
┆ Resources: {Requests: {s"storage": {i: {...}, s: "1Gi", Format: "BinarySI"}}},
┆ VolumeName: "pvc-ff4dc002-3bec-42f0-ae1f-702af7743977",
┆ - StorageClassName: &"local-path",
┆ + StorageClassName: &"longhorn",
┆ VolumeMode: &"Filesystem",
┆ DataSource: nil,
┆ ... // 2 identical fields
┆ }
┆
┆ Last Helm logs:
┆
┆ 2025-05-05T10:52:47.12689658Z: Patch Certificate "baremetal-operator-serving-cert" in namespace metal3-system
┆ 2025-05-05T10:52:47.155763424Z: Patch Certificate "ironic-cacert" in namespace metal3-system
┆ 2025-05-05T10:52:47.168798344Z: Patch Certificate "ironic-cert" in namespace metal3-system
┆ 2025-05-05T10:52:47.180187619Z: Patch Certificate "ironic-inspector-cert" in namespace metal3-system
┆ 2025-05-05T10:52:47.191593347Z: Patch Certificate "ironic-vmedia-cert" in namespace metal3-system
┆ 2025-05-05T10:52:47.203313854Z: Patch Issuer "baremetal-operator-selfsigned-issuer" in namespace metal3-system
┆ 2025-05-05T10:52:47.213764686Z: Patch Issuer "selfsigned-issuer" in namespace metal3-system
┆ 2025-05-05T10:52:47.225586521Z: Patch Issuer "ca-issuer" in namespace metal3-system
┆ 2025-05-05T10:52:47.239644705Z: Patch ValidatingWebhookConfiguration "baremetal-operator-validating-webhook-configuration" in namespace
┆ 2025-05-05T10:52:47.263767422Z: warning: Upgrade "metal3" failed: cannot patch "ironic-shared-volume" with kind PersistentVolumeClaim: PersistentVolumeClaim "ironic-shared-volume" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
┆ core.PersistentVolumeClaimSpec{
┆ ... // 2 identical fields
┆ Resources: {Requests: {s"storage": {i: {...}, s: "10Gi", Format: "BinarySI"}}},
┆ VolumeName: "pvc-c638ad12-17f5-4492-b620-274e919a2c85",
┆ - StorageClassName: &"local-path",
┆ + StorageClassName: &"longhorn",
┆ VolumeMode: &"Filesystem",
┆ DataSource: nil,
┆ ... // 2 identical fields
┆ }
┆ && cannot patch "mariadb-volume-claim" with kind PersistentVolumeClaim: PersistentVolumeClaim "mariadb-volume-claim" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims
┆ core.PersistentVolumeClaimSpec{
┆ ... // 2 identical fields
┆ Resources: {Requests: {s"storage": {i: {...}, s: "1Gi", Format: "BinarySI"}}},
┆ VolumeName: "pvc-ff4dc002-3bec-42f0-ae1f-702af7743977",
┆ - StorageClassName: &"local-path",
┆ + StorageClassName: &"longhorn",
┆ VolumeMode: &"Filesystem",
┆ DataSource: nil,
┆ ... // 2 identical fields
┆ }
Details
The purpose is to enchance documentation and validation that can detect this scenario and provide relevant message or check when longhorn is enabled using apply.sh