Skip to content

Add warning during upgrade from an old version with null ssh_genkeytypes config

Ani Sinha requested to merge anisinha/cloud-init:bg2210012 into c9s

BZ: 2212670

ssh_genkeytypes configuration is documented in cloud-init is a list of
ssh keytypes that cloud-init needs to generate. For RHEL versions older than
8.4, this value is set to "~" to express null or empty list. In RHEL 8.4, we
set this value to a list of ssh keytypes ['rsa', 'ecdsa', 'ed25519'] in order
to fix BZ 1957532.
Please see commit in c9s:
967a4405b759 ("rhel/cloud.cfg: remove ssh_genkeytypes in settings.py and set in cloud.cfg")

Unfortunately, when we upgrade from a release older than RHEL 8.4 that contains
a null ssh_genkeytypes configuration to a release that has a list of ssh
keytypes (and hence a fix for BZ 1957532), on some occasions rpm will not
replace the older config file with the newer one. This happens when the
old configuration file has modifications and the "noreplace" attribute in
the config specification in the cloud-init rpm spec file will leave the old
configuration file as is and generate the new configuration file with .rpmnew
suffix. Please see the function rpmfilesDecideFate() in the rpm source
available from https://github.com/rpm-software-management/rpm.git .
Therefore, upon reboot, as cloud-init continues to use the old configuration
file with empty ssh_genkeytypes and since ssh_deletekeys is set to be true for
most newer versions of RHEL [1], no ssh keys will be generated by cloud-init
and ssh connectivity will be broken.

This issue will only happen when the old config file has modifications and the
hash values of the pristine file from the old rpm archive does not match the
one on the disk. If there are no local on-disk modifications to the config file,
rpm will simply replace the old config file with the new file from the upgraded
rpm and the issue will not be seen.

This change adds a warning during upgrade if the null syntax in ssh_genkeytypes
is detected in cloud-init config file. It instructs the user to fix the
problematic configuration in accordance with instuctions available in the KCS
https://access.redhat.com/solutions/6988034.

An inline updation of the ssh_genkeytypes option in the on-disk config
file from the post install scriptlet is not done due to two main reasons:
(a) We do not know what modifications might exist in the config file on the
    disk and we believe a human should manually inspect the local changes and
    reconcile the differences between the old config and the new config files
    as per the instructions in the KCS. An automated inline replacement might
    be dangerous.
(b) We do not want to maintain the default value for ssh_genkeytypes setting
    in two different places - one in the spec file and another in the RHEL
    config file.
(c) The issue seems to be present only in a narrow set of upgrade cases and
    a warning seems to be enough to make the upgrade experience friendlier.

1. Please check RHBZ 1574338 and https://access.redhat.com/errata/RHSA-2020:3050
   for details as to why we enable ssh_deletekeys. This setting is enabled for
   RHEL versions 7.9 and 8.2 and above.

rpm upgrade shows this warning message:

# rpm -Uvh cloud-init-23.1.1-5.el9.anisinha202306051714.noarch.rpm 
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:cloud-init-23.1.1-5.el9.anisinha2warning: /etc/cloud/cloud.cfg created as /etc/cloud/cloud.cfg.rpmnew
################################# [ 50%]
***********************************************
*** WARNING!!!! ***

ssh_genkeytypes set to null in /etc/cloud/cloud.cfg!
SSH access might be broken after reboot. Please check the following KCS
for more detailed information:

https://access.redhat.com/solutions/6988034

Please reconcile the differences between /etc/cloud/cloud.cfg and 
/etc/cloud/cloud.cfg.rpmnew and update ssh_genkeytypes configuration in 
/etc/cloud/cloud.cfg to a list of keytype values, something like:
ssh_genkeytypes:  ['rsa', 'ecdsa', 'ed25519']

************************************************
Cleaning up / removing...
   2:cloud-init-22.1-9.el9            ################################# [100%]

Additionally there is a LEAPP ticket to warn users similarly when upgrading from RHEL 7.9 to RHEL8: https://issues.redhat.com/browse/OAMG-9150

This patch once merged needs to be cherry-picked to all RHEL releases 8.7 and above (8.7,8.8,8.9,9.0,9.1,9.2).

Edited by Ani Sinha

Merge request reports