Fix get-openstack-images creating duplicate images (API pagination issue)

Closes #1962 (closed)

What does this MR do and why?

After doing some digging it seems that the culprit for this issue was using paginated=False when doing the search inside image_exists_in_glance .

Using this, returns incomplete results as opposed to letting the api handle pagination internally.

For example, i used 2 images:

 ubuntu-jammy-hardened-rke2-1-29-9:
   md5: 26205ddd8f5868e6b9be095617eff025
 ubuntu-jammy-plain-rke2-1-28-14:
   md5: 285dddc1296106e490c49cfd2a76f474

For one of them - ubuntu-jammy-hardened-rke2-1-29-9 get-openstack-images.py would find a hit when doing the search (although there were more than one images matching the md5 with community visibility):

>>> sorted_list = conn.image.images(tags=["sylva-md5-26205ddd8f5868e6b9be095617eff025"], visibility="community", paginated=False)
>>> [print(x.name, x.tags, x.status) for x in sorted_list]
ubuntu-jammy-hardened-rke2-1-29-9-sylva-diskimage-builder-0.3.3 ['sylva-md5-26205ddd8f5868e6b9be095617eff025'] active

With paginated=False removed (Default is True):

>>> sorted_list = conn.image.images(tags=["sylva-md5-26205ddd8f5868e6b9be095617eff025"], visibility="community")
>>> [print(x.name, x.tags, x.status) for x in sorted_list]
ubuntu-jammy-hardened-rke2-1-29-9-sylva-diskimage-builder-0.3.3 ['sylva-md5-26205ddd8f5868e6b9be095617eff025'] active
ubuntu-jammy-hardened-rke2-1-29-9-sylva-diskimage-builder-0.3.3 ['sylva-md5-26205ddd8f5868e6b9be095617eff025'] active
ubuntu-jammy-hardened-rke2-1-29-9-sylva-diskimage-builder-0.3.3 ['sylva-md5-26205ddd8f5868e6b9be095617eff025'] active
ubuntu-jammy-hardened-rke2-1-29-9-sylva-diskimage-builder-0.3.3 ['sylva-md5-26205ddd8f5868e6b9be095617eff025'] active
ubuntu-jammy-hardened-rke2-1-29-9-sylva-diskimage-builder-0.3.3 ['sylva-md5-26205ddd8f5868e6b9be095617eff025'] active
ubuntu-jammy-hardened-rke2-1-29-9-sylva-diskimage-builder-0.3.3 ['sylva-md5-26205ddd8f5868e6b9be095617eff025'] active
ubuntu-jammy-hardened-rke2-1-29-9-sylva-diskimage-builder-0.3.3 ['sylva-md5-26205ddd8f5868e6b9be095617eff025'] active
ubuntu-jammy-hardened-rke2-1-29-9-sylva-diskimage-builder-0.3.3 ['sylva-md5-26205ddd8f5868e6b9be095617eff025'] active

While for the ubuntu-jammy-plain-rke2-1-28-14, the script would not find any, and started the download and push to glance process:

>>> sorted_list = conn.image.images(tags=["sylva-md5-285dddc1296106e490c49cfd2a76f474"], visibility="community", paginated=False)
>>> [print(x.name, x.tags, x.status) for x in sorted_list]
[]

Removing the paginated=False (default is True):

>>> sorted_list = conn.image.images(tags=["sylva-md5-285dddc1296106e490c49cfd2a76f474"], visibility="community")
>>> [print(x.name, x.tags, x.status) for x in sorted_list]
ubuntu-jammy-plain-rke2-1-28-14-sylva-diskimage-builder-0.3.3 ['sylva-md5-285dddc1296106e490c49cfd2a76f474'] active
ubuntu-jammy-plain-rke2-1-28-14-sylva-diskimage-builder-0.3.3 ['sylva-md5-285dddc1296106e490c49cfd2a76f474'] active
ubuntu-jammy-plain-rke2-1-28-14-sylva-diskimage-builder-0.3.3 ['sylva-md5-285dddc1296106e490c49cfd2a76f474'] active

This way, running the script relying on the default pagination settings we get the correct behavior from:

2025-01-19 19:37:08,683 INFO __main__ <module>: Working on image: ubuntu-jammy-plain-rke2-1-28-14 with MD5 checksum 285dddc1296106e490c49cfd2a76f474
2025-01-19 19:37:08,836 INFO __main__ <module>: image not in Glance: ubuntu-jammy-plain-rke2-1-28-14 / md5 285dddc1296106e490c49cfd2a76f474
2025-01-19 19:37:08,836 INFO __main__ <module>: Pulling image: ubuntu-jammy-plain-rke2-1-28-14 from artifact uri: oci://registry.gitlab.com/sylva-projects/sylva-elements/diskimage-builder/ubuntu-jammy-plain-rke2-1-28-14:0.3.3 to /tmp/tmpdqhunsv2

to

2025-01-19 19:38:33,744 INFO __main__ <module>: Working on image: ubuntu-jammy-plain-rke2-1-28-14 with MD5 checksum 285dddc1296106e490c49cfd2a76f474
2025-01-19 19:38:35,161 WARNING __main__ image_exists_in_glance: Image with name 'ubuntu-jammy-plain-rke2-1-28-14-sylva-diskimage-builder-0.3.3' already exists.
2025-01-19 19:38:35,161 INFO __main__ <module>: Image already in Glance: ubuntu-jammy-plain-rke2-1-28-14 with MD5 checksum 285dddc1296106e490c49cfd2a76f474
2025-01-19 19:38:35,161 INFO __main__ <module>: Existing image details - Name: ubuntu-jammy-plain-rke2-1-28-14-sylva-diskimage-builder-0.3.3, UUID: 7c12146a-9463-4c73-8fb8-c98e7c52a24f

Related reference(s)

Test coverage

CI configuration

Below you can choose test deployment variants to run in this MR's CI.

Click to open to CI configuration

Legend:

Icon	Meaning	Available values
☁️	Infra Provider	capd, capo, capm3
🚀	Bootstrap Provider	kubeadm (alias kadm), rke2
🐧	Node OS	ubuntu, suse
🛠️	Deployment Options	light-deploy, oci, ha, misc
🎬	Pipeline Scenarios	rolling-update, mgmt-rolling-update, k8s-upgrade, sylva-upgrade, sylva-upgrade-from-x.x.X, simple-update, preview, nightly

• 🎬preview ☁️capd 🚀kadm 🐧ubuntu 🛠️oci

• 🎬preview ☁️capo 🚀rke2 🐧suse

• 🎬preview ☁️capm3 🚀rke2 🐧ubuntu

• ☁️capd 🚀kubeadm 🛠️light-deploy 🐧ubuntu

• ☁️capd 🚀rke2 🛠️oci,light-deploy 🐧suse

• ☁️capo 🚀rke2 🛠️oci 🐧suse

• ☁️capo 🚀kadm 🛠️oci 🐧ubuntu

• ☁️capo 🚀rke2 🎬rolling-update 🛠️ha 🐧ubuntu

• ☁️capo 🚀kadm 🎬k8s-upgrade 🐧ubuntu

• ☁️capo 🚀rke2 🎬mgmt-rolling-update 🛠️ha,misc 🐧suse

• ☁️capo 🚀rke2 🎬sylva-upgrade 🛠️misc 🐧ubuntu

• ☁️capm3 🚀rke2 🐧suse

• ☁️capm3 🚀kadm 🛠️oci 🐧ubuntu

• ☁️capm3 🚀kadm 🎬mgmt-rolling-update 🛠️ha,misc 🐧ubuntu

• ☁️capm3 🚀rke2 🎬k8s-upgrade 🐧suse

• ☁️capm3 🚀kadm 🎬rolling-update 🛠️ha 🐧ubuntu

• ☁️capm3 🚀rke2 🎬sylva-upgrade 🛠️misc,ha 🐧suse

• ☁️capm3 🚀kadm 🎬rolling-update 🛠️ha 🐧suse

Be aware: after configuration change, pipeline is not triggered automatically. Please run it manually (by clicking the run pipeline button in Pipelines tab) or push new code.