Fixing mqtt message bus after cbs kojihub migration to RDU3

CentOS Infra issue/request

Please file this generic issue template or feel free to select another one in the list above (Description / Choose a template)

Issue Type

Issue - problem with something that was working
[] Feature Request - something you'd like us to add/modify/change
[] Task - mostly used by CentOS Infra team for day-to-day operations or bigger plans

Description

While the mqtt broker/bus was already migrated to RDU3 and working fine (even with kojihub/cbs and the rest of infra in rdu2), now that all components are migrated to same DC, we have some issues between kojihub (cbs.centos.org) and mqtt. Transient errors, as some messages/payloads are passing on the bus (triggering correctly the signing service) and some are failing with error (mosquitto side) like :

mosquitto[2375809]: 1763495545: OpenSSL Error[0]: error:80000020:system library::Broken pipe
mosquitto[2375809]: 1763495545: Client auto-6C3F402C-E642-B965-BF38-CFCDE61216F4 disconnected: Protocol error.

That doesn't make sense as 4 consequent tag-builds operation, using same plugin are used, and some are ok and some are failing, while topic/host/tls settings and also payload are identical

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information