connection encoding issue (latin1 vs utf8mb4)

I run the latest code state of TangoDatabase on a debian in systemd. This works fine, however the database ds does not allow me to set utf8 characters inside strings: eg the following:

ref = "some/device/ref"
import tango
device = tango.DeviceProxy(ref)
props = {}
props["init_dynamic_attributes"] = "Here comes a special chäracter"
device.put_property(props)

fails with:

DevError[
    desc = Failed to query TANGO database (error=Incorrect string value: '\xE4racte...' for column `tango`.`property_device`.`value` at row 1)
           .The query was: INSERT INTO property_device SET device='scadawire/dd-30007/d-20111',name='init_dynamic_attributes',count='1',value='Here comes a special chäracter',updated=NOW(),accessed=NOW()
  origin = DataBase::db_put_device_property
  reason = DB_SQLError
severity = ERR]

The database and the tables are in utf8mb4. The problem is the default character set for connections in recent debians is also utf8mb4 (to mysql and mariadb servers). Probably in other derivates, too. The source code actually expects a latin1 connection and fails with the above issue indicating wrongly encoding issue between latin1 / utf8. In general i would suggest to make sure all parts involved are running in utf-8. However the issue seems to stem from CORBA directly, and the ipc system in general, which does transport strings in latin1: https://omniorb.sourceforge.io/omni42/omniORB/omniORB008.html Hence probably the ominious info in the readme about latin1 requirement. Since the latin1 type mysql connection is a requirement of the TangoDatabase the solution is to force the connection to be latin1 and not depend on the os level config being correct or wrong depending on the my.cnf applied.

Suggestion:

  1. Inside DatabaseConnectionPool.cpp

add the following on method (confirmed resolving the issue with utf8 characters):

on method:
void DatabaseConnectionPool::create_connection_pool(const char *mysql_user,
                                                    const char *mysql_password,
                                                    const char *mysql_host,
                                                    const char *mysql_db_name)
{


enforce the latin1 connection (tested and confirmed working):

WARN_STREAM << "Going to connect to MySQL for conn. " << loop << std::endl;

+ mysql_options(m_conn_pool[loop].db, MYSQL_SET_CHARSET_NAME, "latin1");
+ WARN_STREAM << "MySQL client character set for connection " << loop << ": "
            << mysql_character_set_name(m_conn_pool[loop].db) << std::endl;
...

Also inside the readme, i would suggest to add a hint regarding the re-coding parts, especially something like:

See also:
https://omniorb.sourceforge.io/omni42/omniORB/omniORB008.html
"...The defaults are ISO 8859-1 (Latin 1) for char and string..."
Since all strings received over corba on the db device are latin1 encoded by default, the connection to the database MUST also be done with latin1.
Note that the tables themselves are utf8mb4. transformation in read/write is done automatically by the database.
Edited by Sebastian Jennen
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information