rosbags fails to load one of my custom messages

• OS: Ubuntu 22.04
• Python 3.10.12
• rosbags installed with pip install rosbags
• rosbags version 0.10.7

---

I am porting a tool to load and analyze data from my rosbags to work with ros2.

I got rosbags all set up and, for the most part, it works well, but I have one strange issue. One of the many custom messages I have defined fails to load with rosbags

Loading a rosbag (rosbag_example.bag) with the following code results in an error.

  from pathlib import Path
  from rosbags.highlevel import AnyReader
  from rosbags.typesys import Stores, get_typestore

  typestore = get_typestore(Stores.ROS2_HUMBLE) # The rosbag is actually recorded in NOETIC but this dosent seem to make a difference here
  bagpath = Path('rosbag_example.bag')
  with AnyReader([bagpath], default_typestore=typestore) as reader:
    for connection, timestamp, rawdata in reader.messages(connections=reader.connections):
      msg = reader.deserialize(rawdata, connection.msgtype)
      #do stuff with the msg

For the one message type that this fails on (the topic is anansi/vehicle_velocity_control, msg type is anansi_msgs/NavCmd). I am getting a KeyError in reader.deserialize

this is the message definition (NavCmd.msg):

builtin_interfaces/Time t      # Optionally populated w/ an associated ros::Time
std_msgs/Header header


float64[3]      velocity_ned
float64[3]      accel_ned
float64         yaw_rate

When it loads the data from this topic, for some reason, it thinks that the message type for header is anansi_msgs/msg/Header rather than std_msgs/msg/Header (anansi_msgs is the package where the NavCmd message type is defined) so when it tries to parse the message, it fails to find the anansi_msgs/msg/Header because it doesn't exist.

I have several custom message types defined within anansi_msgs and none of the other ones have any problems. I did some testing and found that the problem with this message type is that header is the second entry in the msg file rather than the first. If I switch it to the first, then it seems like it is able to load fine.

The problem is, changing the message definition isn't a full solution because I need this tool to be able to read logs that already exist. So I am stuck with how it is defined.

Any idea why it is hallucinating the anansi_msgs/msg/Header message type and how to work around or avoid that problem?

I would love to use this tool, but if it can't read this message type I don't know if I will be able to.

If it helps, here is the full python error I am getting:

KeyError                                  Traceback (most recent call last)
<ipython-input-6-15845f67462f> in <module>
      3 with AnyReader([bagpath], default_typestore=typestore) as reader:
      4     for connection, timestamp, rawdata in reader.messages(connections=reader.connections):
----> 5         msg = reader.deserialize(rawdata, connection.msgtype)
      6         print(connection.topic)
      7 

~/.local/lib/python3.10/site-packages/rosbags/highlevel/anyreader.py in deserialize(self, rawdata, typ)
    105     def deserialize(self, rawdata: bytes, typ: str) -> object:
    106         """Deserialize message with appropriate helper."""
--> 107         return self._deser_ros2(rawdata, typ) if self.is2 else self._deser_ros1(rawdata, typ)
    108 
    109     def open(self) -> None:

~/.local/lib/python3.10/site-packages/rosbags/highlevel/anyreader.py in _deser_ros1(self, rawdata, typ)
     97     def _deser_ros1(self, rawdata: bytes, typ: str) -> object:
     98         """Deserialize ROS1 message."""
---> 99         return self.typestore.deserialize_ros1(rawdata, typ)
    100 
    101     def _deser_ros2(self, rawdata: bytes, typ: str) -> object:

~/.local/lib/python3.10/site-packages/rosbags/typesys/store.py in deserialize_ros1(self, rawdata, typename)
    180 
    181         """
--> 182         msgdef = self.get_msgdef(typename)
    183         func = msgdef.deserialize_ros1
    184         message, pos = func(rawdata, 0, msgdef.cls, self)

~/.local/lib/python3.10/site-packages/rosbags/typesys/store.py in get_msgdef(self, typename)
    313             entries = self.fielddefs[typename][1]
    314 
--> 315             getsize_cdr, size_cdr = generate_getsize_cdr(entries, self)
    316             getsize_ros1, size_ros1 = generate_getsize_ros1(entries, self)
    317 

~/.local/lib/python3.10/site-packages/rosbags/serde/cdr.py in generate_getsize_cdr(fields, typestore)
    172             is_stat = False
    173 
--> 174         if fnext and aligned < (anext_before := align(fnext[1], typestore)):
    175             lines.append(f'  pos = (pos + {anext_before} - 1) & -{anext_before}')
    176             aligned = anext_before

~/.local/lib/python3.10/site-packages/rosbags/serde/utils.py in align(entry, typestore)
     54         return SIZEMAP[entry[1][0]]
     55     if entry[0] == Nodetype.NAME:
---> 56         return align(typestore.get_msgdef(entry[1]).fields[0][1], typestore)
     57     if entry[0] == Nodetype.ARRAY:
     58         return align(entry[1][0], typestore)

~/.local/lib/python3.10/site-packages/rosbags/typesys/store.py in get_msgdef(self, typename)
    311         """
    312         if typename not in self.cache:
--> 313             entries = self.fielddefs[typename][1]
    314 
    315             getsize_cdr, size_cdr = generate_getsize_cdr(entries, self)

KeyError: 'anansi_msgs/msg/Header'

Ok I did a little bit more investigation. Here is an example of a rosbag (stripped down so it only has 2 messages. The bad one and one good one)rosbag_example.bag

I did switch the header to the first entry in the mesage and it was able to load that rosbag just fine. So the problem is that the header is not first. The problem is that we do still have to be able to load old rosbags that have already been recorded so changing the message definition is not a sufficient fix for this problem for me.

changed the description

changed the description

Update: I was able to find a work-around to get this message to load. Ideally, I could find a more elegant way to do this. but if I do the following, it is able to load the bag without error

from pathlib import Path
from rosbags.highlevel import AnyReader
from rosbags.typesys import Stores, get_typestore, get_types_from_msg

typestore = get_typestore(Stores.ROS2_HUMBLE)

# Manually define this message since it is getting loaded wrong
backup_type_store = get_typestore(Stores.ROS1_NOETIC)
nav_cmd = get_types_from_msg("""
builtin_interfaces/Time t
std_msgs/Header header
float64[3]      velocity_ned
float64[3]      accel_ned
float64         yaw_rate
""", "anansi_msgs/msg/NavCmd")
backup_type_store.register(nav_cmd)

bagpath = Path('rosbag_example.bag')
with AnyReader([bagpath], default_typestore=typestore) as reader:
    for connection, timestamp, rawdata in reader.messages(connections=reader.connections):
        try:
            msg = reader.deserialize(rawdata, connection.msgtype)
        except KeyError:
            msg = typestore.deserialize_ros1(rawdata, connection.msgtype)
        print(connection.topic)

Since it is only failing on this one message type, I just have a backup that it can fall back on that has the correct message definition manually defined (assuming it only has this issue with ros1)

assigned to @durko

@jacobmoroni Thanks for reporting this issue! When designing the .msg parser, we initially assumed that the Header would always be the first field in a message. We've now relaxed this assumption and released a fix in version 0.10.8.

closed

Thank you for the rapid response! that worked. I would agree that we shouldnt have defined the header second. So thanks for fixing that.