Serialization and deserialization of empty sequences
I ran into an issue when trying to deserialize messages of the type rcl_interfaces/msg/Parameter. I was getting an error about unpack_from requiring a larger buffer than it was provided. I spent some time troubleshooting the problem and I think I've found the root cause and a means of fixing it.
The Issue
When deserializing a sequence, you follow these steps:
- Make sure you are 4-aligned
- Read a in the length of the sequence
- Make sure you are N-aligned, where N is the alignment requirement of the sequence element data type
- Read in all the sequence elements
I can't find any definitive answer in the Official CDR Specification as to how you're supposed to handle a sequence of length 0. The rosbag code performs step 3 even when there are no elements in the sequence. After doing some manual deserialization of the data from a bag file, I believe that the rosbag2 code only performs step 3 if there are elements in the sequence.
The result of this discrepancy is that when there is an empty sequence of 8-aligned elements, the rosbags code may read in 4 bytes of actual data, thinking that it's padding to get 8-aligned.
A similar issue would occur during serialization of an empty message, but this time it may add in extra padding.
Proposed Solution
After making the following modifications to my local installation of rosbags, the issue appears to be resolved.
In src/rosbags/serde/cdr.py:
...
# Line 119, in generate_getsize_cdr:
else:
anext = align(subdesc)
if aligned < anext:
lines.append(f' if len(message.{fieldname}) > 0:') # New
lines.append(f' pos = (pos + {anext} - 1) & -{anext}') # Added indentation
aligned = min(aligned, anext) # We may not be anext-aligned
lines.append(f' pos += len(message.{fieldname}) * {SIZEMAP[subdesc.args]}')
...
# Line 270, in generate_serialize_cdr:
else:
lines.append(f' size = len(val) * {SIZEMAP[subdesc.args]}')
if (endianess == 'le') != (sys.byteorder == 'little'):
lines.append(' val = val.byteswap()')
if aligned < (anext := align(subdesc)):
lines.append(' if size > 0:') # New
lines.append(f' pos = (pos + {anext} - 1) & -{anext}') # Added indentation
lines.append(' rawdata[pos:pos + size] = val.view(numpy.uint8)')
lines.append(' pos += size')
aligned = min(aligned, anext) # We may not be anext-aligned
...
# Line 417, in generate_deserialize_cdr:
else:
lines.append(f' length = size * {SIZEMAP[subdesc.args]}')
if aligned < (anext := align(subdesc)):
lines.append(' if length > 0:') # New
lines.append(f' pos = (pos + {anext} - 1) & -{anext}') # Added indentation
lines.append(
f' val = numpy.frombuffer(rawdata, '
f'dtype=numpy.{subdesc.args}, count=size, offset=pos)',
)
if (endianess == 'le') != (sys.byteorder == 'little'):
lines.append(' val = val.byteswap()')
lines.append(' values.append(val)')
lines.append(' pos += length')
aligned = min(aligned, anext) # We may not be anext-aligned
...In src/rosbags/serde/ros1.py:
...
# Line 152, in generate_ros1_to_cdr:
else:
if aligned < (anext := align(subdesc)):
lines.append(' if size > 0:') # New
lines.append(f' opos = (opos + {anext} - 1) & -{anext}') # Added indentation
lines.append(f' length = size * {SIZEMAP[subdesc.args]}')
if copy:
lines.append(' output[opos:opos + length] = input[ipos:ipos + length]')
lines.append(' ipos += length')
lines.append(' opos += length')
aligned = min(aligned, anext) # We may not be anext-aligned
...
# Line 305, in generate_cdr_to_ros1:
else:
if aligned < (anext := align(subdesc)):
lines.append(' if size > 0:') # New
lines.append(f' ipos = (ipos + {anext} - 1) & -{anext}') # Added indentation
lines.append(f' length = size * {SIZEMAP[subdesc.args]}')
if copy:
lines.append(' output[opos:opos + length] = input[ipos:ipos + length]')
lines.append(' ipos += length')
lines.append(' opos += length')
aligned = min(aligned, anext) # We may not be anext-aligned
...