Detect when nodes' incoming messages are skipped
Description
Add a mechanism built into nodes to check that all incoming topics are arriving without skipping any messages. We have ros2 topic hz
, but that requires manually monitoring nodes, and it's not very precise.
Purpose
Currently we don't have a good way when nodes can't keep up with their inputs. I think that's a big hazard – the software should warn about this, or the functional performance will degrade in pretty unpredictable ways.
One type of check is a timestamp frequency check: for an input topic that should be running at 10HZ, the difference of current and previous timestamp should be within e.g. 80ms and 120ms.
Desired Behavior
When timestamp frequency monitoring is active and a message is skipped, the node could exit with a message. We could also log a warning, but it is easy to miss or ignore those.
Implementation
We could create a child class of rclcpp::Node
that wraps create_subscription
and defines optional parameters with the expected delta and tolerance, and then does the check in a wrapper around the actual callback.
That approach would enable many more types of instrumentation in the future, e.g. measuring the time spent in callbacks.
Definition of Done
-
Agreement on the right approach to do this -
Create a subclass with the desired functionality -
Create a new issue tracking the rollout to nodes