|
|
**Table of Contents**
|
|
|
|
|
|
[[_TOC_]] |
|
|
\ No newline at end of file |
|
|
[[_TOC_]]
|
|
|
|
|
|
----
|
|
|
# Introduction
|
|
|
|
|
|
## Protocol Buffer
|
|
|
|
|
|
[Protocol Buffer](https://developers.google.com/protocol-buffers/) is a tool developed by Google for serializing structured data which can then be used to transmit data from one medium to another. For example, sensor readings captured from a microcontroller could be sent to the cloud for logging and diagnostics after encoding the data in a serialized format for easier byte-byte transmission.
|
|
|
|
|
|
Protocol buffer can serialize data from variety of languages such as Java, Python, Objective-C C++, Dart, Go, Ruby, and C# along with running on any platform.
|
|
|
|
|
|
## Nanopb
|
|
|
[Nanopb](https://jpa.kapsi.fi/nanopb/) provides a C based library for encoding and decoding messages in Google's Protocol Buffers format with suitable for microcontrollers with less RAM and code space availability. Google's Protocol Buffer Tool can generate data structures for C++ and not for C, thus making it less suitable for Microcontrollers.
|
|
|
|
|
|
# Installation and Setup
|
|
|
|
|
|
## Setting up Nanopb
|
|
|
To use Nanopb, protocol compiler needs to be installed. Follow the [installation instructions](https://github.com/protocolbuffers/protobuf#protocol-compiler-installation) here.
|
|
|
|
|
|
Protocol Buffers messages are defined in a `proto file` as follows:
|
|
|
```
|
|
|
//foo.proto
|
|
|
message Foo {
|
|
|
required int id = 1;
|
|
|
}
|
|
|
```
|
|
|
To use Nanopb Headers(.h) and source files(.c) generation from `.proto` file is required.
|
|
|
Headers(.h) and source files(.c) are generated using a python script provided by Nanopb.
|
|
|
```
|
|
|
python generator/nanopb_generator.py foo.proto
|
|
|
```
|
|
|
Follow the [instructions here](ihttps://github.com/nanopb/nanopb#generating-the-headers).
|
|
|
|
|
|
**Note**: Sibros repository Integrates Nanopb library in [Bazel](https://www.bazel.build/). Refer to [Sibros Nanopb Bazel]() article for more information.
|
|
|
|
|
|
# Implementation and Examples
|
|
|
|
|
|
## Simple Getting started Example
|
|
|
|
|
|
### Protofiles:
|
|
|
Let's say you want to transmit a Can message in a serialized protobuf format. Then we need to define the can message and its contents inside a proto file as follows:
|
|
|
|
|
|
```
|
|
|
//can.proto
|
|
|
message can_message {
|
|
|
required int64 timestamp_ms = 1;
|
|
|
|
|
|
required int32 bus_id = 2; // The bus number which received the CAN message
|
|
|
required int32 message_id = 3; // Standard 11-bit
|
|
|
required int32 data_byte = 4; // Usually 8 bytes but modified for simplified example
|
|
|
}
|
|
|
```
|
|
|
In this file a message to be transmitted is defined using `message` keyword, followed by message name. There are 3 members inside the message with int64 and int32 data types respectively. The member of the message has to be initialized if `required` keyword is used. Alternatively `option` keyword can also be used. The `required int64 timestamp_ms = 1` the number after equality operator is the tag of the message member, which are used to match fields when serializing and deserializing the data.
|
|
|
For more information about `proto` message, visit the [documentation.](https://developers.google.com/protocol-buffers/docs/overview).
|
|
|
|
|
|
On running the python script to generate header and source files from proto file, we get structure declared in (.h) file:
|
|
|
```
|
|
|
/* Struct definitions */
|
|
|
typedef struct _can_message {
|
|
|
int64_t timestamp_ms;
|
|
|
int32_t bus_id;
|
|
|
int32_t message_id;
|
|
|
int32_t data_byte;
|
|
|
/* @@protoc_insertion_point(struct:can_message) */
|
|
|
} can_message;
|
|
|
```
|
|
|
|
|
|
### Encoding:
|
|
|
|
|
|
Now we can use this generated structure to populate our data to be serialized as follows:
|
|
|
|
|
|
```
|
|
|
proto_msg->timestamp_ms = can_msg->timestamp_ms;
|
|
|
proto_msg->bus_id = can_msg->bus_id;
|
|
|
proto_msg->message_id = can_msg->message_id;
|
|
|
proto_msg->data_byte = can_msg->data[0];
|
|
|
```
|
|
|
|
|
|
We then create an output stream to write data to the buffer:
|
|
|
|
|
|
```
|
|
|
pb_ostream_t stream = pb_ostream_from_buffer(encoded_packet->buffer, encoded_packet->encoded_packet_size);
|
|
|
```
|
|
|
|
|
|
The message is can we Encoded using Nanopb using the Nanopb's encoding function
|
|
|
```
|
|
|
if (!pb_encode(&stream, proto_can_message_fields, proto_msg)) {
|
|
|
printf("Encoding failed: %s\n", PB_GET_ERROR(&stream));
|
|
|
return 1;
|
|
|
}
|
|
|
```
|
|
|
|
|
|
### Decoding
|
|
|
|
|
|
To verify integrity of encoded data, we decode the encoded packet. First we create a stream that reads from the encoded buffer.
|
|
|
|
|
|
```
|
|
|
pb_istream_t stream = pb_istream_from_buffer(packet->buffer, packet->bytes_written);
|
|
|
```
|
|
|
The we decode the encoded message by using Nanopb's decoding API such that proto message ie `proto_can_message* proto_msg` is re-populated.
|
|
|
```
|
|
|
pb_decode(&stream, proto_can_message_fields, proto_msg)
|
|
|
```
|
|
|
Finally we re-populate the can message using the newly populated `proto_msg` ie. Copy decoded data to can message
|
|
|
```
|
|
|
can_msg->message_id = proto_msg->message_id;
|
|
|
can_msg->bus_id = proto_msg->bus_id;
|
|
|
can_msg->timestamp_ms = proto_msg->timestamp_ms;
|
|
|
can_msg->data = proto_msg->data_byte;
|
|
|
```
|
|
|
|
|
|
## Using Callback for Nanopb
|
|
|
|
|
|
A CAN message usually has a 8 byte data field (if using CAN-FD the data field is 64 bytes). Thus we modify the can message in the `proto` file such that the CAN proto message can encode 8 byte of data at once.
|
|
|
|
|
|
```
|
|
|
//can.proto
|
|
|
message can_message {
|
|
|
required int64 timestamp_ms = 1;
|
|
|
required int32 bus_id = 2; // The bus number which received the CAN message
|
|
|
required int32 message_id = 3; // Standard 11-bit
|
|
|
bytes data_byte = 4; // Usually 8 bytes but modified for simplified example
|
|
|
}
|
|
|
```
|
|
|
|
|
|
### Encoding:
|
|
|
|
|
|
After generating the (.h) file again the message is modified as follows:
|
|
|
|
|
|
```
|
|
|
/* Struct definitions */
|
|
|
typedef struct _proto_can_message {
|
|
|
int64_t timestamp_ms;
|
|
|
int32_t bus_id;
|
|
|
int32_t message_id;
|
|
|
pb_callback_t data_byte;
|
|
|
/* @@protoc_insertion_point(struct:proto_can_message) */
|
|
|
} proto_can_message;
|
|
|
```
|
|
|
|
|
|
**Callbacks:**
|
|
|
|
|
|
Callbacks are used when the members of a message have variable length and storage is not statically allocated to it. For example if a message contains a string such as `string name` instead of generating `char *name` Nanopb generates the `name` variable of `pb_callback_t` datatype. This allows the user to allocated a name with N number of chars using a custom call back function. Thus callback instance of message member are generated for variable length arrays, strings, repeated specifier
|
|
|
messages or data structures (which is demonstrated shortly). More Information [here.](https://jpa.kapsi.fi/nanopb/docs/concepts.html#field-callbacks)
|
|
|
|
|
|
The callback structure is declared as follows:
|
|
|
```
|
|
|
typedef struct _pb_callback_t pb_callback_t;
|
|
|
struct _pb_callback_t {
|
|
|
union {
|
|
|
bool (*decode)(pb_istream_t *stream, const pb_field_iter_t *field, void **arg);
|
|
|
bool (*encode)(pb_ostream_t *stream, const pb_field_iter_t *field, void * const *arg);
|
|
|
} funcs;
|
|
|
|
|
|
void *arg;
|
|
|
};
|
|
|
```
|
|
|
The pb_callback_t structure contains a function pointer and a void pointer called arg you can use for passing data to the callback. If the function pointer is NULL, the field will be skipped. A pointer to the arg is passed to the function, so that it can modify it and retrieve the value.
|
|
|
|
|
|
The proto message is now populated and encoded follows:
|
|
|
|
|
|
```
|
|
|
proto_msg->timestamp_ms = can_msg->timestamp_ms;
|
|
|
proto_msg->bus_id = can_msg->bus_id;
|
|
|
proto_msg->message_id = can_msg->message_id;
|
|
|
proto_msg->data_byte = can_msg->data[0];
|
|
|
|
|
|
// Populate `pb_callback_t data_bytes` field
|
|
|
proto_msg->data_bytes.arg = (void *)can_msg->data;
|
|
|
proto_msg->data_bytes.funcs.encode = &encode_can_bytes;
|
|
|
|
|
|
if (!pb_encode(&stream, proto_can_message_fields, proto_msg)) {
|
|
|
printf("Encoding failed: %s\n", PB_GET_ERROR(&stream));
|
|
|
return 1;
|
|
|
}
|
|
|
```
|
|
|
|
|
|
A callback function is created to encode the variable amount of bytes in the can message.
|
|
|
|
|
|
```
|
|
|
bool callback_decode_can_bytes(pb_ostream_t *ostream, const pb_field_t *field, void *const *arg) {}
|
|
|
```
|
|
|
We deference the `arg` pointer which points to the array of CAN messages, get the count of CAN messages inside the array and encode all bytes at once using Nanopb's `pb_encode_string` [API](https://jpa.kapsi.fi/nanopb/docs/reference.html#pb-encode-string).
|
|
|
```
|
|
|
const can_message_s *const can_messages = (const can_message_s *)(*arg);
|
|
|
const uint8_t *const can_byte = (const uint8_t *)can_messages->data;
|
|
|
const size_t can_message_byte_count = can_messages->dlc;
|
|
|
|
|
|
if (!pb_encode_tag_for_field(ostream, field)) {
|
|
|
is_success = false;
|
|
|
}
|
|
|
if (!pb_encode_string(ostream, can_byte, can_message_byte_count)) {
|
|
|
is_success = false;
|
|
|
}
|
|
|
```
|
|
|
|
|
|
### Decoding
|
|
|
|
|
|
Decoding the packet is similar to the first example, however we make use Callback structure to decoded multiple CAN bytes.
|
|
|
|
|
|
The `callback_t` structure members are assigned the pointer to callback function and CAN message data structure which will hold the decoded CAN message. The Nanopb's `pb_decode` API is then called.
|
|
|
|
|
|
```
|
|
|
bool decode_packet(proto_can_message *proto_msg, serial_data_packet_s *packet, can_message_s *can_msg) {
|
|
|
....
|
|
|
proto_msg->data_byte.arg = can_msg;
|
|
|
proto_msg->data_byte.funcs.decode = &callback_decode_can_bytes;
|
|
|
|
|
|
pb_decode(&stream, proto_can_message_fields, proto_msg);
|
|
|
....
|
|
|
}
|
|
|
```
|
|
|
The Callback to decode multiple CAN bytes reads all the bytes at once using `pb_read` [API.](https://jpa.kapsi.fi/nanopb/docs/reference.html#pb-read) `pb_byte_t pb_bytes` is essentially a byte array. `istream->bytes_left` argument to `pb_read` API informs the Nanopb how many bytes are left to be read.
|
|
|
|
|
|
```
|
|
|
static bool callback_decode_can_bytes(pb_istream_t *istream, const pb_field_t *field, void **arg)
|
|
|
...
|
|
|
// Read 8 bytes
|
|
|
pb_byte_t pb_bytes[8 + 1] = {0};
|
|
|
|
|
|
pb_read(istream, pb_bytes, istream->bytes_left)
|
|
|
|
|
|
// Copy decoded stream to CAN message data bytes
|
|
|
memcpy(can_messages->data, pb_bytes, sizeof(can_messages->data));
|
|
|
```
|
|
|
|
|
|
|
|
|
# External Resources
|
|
|
* [Protocol Buffer](https://developers.google.com/protocol-buffers/)
|
|
|
* [Nanopb](https://jpa.kapsi.fi/nanopb/)
|
|
|
* [Nanopb Github](https://github.com/nanopb/nanopb)
|
|
|
* [Nanopb simple callback example](https://stackoverflow.com/questions/45979984/creating-callbacks-and-structs-for-repeated-field-in-a-protobuf-message-in-nanop) |