... | ... | @@ -7,12 +7,12 @@ |
|
|
|
|
|
## Protocol Buffer
|
|
|
|
|
|
[Protocol Buffer](https://developers.google.com/protocol-buffers/) is a tool developed by Google for serializing structured data which can then be used to transmit data from one medium to another. For example, sensor readings captured from a microcontroller could be sent to the cloud for logging and diagnostics after encoding the data in a serialized format for easier byte-byte transmission.
|
|
|
[Protocol Buffer](https://developers.google.com/protocol-buffers/) is a tool developed by Google for serializing structured data which can then be used to transmit data from one medium to another. For example, sensor readings captured from a microcontroller could be sent to the cloud for logging and diagnostics after encoding the data in a serialized format for easier transmission of data bytes.
|
|
|
|
|
|
Protocol buffer can serialize data from variety of languages such as Java, Python, Objective-C C++, Dart, Go, Ruby, and C# along with running on any platform.
|
|
|
|
|
|
## Nanopb
|
|
|
Google's Protocol Buffer Tool can generate data structures for C++ and not for C, thus making it less suitable for microcontrollers with limited RAM and code space availability. [Nanopb](https://jpa.kapsi.fi/nanopb/) provides a C based library for encoding and decoding messages in Google's Protocol Buffers format.
|
|
|
Google's Protocol Buffer Tool can generate data structures for C++ and not for C. Since microcontrollers have limited RAM and code memory, the embedded industry does not prefer to program in C++, thus making Google's Protocol Buffer tool less suitable. [Nanopb](https://jpa.kapsi.fi/nanopb/) provides a C based library for encoding and decoding messages in Google's Protocol Buffers format.
|
|
|
|
|
|
## Why do we need Protocol Buffers (or Nanopb)?
|
|
|
Consider the scenario where we need to transmit [CAN message](https://en.wikipedia.org/wiki/CAN_bus) frames from vehicle to the cloud for data logging and analysis. The data of a CAN frame can be held in the following C Data structure:
|
... | ... | @@ -32,7 +32,7 @@ typedef struct { |
|
|
uint8_t data[8]; //* byte data
|
|
|
} can_message_s;
|
|
|
```
|
|
|
Now we need to send this data message to the cloud. A popular choice is using [BSD Sockets](https://en.wikipedia.org/wiki/Berkeley_sockets). After establishing connection with the cloud and obtaining the socket file descriptor `sockfd` we can use [`send` Socket API](https://pubs.opengroup.org/onlinepubs/009695399/functions/send.html) to send our data.
|
|
|
For sending the data held in the data structure above to the cloud, a popular choice is using [BSD Sockets](https://en.wikipedia.org/wiki/Berkeley_sockets). After establishing connection with the cloud and obtaining the socket file descriptor `sockfd` we can use [`send` Socket API](https://pubs.opengroup.org/onlinepubs/009695399/functions/send.html) to send our data.
|
|
|
```c
|
|
|
/**
|
|
|
* @param sockfd Specifies the socket file descriptor.
|
... | ... | @@ -43,7 +43,7 @@ Now we need to send this data message to the cloud. A popular choice is using [B |
|
|
*/
|
|
|
ssize_t send(int sockfd, const void *buf, size_t len, int flags);
|
|
|
```
|
|
|
Since the `send` API requires an array of bytes to be transferred, we need a way to convert our data to CAN based C structure above to serialized data bytes. This is where Protocol Buffers (or Nanopb) comes into play. Protocol Buffers will serialize the CAN data structure as an array of bytes in an efficient manner which can then be transferred to the cloud server using Socket `send` API.
|
|
|
Since the `send` API requires an array of bytes to be transferred, we need a way to convert our CAN based C data structure to serialized data bytes. This is where Protocol Buffers (or Nanopb) comes into play. Protocol Buffers will serialize the CAN data structure as an array of bytes in an efficient manner which can then be transferred to the cloud server using Socket `send` API.
|
|
|
|
|
|
# Getting started
|
|
|
|
... | ... | @@ -58,7 +58,7 @@ Start with reading the following materials which are very useful in getting up t |
|
|
|
|
|
## For Sibros Employees:
|
|
|
|
|
|
Sibros repository Integrates Nanopb library in [Bazel](https://www.bazel.build/). `cc_nanopb_library` Bazel rule is used to auto-generate C strucutres from `.proto` file in the form of an header(.h) file. Refer to the Bazel [`BUILD`](https://gitlab.com/sibros_public/sibros-open-source/nanopb-examples/-/blob/master/ex1/BUILD) file in the [Nanopb examples](https://gitlab.com/sibros_public/sibros-open-source/nanopb-examples) to generate the headers(.h) and source files(.c).
|
|
|
Sibros internal repository integrates Nanopb library with [Bazel](https://www.bazel.build/). Custom `cc_nanopb_library` Bazel rule is used to auto-generate C strucutres from `.proto` file in the form of an header(.h) file. Refer to the Bazel [`BUILD`](https://gitlab.com/sibros_public/sibros-open-source/nanopb-examples/-/blob/master/ex1/BUILD) file in the [Nanopb examples](https://gitlab.com/sibros_public/sibros-open-source/nanopb-examples) to generate the headers(.h) and source files(.c).
|
|
|
|
|
|
## For others:
|
|
|
|
... | ... | @@ -157,7 +157,7 @@ $ cd nanopb |
|
|
$ python3 generator/nanopb_generator.py /path/to/nanopb-examples/ex1/ex1.proto
|
|
|
```
|
|
|
|
|
|
The Nanopb generated header (.h) file will create an proto message equivalent C strucutre as follows:
|
|
|
The Nanopb generated header (.h) file will create an proto message equivalent C structure as follows:
|
|
|
```c
|
|
|
/* Struct definitions */
|
|
|
typedef struct _proto_can_message {
|
... | ... | @@ -169,7 +169,7 @@ typedef struct _proto_can_message { |
|
|
} proto_can_message;
|
|
|
```
|
|
|
|
|
|
Refer to the complete Nanopb generated header [here]().
|
|
|
Refer to the complete Nanopb generated header [here](https://gitlab.com/sibros_public/sibros-open-source/nanopb-examples/-/blob/master/ex1/ex1.npb.h).
|
|
|
|
|
|
### Encoding data using Nanopb
|
|
|
|
... | ... | @@ -206,7 +206,7 @@ typedef struct encoded_packet { |
|
|
} serial_data_packet_s;
|
|
|
```
|
|
|
|
|
|
Now we can use the generated structure to populate our data to be serialized as follows:
|
|
|
Now we can populate the Nanopb generated data strucutre `proto_msg` with our CAN message to be serialized as follows:
|
|
|
|
|
|
```c
|
|
|
size_t encode_packet(proto_can_message* proto_msg, serial_data_packet_s* packet, can_message_s* can_msg) {
|
... | ... | @@ -226,14 +226,14 @@ We then create an output stream for writing the data to the buffer: |
|
|
size_t encode_packet(proto_can_message* proto_msg, serial_data_packet_s* packet, can_message_s* can_msg) {
|
|
|
/* Fill the proto packet */
|
|
|
...
|
|
|
|
|
|
pb_ostream_t stream = pb_ostream_from_buffer(encoded_packet->buffer, encoded_packet->encoded_packet_size);
|
|
|
|
|
|
}
|
|
|
```
|
|
|
`pb_ostream_t pb_ostream_from_buffer(pb_byte_t *buf, size_t bufsize)` constructs an output stream for writing into a memory buffer. It uses an internal callback that stores the pointer in stream state field. `pb_ostream_t stream` will hold the output stream buffer.
|
|
|
* `pb_ostream_t pb_ostream_from_buffer(pb_byte_t *buf, size_t bufsize)` constructs an output stream for writing data bytes into a memory buffer. It uses an internal callback function that stores the pointer in stream state field.
|
|
|
* `pb_ostream_t stream` will hold the output stream buffer.
|
|
|
|
|
|
The Protobuf message is then encoded to a serialized format using Nanopb's `pb_encode()` API. The number of bytes encoded is obtained from `stream.bytes_written` data member which is then returned.
|
|
|
The Protobuf message is then encoded to a serialized format using Nanopb's `pb_encode()` API. The number of bytes encoded is obtained from `stream.bytes_written` data member.
|
|
|
```c
|
|
|
size_t encode_packet(proto_can_message* proto_msg, serial_data_packet_s* packet, can_message_s* can_msg) {
|
|
|
size_t number_bytes_encoded = 0;
|
... | ... | @@ -254,9 +254,9 @@ size_t encode_packet(proto_can_message* proto_msg, serial_data_packet_s* packet, |
|
|
}
|
|
|
```
|
|
|
|
|
|
`pb_encode()` function encodes the contents of a structure as a protocol buffers message and writes it to output stream. Visit [here](https://jpa.kapsi.fi/nanopb/docs/reference.html#pb_encode) for more reference.
|
|
|
`pb_encode()` function encodes the contents of a proto message C structure and writes it to output stream. Visit [Nanopb Docs](https://jpa.kapsi.fi/nanopb/docs/reference.html#pb_encode) for more information on `pb_encode()`.
|
|
|
|
|
|
`proto_can_message_fields` argument to the `pb_encode()` function is an auto-generated Struct field which will encoding specification for Nanopb. Visit [here](https://github.com/nanopb/nanopb/blob/master/pb.h#L315) for more reference. Also visit the auto-generated Nanopb [header file]() for reference.
|
|
|
`proto_can_message_fields` argument to the `pb_encode()` function is an auto-generated Struct field which will encoding specification for Nanopb. Visit [Nanopb Docs]](https://github.com/nanopb/nanopb/blob/master/pb.h#L315) for more information on `proto_can_message_fields`. Also visit the auto-generated Nanopb [header file](https://gitlab.com/sibros_public/sibros-open-source/nanopb-examples/-/blob/master/ex1/ex1.npb.h) for reference.
|
|
|
|
|
|
At this point data bytes are encoded in a serialized format and stored in the `serial_data_packet_s` memory structure.
|
|
|
|
... | ... | @@ -269,13 +269,10 @@ To verify the integrity of encoded data, we decode the encoded packet. First, we |
|
|
```c
|
|
|
pb_istream_t stream = pb_istream_from_buffer(packet->buffer, packet->bytes_written);
|
|
|
```
|
|
|
Visit [here](https://jpa.kapsi.fi/nanopb/docs/reference.html#pb_istream_from_buffer) for more reference on `pb_istream_from_buffer`.
|
|
|
|
|
|
Then we decode the encoded message by using Nanopb's decoding API such that proto message ie, `proto_can_message* proto_msg` is re-populated.
|
|
|
Then we decode the encoded message by using Nanopb's decoding API such that the proto message (ie. `proto_can_message* proto_msg`) is re-populated.
|
|
|
```c
|
|
|
pb_decode(&stream, proto_can_message_fields, proto_msg)
|
|
|
```
|
|
|
Visit [here](https://jpa.kapsi.fi/nanopb/docs/reference.html#pb_decode) for more reference on `pb_decode`.
|
|
|
|
|
|
Finally, we re-populate the CAN message using the newly populated `proto_msg` ie. Copy decoded data to CAN message.
|
|
|
```c
|
... | ... | @@ -284,7 +281,9 @@ Finally, we re-populate the CAN message using the newly populated `proto_msg` ie |
|
|
can_msg->timestamp_ms = proto_msg->timestamp_ms;
|
|
|
can_msg->data = proto_msg->data_byte;
|
|
|
```
|
|
|
Refer the [source files](https://gitlab.com/sibros_public/sibros-open-source/nanopb-examples/-/tree/master/ex1) for more context.
|
|
|
* Refer the [source files](https://gitlab.com/sibros_public/sibros-open-source/nanopb-examples/-/tree/master/ex1) for more context.
|
|
|
* Visit [Nanopb Docs](https://jpa.kapsi.fi/nanopb/docs/reference.html#pb_istream_from_buffer) for more reference on `pb_istream_from_buffer`.
|
|
|
* Visit [Nanopb Docs](https://jpa.kapsi.fi/nanopb/docs/reference.html#pb_decode) for more reference on `pb_decode`.
|
|
|
|
|
|
## Example - 2 : Using Callbacks for Nanopb
|
|
|
|
... | ... | |