... | ... | @@ -14,16 +14,60 @@ Protocol buffer can serialize data from variety of languages such as Java, Pytho |
|
|
## Nanopb
|
|
|
[Nanopb](https://jpa.kapsi.fi/nanopb/) provides a C based library for encoding and decoding messages in Google's Protocol Buffers format with suitable for microcontrollers with less RAM and code space availability. Google's Protocol Buffer Tool can generate data structures for C++ and not for C, thus making it less suitable for Microcontrollers.
|
|
|
|
|
|
## Why do we need Protocol Buffers (or Nanopb)?
|
|
|
Consider the scenario where we need to transmit [CAN message](https://en.wikipedia.org/wiki/CAN_bus) frames from vehicle to the cloud for data logging and analysis. The data of a CAN frame can be held in the following C Data structure:
|
|
|
```c
|
|
|
typedef struct {
|
|
|
// 8 bytes:
|
|
|
uint32_t message_id; ///< Message ID of the CAN bus message
|
|
|
uint32_t timestamp_ms; ///< The receive timestamp(in microseconds) of the message
|
|
|
|
|
|
// 2 bytes:
|
|
|
uint8_t dlc; ///< Data length code, 0-8 bytes
|
|
|
uint8_t bus_id : 4; ///< Identifies which CAN bus this message belongs to
|
|
|
uint8_t ide : 1; ///< ID Extended
|
|
|
uint8_t rtr : 1; ///< Remote Transmission Request
|
|
|
|
|
|
// Usually has 8 bytes.
|
|
|
uint8_t data[8]; //* byte data
|
|
|
} can_message_s;
|
|
|
```
|
|
|
Now we need to send this data message to the cloud. A popular choice is using [BSD Sockets](https://en.wikipedia.org/wiki/Berkeley_sockets). After establishing connection with the cloud and obtaining the socket file descriptor `sockfd` we can use [`send` Socket API](https://pubs.opengroup.org/onlinepubs/009695399/functions/send.html) to send our data.
|
|
|
```
|
|
|
/**
|
|
|
* @param sockfd Specifies the socket file descriptor.
|
|
|
* @param buf Points to the buffer containing the message to send.
|
|
|
* @param len Specifies the length of the message in bytes.
|
|
|
*
|
|
|
* @return number of bytes sent.
|
|
|
*/
|
|
|
ssize_t send(int sockfd, const void *buf, size_t len, int flags);
|
|
|
```
|
|
|
Since the `send` API requires an array of bytes to be transferred, we need a way to convert our data in the CAN based C strucutre above to serialized data bytes. This is where Protocol Buffers (or Nanopb) comes into the play. Protocol Buffers will serialize the CAN data strucuture as an array of bytes in an efficient manner which can then be transfered to the cloud server using Socket `send` API.
|
|
|
|
|
|
# Getting started
|
|
|
|
|
|
Start with reading the following materials which are very useful in getting up to speed about protobufs:
|
|
|
|
|
|
1. [Google's Documentation:](https://developers.google.com/protocol-buffers/docs/tutorials) A great tutorial with simple examples and common API explanation.
|
|
|
|
|
|
2. [Nanopb Documentation](https://jpa.kapsi.fi/nanopb/docs/)
|
|
|
|
|
|
|
|
|
# Installation and Setup
|
|
|
|
|
|
## Setting up Nanopb
|
|
|
## For Sibros Employees:
|
|
|
|
|
|
Sibros repository Integrates Nanopb library in [Bazel](https://www.bazel.build/). `cc_nanopb_library` Bazel rule is used to auto-generate C strucutres from `.proto` file in the form of an header(.h) file. Refer to the Bazel [`BUILD`](https://gitlab.com/sibros_public/sibros-open-source/nanopb-examples/-/blob/master/ex1/BUILD) file in the [Nanopb examples](https://gitlab.com/sibros_public/sibros-open-source/nanopb-examples) to generate the headers(.h) and source files(.c).
|
|
|
|
|
|
## For others:
|
|
|
|
|
|
To use Nanopb, protocol compiler needs to be installed.
|
|
|
Install `Python3` and `Python3-Pip` package:
|
|
|
```
|
|
|
apt-get install python3 python3-pip
|
|
|
```
|
|
|
Install `protobuf` and `grpcio-tools` packages need by Nanopb
|
|
|
Install `protobuf` and `grpcio-tools` packages need by Nanopb:
|
|
|
```
|
|
|
pip3 install protobuf grpcio-tools
|
|
|
```
|
... | ... | @@ -40,14 +84,14 @@ This `.proto` file is used to generate the Nanopb headers(.h) and source files(. |
|
|
python3 ../../generator/nanopb_generator.py foo.proto
|
|
|
```
|
|
|
|
|
|
**Note**: Sibros repository Integrates Nanopb library in [Bazel](https://www.bazel.build/). Refer to the Bazel `BUILD` file in the Nanopb examples to generate the headers(.h) and source files(.c).
|
|
|
|
|
|
# Implementation and Examples
|
|
|
|
|
|
## Simple Getting started Example
|
|
|
|
|
|
Download the example files from our [public repository](https://gitlab.com/sibros_public/sibros-open-source/nanopb-examples).
|
|
|
|
|
|
**Note:** The examples should work out-of-the-box (if bazel targets are correctly defined by the user) for Sibro's Employees since all Bazel rules and BUILD files are setup. However, others need to create a `Makefile` to build and link all the source files (and also Nanopb library) for creating a binary target. I will add a `Makefile` for all the examples soon.
|
|
|
|
|
|
## Example - 1
|
|
|
|
|
|
### Proto Files
|
|
|
Let's say you want to transmit a Can message in a serialized protobuf format. Then we need to define the can message and its contents inside a proto file as follows:
|
|
|
|
... | ... | @@ -120,7 +164,7 @@ Finally we re-populate the can message using the newly populated `proto_msg` ie. |
|
|
can_msg->data = proto_msg->data_byte;
|
|
|
```
|
|
|
|
|
|
## Using Callbacks for Nanopb
|
|
|
## Example - 2 : Using Callbacks for Nanopb
|
|
|
|
|
|
A CAN message usually has a 8 byte data field (if using CAN-FD the data field is 64 bytes). Thus we modify the can message in the `proto` file such that the CAN proto message can encode 8 byte of data at once.
|
|
|
|
... | ... | @@ -263,7 +307,7 @@ static bool callback_decode_can_bytes(pb_istream_t *istream, const pb_field_t *f |
|
|
memcpy(can_messages->data, pb_bytes, sizeof(can_messages->data));
|
|
|
```
|
|
|
|
|
|
## Example: Nested Callback Structures
|
|
|
## Example - 3 : Nested Callback Structures
|
|
|
|
|
|
Consider we have the following `.proto` file:
|
|
|
```
|
... | ... | @@ -444,10 +488,10 @@ static bool decode_callback_can_messages(pb_istream_t *istream, const pb_field_t |
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# External Resources
|
|
|
* [Protocol Buffer](https://developers.google.com/protocol-buffers/)
|
|
|
* [Nanopb](https://jpa.kapsi.fi/nanopb/)
|
|
|
* [Protocol Buffer Introduction](https://developers.google.com/protocol-buffers/)
|
|
|
* [Protocol Buffer Tutorial](https://developers.google.com/protocol-buffers/docs/tutorials)
|
|
|
* [Nanopb Introduction](https://jpa.kapsi.fi/nanopb/)
|
|
|
* [Nanopb Documentation](https://jpa.kapsi.fi/nanopb/docs/)
|
|
|
* [Nanopb Github](https://github.com/nanopb/nanopb)
|
|
|
* [Nanopb simple callback example](https://stackoverflow.com/questions/45979984/creating-callbacks-and-structs-for-repeated-field-in-a-protobuf-message-in-nanop) |