Gentle and in depth introduction to Google protocol buffers(protobufs)

Image result

I met protobufs while trying to understand how messages are sent from device to the cloud using LoRaWAN implementations such as that by TheThingsNetwok.

What protobufs is:

  • A protocol that is used to serially encode and decode structured data to be sent out a wire(usually meaning communication path from sender to receiver). It also defines the endianness of the data along the wire. It can be used to send structured data.
  • It can be used to send data across machines and ensures that the transformation will happen transparently. Example: Sending a 32 bit unsigned integer from an embedded system running C is received as the same data type on a server running node.js. Another example: Sending data between servers with different implementation languages. This avoids the need to avoid sending data as text (XML, Json).

  • Attaining operational efficiency irregardless of language implementation. This is because the implementations are optimised.
  • Optionally – A poor man’s means of attaining variable length compression(I will get to this is a bit).

A typical usage scenario is as below:

  • You have structured, formatted data that is to be sent from a sender to receiver, this can be fields made up of text.
  • You define the fields in the messages that make up the protocol between your sender and receiver(Device to Device for example) and you are aware that the protocol may expand messages and/or fields so it follows semantic versioning to take care of backward compatibility and breaking changes.
  • The sender and receiver implementations use protobuf-supported languages by google or third parties. Embedded device languages included such as C.

How to use protobufs

  1. Define the protocol messages and follow as described here
  2. Create the .proto file as above that describes the messages. This will be the input to the protobuf compiler. The protobuf compiler has backends that generates code for different languages from the same .proto file. This ensures that the same message sent from a client in C is received by a server in Golang and decoded correctly.
  3. Create code in the sender and receiver implementation languages that calls the code generated by the protobuf compiler. The files generated by the protobuf compiler are textual and readable.

In depth view of protobuf message formats

A sample structured message is passed between a sender and receiver that describes a person’s data and is described as below and as opposed to xml and Json which only represent text, this message entity contains text and integers :

message Human {

char name /*Field 1*/

uint8 age /*Field 2*/

message parents= /*Field 3*/

{

bool mother /*has father*/ /*Nested field 1 of field 3*/

bool father /*Has mother*/ /*Nested field 2 of field 3*/

}
}

Each entry in the unencoded message is called a field and messages may be nested to represent the entity to be sent. A protobuf encoding thus transforms the nested tree structure above to a one dimensional structure(buffer/queue) sent along the wire in FIFO(First in-first out) format.

The encoding is done on a per field basis and a field that has been encoded(buffer) looks as below:

The data is sent along the wire as key-value pairs, where the key informs the parser of how to reconstruct structure. The field number of each field is sent first per field.

Field key encoding

The key is defined as the section of the serialised field that uniquely identifies the payload in the nested data structure to be reconstructed at the receiver. The key and value are encoded separately.

The key has the following sub-sections:

  • Field number=defines the entry by a number in the structure. Example: Age field in the human structure is defined as 2.
  • Wiretype- Defines the encoding of the field payload data. Several field payload encodings are available as below:

The Wiretype is defined as 3 bits in the protocol buffer specification(integer 0-7), only 0-5 are occupied and 6,7 reserved for future use.

For the most common field payload encodings, varints are used. Varints apply an encoding known as Little Endian Base 128.(LEB128) .Google protobufs use an LEB128 format know as unsigned LEB128 as shown in the link above on Wikipedia.

Other encodings can be found at https://developers.google.com/protocol-buffers/docs/encoding

How protobuf compiler transforms the key field section to a buffer format

The key section of the key-value pair undergoes Varint encoding. The key of a key-value pair is defined as:

Each Varint encoding occurs as groups of 7 bits.

Google states:

Therefore:

  • Minimum number of fields that can be sent is 1. This field key is thus encoded as a byte.
  • The key for a structure with 1-15 fields whose field number/TAG is in range 1-15 is represented as 4 bits(0000-1111), The field type adds 3 bits to give 7 bits therefore this is encoded as a byte in varint encoding.
  • The key for a structure with 16-2047 fields whose field number/TAG is in range 16-2047 is represented as 11 bits.The field type adds 3 bits to give 14 bits therefore this is encoded as two bytes in varint encoding.
  • The largest number of fields than can be sent is 229-1 which gives 536,870,911 fields. The key is encoded as 5 bytes.

You can’t have structures sent over with fields numbers in the range 19000-19999 as they are reserved.

Performing variable length encoding of fields

The fields have a higher probability/explicitely required can be given ranges in the 0-15 range, this will give the key as a byte long  thus optimising bandwidth and ensuring effecient coding. of the field packet. Due to this, you will notice that messages in the proto files have the following formats with the required fields leading in the specification within that file. Example:

Be sure to have a read through the specification especially the techniques section that defines how good a fit protobufs is for you.