Postbag

A binary Serde format that lets your types keep changing.

Postbag is a compact binary Serde codec with built-in support for schema evolution. It preserves Rust's data model, including distinctions such as Some(None) != None, and allows programs built with different versions of a type to exchange data as fields and enum variants are added, removed, renamed or reordered.

Documentation crates.io Release notes Source on GitHub
cargo add postbag

Two evolvable format variants

Postbag provides two wire formats with different trade-offs between schema flexibility and encoded size.

Postbag Full format with field names recommended

The Postbag Full format writes each field with its name and the length of its value. That lets fields be added, removed and reordered. Writing names as strings usually costs a lot of space but Postbag reduces it to a single byte when #[serde(rename = "_n")] is applied to a struct field or enum variant.

How Postbag encodes a struct with identifiers A Rust struct with three fields on the left, and on the right the fourteen bytes Postbag writes for it in its Full format, in three columns: the identifier of each field, the length of its value in bytes, and the value. Every field is numbered, so each identifier takes a single byte. Your types 14 bytes 03 three fields follow identifier length value #[derive(Serialize, Deserialize)] struct Reading { sensor: u32, #[serde(rename = "_0")] = 300 41 _0 02 ac 02 300 as a varint label: String, #[serde(rename = "_1")] = "temp" 42 _1 04 74 65 6d 70 "temp" unit: Unit, #[serde(rename = "_2")] = Celsius 43 _2 01 41 variant _0 } enum Unit { #[serde(rename = "_0")] Celsius, #[serde(rename = "_1")] Other(String), }
Numbering a field is optional: _0 to _59 take a single byte, any other name is written out; keeping sensor as a name would cost seven bytes rather than one. Enum variants can be numbered the same way, and integers are varint-encoded, so a u32 of 300 just costs two bytes.

Postbag Slim format without field names minimal size

The Postbag Slim format writes the values and nothing else, in the order the fields are declared. Fields can still be added and removed at the end, but nothing says which field is which, so backwards compatible changes to structs and enums are limited.

How Postbag encodes the same struct without identifiers The same Rust struct on the left, and on the right the ten bytes Postbag writes for it in its Slim format: a field count, the length of the whole struct, and then the three values one after another. Nothing names the fields, so they can only be read back in the order they are declared. Your types 10 bytes 03 three fields 08 8 bytes of values value #[derive(Serialize, Deserialize)] struct Reading { sensor: u32, = 300 ac 02 300 as a varint label: String, = "temp" 04 4 bytes 74 65 6d 70 "temp" unit: Unit, = Celsius 00 variant number 0 } enum Unit { Celsius, Other(String), }
An enum variant is written as its number rather than its name, and a string still carries its own length, since nothing else states where it ends.

Use Full by default. Choose Slim only when both sides evolve together and minimum encoded size is more important than flexible schema evolution.

Backwards and forwards compatibility

Serializer and deserializer can work with different versions of your type, provided that the following rules are respected.

A field a reader expects but does not receive takes its #[serde(default)], and a variant it does not know needs a #[serde(other)] fallback.
Change to your types Postbag Fullrecommended Postbag Slimminimal size
Structs
Add a field anywhere at the end
Remove a field anywhere at the end
Rename a field when numbered always
Reorder fields yes no
Enums
Add a variant anywhere at the end
Remove a variant anywhere at the end
Rename a variant when numbered always
Reorder variants yes no
The struct above 14 bytes 10 bytes
When to use small size, best compatibility smaller size, less compatibility

Recover from incompatible values

Sometimes a type has to change incompatibly. With the Full format, Postbag can confine a deserialization failure to a marked field, recover the rest of the enclosing value, and replace that field with its default.

Make one field recoverable
use serde::{Deserialize, Serialize};

#[derive(Default, Deserialize, Serialize)]
struct Details {
    size: u32,
}

#[derive(Deserialize, Serialize)]
struct Reading {
    sensor: String,
    #[serde(with = "postbag::recoverable")]
    details: Details,
    count: u16,
}

let bytes = postbag::to_full_vec(&reading)?;
let restored: Reading = postbag::from_full_slice(&bytes)?;

// If Details changed incompatibly, sensor and count are still
// restored, while details becomes Details::default().

Recoverable values can be added to an existing field without changing its wire representation.

Format specifications

The complete version 1.0 wire formats are specified independently of the Rust API: