Why the next release breaks old files
When a file is written by serializing your structs, the file format is your structs, and every release changes them. With a positional binary format such as bincode or postcard, any added, removed or reordered field makes old files unreadable. MessagePack and CBOR can carry field names, but the Serde libraries do not always use them; rmp-serde writes structs as arrays by default, which breaks in exactly the same way.
The usual escapes are keeping every struct frozen and adding
ConfigV2, ConfigV3 and migration code for each
step, or moving to JSON and accepting larger files and a flattened data
model. Both grow with every release, and neither helps when a user opens a
file from a version newer than the installed one.
Evolving the format instead of migrating files
Postbag's Full format writes each field with its identifier
and the length of its value, so a reader matches fields by identifier
instead of position. A field added in version 2 is skipped by version 1 and
takes its #[serde(default)] when version 2 reads an old file.
Removed fields, reordered fields and new enum variants work likewise, so
the files stay backward and forward compatible. For these changes no version field, no migration ladder
and no frozen struct copies are needed; you just change your types.
Furthermore, a field can be marked recoverable. If it changed incompatibly, reading the file does not fail; the rest of the document is loaded and that field takes its default. For a settings file this means one setting resets instead of the user losing the whole configuration.
Evolution and recovery in practice
Version 2 of this config added font_size; files load in both
directions, with version 1 skipping the unknown field. The key bindings are
additionally marked recoverable, which is unique to Postbag: even a change
to them that no rule covers does not lose the file.
use serde::{Deserialize, Serialize};
#[derive(Serialize, Deserialize)]
struct Config {
theme: String,
// Added in version 2. Files from
// version 1 use the default.
#[serde(default)]
font_size: u32,
}
let config: Config =
postbag::from_full_slice(&bytes)?;#[derive(Serialize, Deserialize)]
struct Config {
theme: String,
#[serde(with = "postbag::recoverable")]
bindings: KeyBindings,
}
// If KeyBindings changed incompatibly,
// theme is still loaded and bindings
// becomes KeyBindings::default().
let config: Config =
postbag::from_full_slice(&bytes)?;Questions
My format is MessagePack or CBOR. Is that not already self-describing?
The format is, but the way Serde uses it may not be. rmp-serde, for example, writes structs positionally as arrays by default, so adding or removing a field breaks old files although MessagePack could express field names. In named mode field changes work, but every value carries its full field name, renames break without aliases, and a field whose type changed still fails the whole file.
Do I need a version number in my files?
Not for the covered changes; old and new types read each other without version negotiation. Starting the file with a few magic bytes that identify your application and format is still good practice, and independent of Postbag.
Can an old release of my application read files written by a new one?
Yes, this is the forward direction of compatibility. Fields the old release does not know are skipped, and enum variants it does not know are caught by a variant marked with #[serde(other)].
What if a field's type changes completely?
Wrap the field with #[serde(with = "postbag::recoverable")]. If it cannot be deserialized, the rest of the file is still read and the field takes its default value, so the user loses one setting instead of the whole document.
How large are the files?
Postbag uses variable-length integer encoding and can number field identifiers so that each costs a single byte instead of its name. Where minimal size matters more than free evolution, the Slim format writes the values only.
Where to go next
Binary data that survives schema changes explains how the encoding achieves this, and the Postbag overview has the quick start and the complete compatibility rules. The same properties apply when the bytes travel between programs instead of into a file, as Remoc uses them.