Add an opt-in Data-first field ordering for PBNode protobuf messages, so
that streaming readers can process UnixFS metadata before links, and
formalize that readers accept both orderings. No profile enables it: all
profiles keep the canonical Links-first ordering and their CIDs, and
writing Data-first requires an explicit opt-in setting.
Regular UnixFS and HAMT-sharded directories are encoded as PBNode protobuf
messages.
HAMT-sharded directory entries have the characteristic of prefixing the name of each entry with a number of characters drawn from the hash of the directory entry name.
When multiple entries land in the same bucket, that bucket is replaced by a sub-shard with its own CID/block that contains those entries.
The settings used to derive the prefix characters are stored in the Data
field of the PBNode protobuf message.
This means that all Link messages must be read from the PBNode message
before we can read the hashType (a multihash code) and fanout values that
let us calculate the prefix length for a given directory entry.
When the reader is attempting to traverse to a single entry deep in the shard, they are forced to read all entries for the current sub-shard before they can move deeper within the shard, which leads to inefficient traversals.
Independent of any efficiency gains, both orderings already exist in the wild: the DAG-PB codec specification requires decoders to accept either order "as IPFS data exists in both forms". Implementations comply, but by accident. Neither the UnixFS specification nor any test suite said so, and tolerance that is unspecified and untested is how interoperability decays: the code path works until the day it silently does not, and nothing catches it.
The primary goal of this IPIP is the read side: every implementation can
read UnixFS DAGs in either field order, and fixtures and conformance tests
prove it. Producing the non-canonical Data-first order is optional and
opt-in, but keeping a conforming writer around matters too: it is what keeps
the tolerant read path exercised, so reading and writing such DAGs both stay
interoperable instead of becoming an untested promise.
This is the robustness principle applied: "Be strict about the outcomes, be tolerant about the methods". The outcome is strict: the same logical UnixFS DAG, pinned by byte-exact fixtures and CIDs. The method, the field order on the wire, is tolerated in both forms, and, per the same principle's warning about silent tolerance, kept honest with tests rather than left to accident.
If content authors are allowed to write the Data field first, readers can
process PBNode messages with a streaming parser: the UnixFS metadata (for
HAMT shards, the hashType and fanout parameters) arrives before the links,
so a reader looking for a specific entry can stop reading links as soon as it
finds the one it needs.
PBNode field order parameterWriting the Data field first changes the CID generated for a piece of
content, so this is an opt-in change.
We add a new parameter to the set defined in IPIP-0499:
| Parameter | Values |
|---|---|
| PBNode field order | links-first (default) or data-first |
When a profile does not define PBNode field order, it is links-first.
This IPIP amends the IPIP-0499 profile tables so unixfs-v0-2015 and
unixfs-v1-2025 state links-first explicitly; they continue to produce the
same CIDs as today.
No profile sets data-first (see Alternatives for why).
Implementations MAY expose an explicit low-level opt-in setting for writers
that need it; enabling it changes the CID of every written node.
This IPIP amends the UnixFS specification:
PBNode field orderings: Links-first as the canonical
form, Data-first as the explicit opt-inLinks-first unless the user explicitly opted into
Data-first; specialized implementations MAY support a single orderingIt also amends the profile tables in
IPIP-0499 with an explicit
PBNode field order: links-first row, so the preexisting profiles are
protected from unintentional change.
Traversing HAMT shards is more expensive than it needs to be, which disproportionately affects resource-constrained environments and inefficient runtimes.
Working code: opt-in data-first writing merged in boxo
(ipfs/boxo#1212) and Kubo
(ipfs/kubo#11439), and shipped in
@ipld/dag-pb v4.2.0
(ipld/js-dag-pb#111);
cross-ordering read tests and CAR fixtures shipped in
gateway-conformance v0.14.1.
Traversing HAMT shards will become faster in resource-constrained environments and inefficient runtimes.
Existing content and workflows are unaffected: the canonical ordering remains the default everywhere, and an explicit opt-in is available for new developments that want the more efficient streaming reads.
PBNode wire ordering is governed by the
DAG-PB codec specification,
which requires decoders to accept both field orders ("as IPFS data exists in
both forms") and, since ipld/ipld#383,
explicitly permits encoders to write either order. Deployed implementations
(go-codec-dagpb, js-dag-pb) already read both orders, so Data-first blocks
are readable by existing software, and the two specifications agree. Should
they diverge again, the UnixFS specification
governs UnixFS data.
Backward compatibility is preserved by keeping the new ordering opt-in. Writers
emit it only when a user explicitly enables the setting; every profile and
default keeps the canonical ordering, so already-published CIDs and the CID
determinism guarantees of unixfs-v1-2025 are unchanged.
Two valid encodings for the same data are possible.
data-first block, or a
data-first writer re-encoding a canonical block.
unixfs-v1-2025
(IPIP-0499) guarantees.Links fields are
length-delimited, so a reader holding a complete block can skip them cheaply
and read Data at the tail. This helps whole-block parsing, but does not
help streaming parsers, which cannot skip ahead in data that has not arrived
yet.unixfs-v1-YYYY profile that enables data-first. Considered
and dropped: a dated successor to unixfs-v1-2025 reads as "the
recommended latest", inviting unintentional adoption and a de facto new
CIDv1 default, exactly the CID churn this IPIP avoids. Creating a new CID
profile should be a separate IPIP, and its review should take a long time:
a new dated profile effectively informs the new suggested default for
CIDv1, and this IPIP deliberately changes no defaults. The parameter
machinery stays, so such a future IPIP can define the profile if
multi-implementation demand for deterministic data-first CIDs appears.The table below lists minimal single-block vectors for both orderings. To
verify compliance, decode each block, confirm the CID, and confirm that
re-encoding the decoded node with the stated ordering reproduces the block
bytes. Readers MUST resolve hello.txt through all four directory roots.
| Description | Ordering | CID | Block (base16 encoded) |
|---|---|---|---|
hello.txt, file content "hello\n" |
raw leaf | bafkreicysg23kiwv34eg2d7qweipxwosdo2py4ldv42nbauguluen5v6am |
68656c6c6f0a |
Directory containing hello.txt |
Data-first (opt-in) |
bafybeigqvyloizmfcdy6scaxnyltftzptaruqa3hnnplfzsbf4sqteiwlm |
0a02080112330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e7478741806 |
Directory containing hello.txt |
Links-first (canonical) |
bafybeigdcg7pksx2zk5336vrfsktjodlr4rbfz37qr3koc5xboxe5ekv24 |
12330a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120968656c6c6f2e74787418060a020801 |
HAMTShard containing hello.txt |
Data-first (hand-crafted) |
bafybeicwgy2rlqmqqu3yy2tqvm2wbgdvy3snu4sbbv4wqpvpnoplpzxz74 |
0a250805121c80000000000000000000000000000000000000000000000000000000282230800212350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e7478741806 |
HAMTShard containing hello.txt |
Links-first (hand-crafted) |
bafybeicjwkfslu7gwyywffvqgse5kiibojtktxcdqhgv7ldj5fjdacuceq |
12350a24015512205891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03120b444668656c6c6f2e74787418060a250805121c800000000000000000000000000000000000000000000000000000002822308002 |
The HAMTShard blocks are hand-crafted for parser testing; no profile shards a
single-entry directory via import (sharding starts above the 256 KiB
threshold). The entry name hello.txt hashed with murmur3-x64-64 (hashType
0x22) yields 0xDF as the first byte, selecting bucket 223 at fanout 256:
the link name is DF + hello.txt, and bit 223 is set in the
decode(PBNode.Data).Data bitfield.
All five blocks ship as
pbnode-field-orders.car in gateway-conformance v0.14.1,
whose conformance tests exercise both orderings.
Copyright and related rights waived via CC0.
We gratefully acknowledge the following individuals for their valuable contributions, ranging from minor suggestions to major insights, which have shaped and improved this specification.
