IPIP-0499: UnixFS CID Profiles

2.1 UnixFS parameters

The following UnixFS parameters were identified as factors that affect the resulting CID:

1. CID version, e.g. CIDv0 or CIDv1
2. Multibase encoding for the CID, e.g. base32
3. Hash function used for all nodes in the DAG, e.g. sha2-256
4. UnixFS file chunking algorithm and chunk size (e.g., fixed-size chunks of 256KiB)
5. UnixFS DAG layout:
   • balanced: builds a balanced tree where all leaf nodes are at the same depth. Optimized for random access, seeking, and range requests within files (e.g., video).
   • balanced-packed: variant of balanced that may produce different tree structure for large files. See Balanced DAG layout variants below.
   • trickle: builds a tree optimized for on-the-fly one-time streaming, where data can be consumed before the entire file is available. Useful for logs and other append-only data structures where random access is not important.
6. UnixFS DAG width (max number of links per File node)
7. HAMTDirectory fanout: the branching factor at each level of the HAMT tree (e.g., 256 leaves).
8. HAMTDirectory threshold: max Directory size before converting to HAMTDirectory, based on PBNode.Links count or estimated serialized dag-pb size. See Historical inconsistency in HAMT sharding below.
   • links-count: PBNode.Links length (child count). Simple but ignores varying entry sizes.
   • links-bytes: sum of PBNode.Links[].Name and PBNode.Links[].Hash byte lengths. Underestimates actual size by ignoring UnixFS Data, Tsize, and protobuf overhead.
   • block-bytes: full serialized dag-pb node size. Most accurate, accounts for varint Tsize and optional metadata such as mode or mtime.
9. Leaves: either dag-pb wrapped or raw
10. Whether empty directories are included in the DAG. Some implementations may apply filtering.
11. Whether hidden entities (including dot files) are included in the DAG. Some implementations may apply filtering.
12. Directory wrapping for single files: in order to retain the name of a single file, some implementations have the option to wrap the file in a Directory with link to the file.
13. Presence and accurate setting of Tsize (correct UnixFS has Tsize of child sub-DAGs).
14. Symlink handling: preserved as UnixFS Type=4 nodes, or followed (dereferenced to target).
15. Mode: optional POSIX file permissions.
16. Mtime: optional modification timestamp.

2.2 Balanced DAG layout variants

The balanced DAG layout has implementation variants that affect CID determinism for large files. CID mismatches have been observed and investigated when comparing kubo and Singularity outputs for files exceeding 1 GiB. This IPIP introduces the name balanced-packed to distinguish Singularity's variant from the original balanced layout.

Implementations adopting a profile SHOULD specify which balanced variant they use. The unixfs-v1-2025 profile uses balanced for maximum compatibility with existing implementations.

2.3 Historical inconsistency in HAMT sharding

The IPFS ecosystem was never fully consistent in HAMT directory sharding behavior. This section documents the implementation history to explain why standardization through profiles is necessary.

Timeline of Go implementation changes:

• 2017-03: kubo#3042 introduced HAMT sharding with a global Experimental.ShardingEnabled flag. When enabled, all directories were sharded regardless of size. This is why historical snapshots like /ipfs/bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze (Wikipedia) have HAMTDirectory nodes even for parent directories with few entries.

• 2021-05: go-unixfs#91 introduced HAMTShardingSize threshold for automatic sharding based on estimated directory size, using links-bytes estimation. This was part of the work tracked in kubo#8106.

• 2021-11: go-unixfs#94 added size-based unsharding (switching from HAMT back to basic directory), completing bidirectional automatic sharding with >= comparison. The go-unixfs repository has since been archived; its code now lives in boxo.

• 2021-12: go-ipfs v0.11.0 (now Kubo) shipped with automatic HAMT autosharding, deprecating the global Experimental.ShardingEnabled flag.

• 2023-03: boxo created via Über Migration, inheriting the >= comparison behavior from go-unixfs.

• 2026-01: boxo#1088 fixed threshold comparison from >= to >, aligning with JS implementation and documentation. Shipped in Kubo 0.40.

Timeline of JavaScript implementation changes:

• 2017-03: js-ipfs-unixfs#14 added HAMT data types to UnixFS protobuf definitions.

• 2018-12: js-ipfs#1734 added HAMT sharding support to MFS, using entry count threshold (shardSplitThreshold, default 1000 entries).

• 2019-01: js-ipfs v0.34.0 shipped with HAMT support in MFS. The threshold was based on entry count, not size, which differed from Go's size-based approach.

• 2022-10: Helia created as the successor to js-ipfs.

• 2023-02: js-ipfs-unixfs#171 changed from entry count to DAGNode size threshold (shardSplitThresholdBytes, default 256 KiB), aligning with Go implementation. Uses > comparison. This was tracked in js-ipfs-unixfs#149. The js-ipfs-unixfs library remains active and is used by Helia.

• 2023-05: js-ipfs archived; Helia became the recommended JS implementation.

The JavaScript implementation in Helia uses size > threshold (strictly greater than) in is-over-shard-threshold.ts, consistent with Go after the 2026 fix.

These inconsistencies between Go and JS implementations over the years, combined with differing threshold methods (entry count vs size) and comparison operators (>= vs >), meant cross-implementation CID determinism for large directories was never reliably achievable. The unixfs-v1-2025 profile addresses this by standardizing on block-bytes estimation and explicit > comparison.

2.4 Divergences across ecosystem

We analyzed the default settings across the most popular UnixFS implementations in the ecosystem. The table below documents the divergences that prevent deterministic CID generation today:

Terminology:

• included: Always included in the DAG (no option to exclude)
• excluded: Always excluded from the DAG (no option to include)
• opt-in: Excluded by default; implementations provide a flag to include (e.g., --hidden in Kubo/Storacha, hidden: true in Helia)
• opt-out: Included by default; implementations provide a flag to exclude
• preserved: Symlinks stored as UnixFS Type=4 nodes with target path (per UnixFS spec). Note: Kubo (v0.39) --dereference-args only follows symlinks passed as CLI arguments; symlinks found during recursive traversal are always preserved.
• followed: Symlinks dereferenced and treated as target files/directories
• skipped: Symlinks ignored during traversal (not included in DAG)
• (rclone): Singularity delegates file traversal to rclone; values shown reflect rclone defaults
• (boxo): Singularity overrides some boxo defaults but relies on implicit boxo defaults for these values
• ?: Service did not provide implementation details when queried in ipfs/specs#499

3.1 The unixfs-v1-2025 modern profile

Based on the research above, we define unixfs-v1-2025 as an opinionated profile for implementations that want to adopt deterministic CID generation for UnixFS DAGs with CIDv1.

3.2 The unixfs-v0-2015 legacy profile

This profile documents the default UnixFS DAG construction parameters used by Kubo through version 0.39 when producing CIDv0. It is provided for users who depend on CIDv0 identifiers generated by Kubo and need to reproduce them with other implementations, or verify content against existing CIDv0 references. The year 2015 in the name indicates that the majority of these parameters were picked a decade ago, when the initial go-ipfs alpha software was implemented, and these defaults were never contested since then.

Note: this profile is a best-effort approximation of historical behavior. It produces deterministic CIDs for files and smaller directories. However, as documented in Historical inconsistency in HAMT sharding, there is a risk of divergence when directories exceed the HAMT sharding threshold, due to differences in threshold comparison operators and estimation methods across software versions. In such cases, the only recourse is to identify which version of software originally created the content and manually adjust import parameters to match those historic settings.

4.1 User benefit

Profiles provide key advantages for working with content-addressed data:

1. Predictable, deterministic behavior: Profiles restore intuitive hash-like behavior: identical input data always produces identical CIDs, regardless of which implementation generates them.

2. Lightweight verification: Users can verify content without needing to rely on additional merkle proofs or CAR files.

3. Simplified workflow: Users can select a profile and automatically get consistent CIDs across all implementations, without needing to configure or understand the underlying parameters.

4. Improved efficiency: The unixfs-v1-2025 profile uses 1 MiB chunks with 1024 links per node, compared to the legacy 256 KiB chunks with 174 links. This results in:

   • Shallower DAG trees (3 levels for a 1 TiB file vs 4 levels with legacy parameters)
   • Approximately 4x fewer total nodes for the same content
   • Faster random access and seeking in large files (fewer round-trips to traverse the tree)
   • Fewer CIDs to announce, reducing stress on public good routing infrastructure such as the Amino DHT

4.2 Compatibility

UnixFS data encoded with the CID profiles defined in this IPIP remains fully compatible with existing implementations, since it conforms to the UnixFS specification.

To generate CIDs in compliance with this IPIP, implementations MUST support the unixfs-v1-2025 profile. The unixfs-v0-2015 profile is provided for backward compatibility and MAY be supported by implementations that need to produce CIDs matching historical Kubo output.

Implementations SHOULD allow users to inspect default values and adjust configuration options related to CID generation.

4.3 Alternatives

As an alternative to profiles, users can store and transfer CAR files of UnixFS content, which include the merkle DAG nodes needed to verify the CID.

5.1 Verification process

1. Fetch the reference DAG by CID from the IPFS network or a gateway
2. Export the DAG content to the local filesystem (e.g., ipfs get <CID>)
3. Re-import using the implementation under test with the specified profile
4. Confirm the resulting CID matches the reference

5.2 unixfs-v0-2015 profile

5.3 unixfs-v1-2025 profile

5.4 HAMT threshold behavior

The HAMT threshold comparison uses > (strictly greater than), not >=. A directory with estimated size exactly equal to the threshold (262144 bytes) remains a basic directory. Only when the size exceeds the threshold does it convert to HAMT.

5.5 DAG layout

Both profiles use the balanced layout where distance from root to each leaf is the same number of hops.

balanced:                              trickle:

         root                                root
        /    \                         /   /  |  \   \
     node     node                  leaf node leaf node ...
    / | \     / | \                      |         |
 leaf leaf leaf leaf ...              node       node
                                      / \        / \
                                   leaf leaf  leaf leaf
 ^^^^^^^^^^^^^^^^^^^^^^
    uniform depth                    ^^^^     ^^^^^^
                                     varying leaf depth

Implementations MAY use below CIDs to verify DAG layout handling. The trickle layout is not part of official profiles, but is included for verifying that UnixFS readers can parse non-standard DAG layouts produced by other software in the ecosystem:

5.6 Additional test vectors

For the following cases, see existing test vectors in the UnixFS spec:

• Empty directory: well-known CIDs QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn (v0) and bafybeiczsscdsbs7ffqz55asqdf3smv6klcw3gofszvwlyarci47bgf354 (v1)
• Preserved symlink: QmWvY6FaqFMS89YAQ9NAPjVP4WZKA1qbHbicc9HeSKQTgt (v0)

5.7 Copyright

Copyright and related rights waived via CC0.