IPIP-0524: Remove cross-codec conversion from HTTP Gateways

Related Issue
ipfs/gateway-conformance/issues/200
History
Commit History
Feedback
GitHub ipfs/specs (inspect source, open issue)

1. Summary

Make IPFS HTTP Gateway responses easier to reason about by not requiring IPLD Data Model translations

2. Motivation

When sending an Accept header or format query parameter to specify the response format of a request, the IPFS HTTP Gateway specs allow translation of the requested content into the IPLD Data Model.

This adds significant complexity to HTTP Gateway implementations, since they need to be able to translate between arbitrary data types and handle all the various failure states.

The conversions are also lossy due to differences in supported data types across different formats so lack general-purpose utility and are ultimately something that could be done on an interested client if required.

3. Detailed design

When the block's CID codec matches the requested response format, implementations MAY return the block as-is without parsing or validating it. This is effectively equivalent to requesting ?format=raw but with a codec-specific Content-Type header.

When the CID codec does not match the requested format, the gateway SHOULD return a 406 Not Acceptable unless the server provides cross-codec conversion as an extra feature outside of this specification.

For example, requesting a DAG-JSON block with the application/cbor format would result in a 406 response.

Where a human-readable rendering of the data is desired, the text/html format can be requested. This would allow browsing DAG-PB data, for example.

A 400 may be returned if the request was invalid (for example an unsupported format was requested).

A 500 may be returned in other circumstances.

4. Design rationale

Simplifying the HTTP Gateway spec to remove these format translations and the additional logic required makes it more straightforward to create new implementations, and makes the returned data more transparent and so easier to understand since the data is not modified to fit the output format.

Clients that wish to translate between different data formats may request raw blocks and do the translation themselves.

4.1 User benefit

For gateway operators and implementers, removing the requirement to perform codec conversions server-side significantly reduces implementation complexity.

For end users and application developers, the change makes gateway behavior easier to reason about: a request either returns data deserialized according to the rules of the CID's original codec, or fails with 406. This moves conversion to userland, encouraging users to fetch raw blocks with ?format=raw and convert client-side, putting the application in full control and producing deterministic results regardless of which gateway is used.

This matters in practice because codec libraries do not behave identically. Cross-library dag-cbor tests (2026) show each implementation differs on edge cases like float handling, map key ordering, and encoding strictness. Relying on server-side conversion means the output depends on whichever library the gateway happens to use, which is not a foundation for robust software.

4.2 Compatibility

Formally this is a breaking change: server-side IPLD Data Model translations between codecs are removed.

In practice, nobody could build reliable software on top of conversion logic that behaved non-deterministically across gateways written in different languages. Clients that needed data in a different format often chose to fetch ?format=raw and convert client-side already.

This IPIP standardizes that robust real-world pattern and removes an unreliable niche feature that has seen limited use.

4.2.2 json and dag-json independence

application/json and application/vnd.ipld.dag-json are now treated as independent formats, each matching only their respective CID codec (json 0x0200 and dag-json 0x0129). The old behavior where application/json was an alias for application/vnd.ipld.dag-json (falling back to dag-json conversion) no longer applies.

4.2.3 UnixFS interop exception for Accept: application/json

Note: the codec match requirement and 406 behavior described above do not apply to deserialized UnixFS file responses. Users commonly store valid JSON as UnixFS files (with dag-pb or raw codec), and serving those files with Accept: application/json is regular HTTP content serving, not codec conversion. See the application/json entry in the Accept request header section of the Path Gateway spec for normative requirements.

4.2.4 Opt-in backward compatibility

Implementations MAY offer an opt-in configuration flag to restore the old codec conversion behavior for backward compatibility.

4.2.5 Implementation-defined behavior

  • The content of the 406 error response body (e.g. actionable hints).
  • Handling of ?format=json / Accept: application/json on non-json-codec content (like dag-pb UnixFS files).
  • Whether to offer an opt-in flag for restoring codec conversion.
  • Validation of block data when the CID codec matches the requested format.

4.3 Security

No security implications. This change restricts gateway behavior (returning 406 instead of converting), which reduces attack surface.

5. Test fixtures

Implementers can run the gateway-conformance test suite v0.10 or later. The following behaviors are verified by the test suite:

A. References

[rfc2119]
Key words for use in RFCs to Indicate Requirement Levels. S. Bradner. IETF. March 1997. Best Current Practice. URL: https://www.rfc-editor.org/rfc/rfc2119

B. Acknowledgments

We gratefully acknowledge the following individuals for their valuable contributions, ranging from minor suggestions to major insights, which have shaped and improved this specification.

Editors
Alex Potsides (Shipyard) GitHub
Marcin Rataj (Shipyard) GitHub