IPIP-0337: Delegated Content Routing HTTP API

Gus Eggert GitHub
Marcin Rataj GitHub
Related Issue
Commit History
GitHub ipfs/specs (pull requests, new issue, open issues)

1. Summary

This IPIP specifies an HTTP API for delegated content routing.

2. Motivation

Idiomatic and first-class HTTP support for delegated routing is an important requirement for large content routing providers, and supporting large content providers is a key strategy for driving down IPFS content routing latency. These providers must handle high volumes of traffic and support many users, so leveraging industry-standard tools and services such as HTTP load balancers, CDNs, reverse proxies, etc. is a requirement. To maximize compatibility with standard tools, IPFS needs an HTTP API specification that uses standard HTTP idioms and payload encoding. The Reframe spec for delegated content routing is an experimental attempt at this, but it has resulted in a very unidiomatic HTTP API which is difficult to implement and is incompatible with many existing tools. The cost of a proper redesign, implementation, and maintenance of Reframe and its implementation is too high relative to the urgency of having a delegated content routing HTTP API.

Note that this does not supplant nor deprecate Reframe. Ideally in the future, Reframe and its implementation would receive the resources needed to map the IDL to idiomatic HTTP, and implementations of this spec could then be rewritten in the IDL, maintaining backwards compatibility.

We expect this API to be extended beyond "content routing" in the future, so additional IPIPs may rename this to something more general such as "Delegated Routing HTTP API".

3. Detailed design

See the Delegated Content Routing HTTP API spec ([http-routing-v1]) included with this IPIP.

4. Design rationale

To understand the design rationale, it is important to consider the concrete Reframe limitations that we know about:

So this API proposal makes the following changes:

4.1 User benefit

The cost of building and operating content routing services will be much lower, as developers will be able to maximally reuse existing industry-standard tooling. Users will not need to learn a new RPC protocol and tooling to consume or expose the API. This will result in more content routing providers, each providing a better experience for users, driving down content routing latency across the IPFS network and increasing data availability.

4.2 Compatibility

4.2.1 Backwards Compatibility

IPFS Stewards will implement this API in go-delegated-routing, using breaking changes in a new minor version. Because the existing Reframe spec can't be safely used in JavaScript and we won't be investing time and resources into changing the wire format implemented in edelweiss to fix it, the experimental support for Reframe in Kubo will be deprecated in the next release and delegated content routing will subsequently use this HTTP API. We may decide to re-add Reframe support in the future once these issues have been resolved.-

4.2.2 Forwards Compatibility

Standard HTTP mechanisms for forward compatibility are used:

  • The API is versioned using a version number prefix in the path
  • The Accept and Content-Type headers are used for content type negotiation, allowing for backwards-compatible additions of new MIME types, hypothetically such as:
    • application/cbor for binary-encoded responses
    • application/x-ndjson for streamed responses
    • application/octet-stream if the content router can provide the content/block directly
  • New paths+methods can be introduced in a backwards-compatible way
  • Parameters can be added using either new query parameters or new fields in the request/response body.
  • Provider records are both opaque and versioned to allow evolution of schemas and semantics for the same transfer protocol

As a proof-of-concept, the tests for the initial implementation of this HTTP API were successfully tested with a libp2p transport using libp2p/go-libp2p-http, demonstrating viability for also using this API over libp2p.

4.3 Security

4.4 Alternatives

A. References

Routing V1 HTTP API. Gus Eggert; Masih H. Derkani; Henrique Dias. 2023-03-22. URL: https://specs.ipfs.tech/routing/http-routing-v1/
IPIP-0379: Delegated IPNS HTTP API. Masih H. Derkani; Marcin Rataj. 2023-02-13. URL: https://specs.ipfs.tech/ipips/ipip-0379/