IPIP-0410: Streaming NDJSON in Routing HTTP API

Henrique Dias GitHub
Related Issues
Commit History
GitHub ipfs/specs (pull requests, new issue, open issues)

1. Summary

Introduce backwards-compatible streaming support to the Routing V1 HTTP API. For this, we use the Accept HTTP header ([rfc9110]) for content type negotiation, as well as the Newline Delimited JSON (NDJSON) format.

2. Motivation

The main motivation for this change is to allow servers to respond faster to the client with provider records, as soon as they are available. In the current state, the client requests a list of providers for a CID from the server. Then, the client has to wait for the server to collect their final list of providers. After that, the server can respond with the full list of providers.

This is a big source of latency when /routing/v1 is used for delegating DHT lookups, where the client is forced to wait for the server to finish DHT walk.

With streaming support, the server is able to respond with provider records as soon as they are available. This reduces latency and allows for faster content discovery.

In addition, streaming responses may produce an unlimited amount of results, which is not the case for non-streamed responses.

3. Detailed Design

In summary, streaming is supported by using the Accept HTTP header, which is used for content type negotiation as described in [rfc9110]. The client sends an Accept HTTP header starting with application/x-ndjson, which is the content type for NDJSON. The following happens:

For more details regarding the design, check [http-routing-v1].

4. Design Rationale

This feature is designed such that it does not break compatibility with existing clients and servers. The Accept HTTP header is OPTIONAL. By default, the server MUST respond with application/json unless the client explicitly asked for application/x-ndjson. If the server does not support NDJSON, it is allowed to still respond with non-streamed JSON.

4.1 User Benefit

Users (clients) will benefit from this change as the servers will now be able to respond more promptly to provider record requests. Instead of waiting for the whole list to be constructed, servers can now return each provider record one by one, in a streaming fashion.

The client will be able to close connection at any time, reducing load on both ends.

The main use cases for this IPIP are light clients and services which want to delegate DHT lookups to external service. With streaming, clients will be able to receive results as soon the delegated service learns about new record, which directly impacts the content load speeds perceived by the end user.

4.2 Compatibility

The introduced changes are backwards-compatible. The introduced header is completely optional, and a server that does not support streaming is able to respond with a non-streaming response to the client. Equally, non-streaming responses are the default. Therefore, a client that does not support streaming will not receive a streamed response.

4.3 Security

Security considerations are equivalent as the ones in [ipip-0337].

A. References

Delegated Routing V1 HTTP API. Gus Eggert; Masih H. Derkani; Henrique Dias; Marcin Rataj. 2023-08-31. URL: https://specs.ipfs.tech/routing/http-routing-v1/
IPIP-0337: Delegated Content Routing HTTP API. Gus Eggert; Marcin Rataj. 2022-10-18. URL: https://specs.ipfs.tech/ipips/ipip-0337/
Key words for use in RFCs to Indicate Requirement Levels. S. Bradner. IETF. March 1997. Best Current Practice. URL: https://www.rfc-editor.org/rfc/rfc2119
HTTP Semantics. R. Fielding, Ed.; M. Nottingham, Ed.; J. Reschke, Ed.. IETF. June 2022. Internet Standard. URL: https://httpwg.org/specs/rfc9110.html