IPIP-0484: Opt-in Filtering in Routing V1 HTTP API

2. Motivation

IPFS aims to allow ubiquitous data exchange across different runtimes and platforms. One of the most challenging aspects of this goal is the diversity of network conditions and capabilities across different environments. Web browsers have a very limited network stack, and most web browsers do not support the full range of network transport protocols that are commonly used in other IPFS implementations.

The Delegated Routing v1 API empowers resource constrained clients like web browsers by significantly reducing the number of network connections necessary to fetch content addressed data directly from provider peers.

However, there are many cases where most of the results from the Delegated Routing v1 API are not actionable by clients, because the client does not support either the network transport protocol or the transfer protocol of the provider.

For instance, web browsers are limited to a specific set of network transport protocols, namely HTTPS, Secure WebSockets, WebTransport (emerging), and WebRTC. Consequently, providing information about providers that exclusively support TCP and/or UDP is not beneficial for browser-based clients, as they are unable to utilize such connections.

Moreover, Helia, the most actively maintained browser IPFS implementation, supports block retrieval by CID with Bitswap and Trustless Gateways, but does not support Graphsync.

This means that returning providers that only support raw TCP, raw UDP/QUIC, or Graphsync from the Delegated Routing API is not useful for browser clients, and results in unnecessary network traffic for browser clients.

4. Detailed design

4.1 Network Address Filtering

The proposed change is to add a ?filter-addrs parameter to the GET /routing/v1/providers/{cid} and GET /routing/v1/peers/{peer-id} endpoints of [http-routing-v1]:

Add a ?filter-addrs=<comma-separated-list> optional parameter to GET /routing/v1/providers/{CID} that indicates which network transports to return by filtering the multiaddrs in the Addrs field of the Peer schema.
The value of the filter-addrs parameter is a comma-separated list of network transport protocol name strings as defined in the multiaddr protocol registry, e.g. ?filter-addrs=webtransport.
unknown can be be passed to include providers whose multiaddrs are unknown, e.g. ?filter-addrs=unknown. This allows filtering providers whose multiaddrs are unknown at the time of filtering.
Multiaddrs are filtered by checking if the protocol name appears in any of the multiaddrs (logical OR).
Negative filtering is done by prefixing the protocol name with !, e.g. to skip IPv6 and QUIC addrs: ?filter-addrs=!ip6,!quic-v1. Note that negative filtering is done by checking if the protocol name does not appear in any of the multiaddrs (logical AND).
If no parameter is passed, the default behavior is to return the original list of addresses unchanged.
If only negative filters are provided, addresses not passing any of the negative filters are included.
If positive filters are provided, only addresses passing at least one positive filter (and no negative filters) are included.
If both positive and negative filters are provided, the address must pass all negative filters and at least one positive filter to be included.
If there are no multiaddrs that match the passed transports, the provider is omitted from the response.
Filtering is case-insensitive.

4.2 IPFS Protocol Filtering

The proposed change is to add a ?filter-protocols parameter to the GET /routing/v1/providers/{cid} and GET /routing/v1/peers/{peer-id} endpoints of [http-routing-v1]:

Add a ?filter-protocols=<comma-separated-list> optional parameter to GET /routing/v1/providers/{CID} to filter providers based on the Protocol field of the Peer schema.
The filter-protocols parameter is a comma-separated list of transfer protocol names, e.g. ?filter-protocols=transport-bitswap.
Transfer protocols names should be treated as opaque strings and have a max length of 63 characters. A non-exhaustive list of transfer protocols are defined per convention in the multicodec registry.
Implementations MUST preserve all transfer protocol names when returning a positive result that matches one or more of them.
A special unknown name can be be passed to include providers whose transfer protocol list is empty (unknown), e.g. ?filter-protocols=unknown. This allows for including providers returned from the DHT that do not contain explicit transfer protocol information.
Providers are filtered by checking if the transfer protocol name appears in the Protocols array (logical OR).
If the provider doesn't match any of the passed transfer protocols, the provider is omitted from the response.
If a provider passes the filter, it is returned unchanged, i.e. the full set of protocols is returned including protocols that not included in the filter. (note that this is different from filter-addrs where only the multiaddrs that pass the filter are returned)
Filtering is case-insensitive.
If no parameter is passed, the default behavior is to not filter by transfer protocol.

Even though some of existing IPFS transfer protocol names start with transport, e.g. transport-bitswap, transport-graphsync-filecoinv1, and transport-ipfs-gateway-http, they should not to be confused with the network transport protocols used in peer addresses, which are filtered using the filter-addrs parameter.

5. Design rationale

Using these query parameters improves cache efficiency, as the response will be smaller and more specific to the client's needs.
Backward compatibility is maintained by not changing the default behavior of the API.
Use of protocol name rather than codes makes it easier for human debugging.
DHT providers currently do not contain any transfer protocol information. unknown can be passed to filter-protocols to include such providers.
Since provider records are independent of peer records, and it's pretty common to have provider records without up-to-date multiaddrs for that peer, unknown can be passed to filter-addrs to include such providers.
Combining transfer protocol and transport protocol filters is done by ANDing the results of the filters, e.g. ?filter-addrs=webtransport&filter-protocols=transport-bitswap will return providers that support bitswap and have a webtransport multiaddr.

5.1 User benefit

By filtering out providers that do not support the desired network transport protocol and/or transfer protocol, the client can reduce the traffic necessary in order to fetch the data.

Moreover, it makes it much easier to determine whether there are any browser-usable providers for a given CID, which is a common use case for clients.

5.2 Compatibility

This should not effect existing clients or servers.

The default behavior when ?filter-addrs and ?filter-protocols is not passed is left unspecified, this IPIP is limited to opt-in behavior.

IPIP-0484: Opt-in Filtering in Routing V1 HTTP API

1. Summary

2. Motivation

3. Note on terminology

4. Detailed design

4.1 Network Address Filtering

4.2 IPFS Protocol Filtering

5. Design rationale

5.1 User benefit

5.2 Compatibility

5.3 Copyright

A. References

B. Acknowledgments