Datagram libp2p API

[stormshield-pj50]

Hi lib2p folks,

At Stormshield we have built a tunneling vpn using libp2p. We start several instances of libp2p, basically one instance for each tunneled network and we manage our own custom virtual transport/muxer that delegates swarm dial and listen requests to our front component. Our front handles real transport connections using Quinn directly (the reason why we use this architecture is not the point here).

So you'd guess it, our goal is to tunnel packets using libp2p facilities and we have implemented our own libp2p protocol for this. But doing it using Substream's traits AsyncRead/AsyncWrite is not easy for two reasons:

• AsyncRead/AsyndWrite provides a streaming API but we want to send packets. Moreover our front code relies on Quinn datagram support to send tunneled packets so it makes little sense to use the AsyncRead/AsyncWrite API.
• Using AsyncRead/AsyncWrite with our architecture involves multiple packets copies and thus achieves poor performance.

For those reasons we have patched libp2p and added two new traits DgramRead and DgramWrite to libp2p Substream. Substreams returned by our virtual muxer creates structs that both implement AsyncRead/AsyncWrite and DgramRead/DgramWrite so libp2p swarm can negotiate protocols using the AsyncRead/AsyncWrite API, and our own libp2p protocol can uses the new API in order to tunnel packets. Regarding performance our code allows to use both the two APIs to tunnel packets, and we get throughput multiplied by 280 when using the DgramRead/DgramWrite API with a basic iperf3 test. Is is important to note that the DgramRead/DgramWrite API use the crate Bytes, allowing a zero copy packet pathway.

Now my question is: have you some plans or have you ever thought providing such an API in libp2p ?

Thanks for your response,
Philippe

[mxinden]

Thanks for the write-up. Will reply more in depth. In the meantime:

When libp2p calls Dgramwrite::write, how does that translate to a datagram? Does that send a plain UDP packet? Is that packet encrypted / authenticated? Is there any flow-control in place? Are you using QUIC's datagram extension?

[stormshield-pj50]

Yes our front component send tunneled packets using QUIC's datagram extension, so thanks to QUIC it is encrypted and authenticated. Our virtual muxer substreams use mpsc bounded channels between a swarm instance and our front component so packets are flow-controlled this way. I believe datagrams are also flow-controlled by Quinn before being sent on the socket.
If you want more details maybe we could setup a call to talk about it.

[mxinden]

If you want more details maybe we could setup a call to talk about it.

Sounds good. Want to propose a time, or join either the next libp2p community call, or the next rust-libp2p maintainers call (next week Tuesday, 09:00 UTC)?

[stormshield-pj50]

I sent you a message on your github profile email address.

[thomaseizinger]

How does DgramWrite::write behave if the Swarm is configured with a transport configuration like TCP + noise + yamux?

[stormshield-pj50]

Currently we do not handle the Dgram API with yamux, we only use our virtual muxer that creates our own virtual substreams which implement the Dgram API. But datagrams could be written in a yamux substream using DgramWrite:write.

[thomaseizinger]

I am not sure if I understand correctly. Yamux requires an ordered and reliable stream. Are you suggesting to send datagrams as messages on top of this stream?

[stormshield-pj50]

Yes that's the idea.

[thomaseizinger]

@mxinden filled me in on this topic in an out-of-band discussion. As far as I understand, you "just" want to reuse the QUIC endpoint to send your own datagrams.

Instead of tunneling that through (multiple) libp2p abstractions, would it be feasible to redesign our QUIC transport such that the "Endpoint" can be shared with other parts of the application?

The design would roughly be that the protocols you run through libp2p end up emitting events you can capture from the Swarm at which point you instruct other parts of the application to send datagrams through the QUIC endpoint instead of your protocols starting to send datagrams through a libp2p stream.

[stormshield-pj50]

That's an interesting idea, however in our usecase one libp2p connection can be "plugged" to our front component through several "real" transport connections (QUIC and/or TCP/TLS). So we end up managing several endpoints. However if the "Endpoint" is generic enough I guess we could dispatch the "send datagrams commands" to the right transport.

[thomaseizinger]

The "Endpoint" is - in our case - the quinn concept. It combines a UDP socket with the necessary encryption and multiplexing capabilities.

What do you mean with generic enough? It would provide you with everything that QUIC has to offer.

[thomaseizinger]

UDP is connection-less so in theory, a QUIC endpoint doesn't need to have a single owner but could happily be used by multiple parts of the application concurrently.