How does IORING_CQE_F_BUF_MORE work?

[AomaYple]

Generally the presence of IORING_CQE_F_BUFFER means that the kernel has selected a buffer and returned the buffer to the application, and the application itself can choose when to give the buffer to the kernel.

For example, there is now a completion event with the IORING_CQE_F_BUFFER and IORING_CQE_F_BUF_MORE flags.

IORING_CQE_F_BUFFER indicates that the kernel selected a buffer and returned it to the application, but the application chooses not to give the buffer to the kernel at this point, choosing for itself when to return it.

IORING_CQE_F_BUF_MORE indicates that more data is coming in for this buffer.

So I have a question, at this point the kernel has already given the buffer to the application, how does the kernel write to the same buffer again on the next completion event

[axboe]

If IORING_CQE_F_BUF_MORE is set, the kernel has not returned the full buffer back to the application yet. More completions will be coming to this buffer. Let's say the buffer is 8k in size, and you get a completion with BUFFER|BUF_MORE set and cqe->res being 1024. This means that there's 1k of data available from the start of the buffer, buf->addr. The next receive point for this buffer will be buf->addr + 1024.

Now you get another 1024b completion for the same buffer, that data is available as specified above. The next receive point is now buf->addr + 2048, as that is how much has been received so far.

Also see the original kernel commit for incremental buffers, it explains it as well:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ae98dbf43d755b4e111fcd086e53939bef3e9a1a

[AomaYple]

If IORING_CQE_F_BUF_MORE is set, the kernel has not returned the full buffer back to the application yet. More completions will be coming to this buffer. Let's say the buffer is 8k in size, and you get a completion with BUFFER|BUF_MORE set and cqe->res being 1024. This means that there's 1k of data available from the start of the buffer, buf->addr. The next receive point for this buffer will be buf->addr + 1024.

Now you get another 1024b completion for the same buffer, that data is available as specified above. The next receive point is now buf->addr + 2048, as that is how much has been received so far.

Also see the original kernel commit for incremental buffers, it explains it as well:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ae98dbf43d755b4e111fcd086e53939bef3e9a1a

Allow me to ask another small question? For io_uring_buf_ring, is it better to share one io_uring_buf_ring for all sockets or, one io_uring_buf_ring for one socket? which is better?

[axboe]

You should use a buffer ring per size of buffers you need, one per socket would be pretty nuts and not scale at all.