confusion around bundles and buffer ids

[romange]

I am integrating bundles into helio/dragonfly but I am having trouble to understand how bundles are extended.
Based on io_uring_buf_ring_add docs:

liburing  (or  the  kernel, for that matter) doesn't care about what buffer ID maps to what buffer, and in fact when recycling buffers after use, the application is free to add a different buffer into the same buffer ID loca‐
       tion. All that matters is that the application knows what a given buffer ID in time corresponds to in terms of virtual memory. There's no liburing or kernel assumption that these mappings are persistent over  time,  they  can
       very well be different every time a given buffer ID is added to the provided buffer ring.

so based on my understanding, I could add non-contiguous ids (i.e. 2, 5, 7, 18) to the ring. Moreover, their order can be arbitrary. In fact it will be arbitrary when I reinsert them on completion. Now given all this, how can we advance a buffer id when processing a bundle CQE?

the docs say:

The application must iterate from the indicated initial buffer ID and until all
.I res
bytes have been seen to know which is the last buffer in the receive operation.
The buffer IDs consumed will be contiguous from the starting ID, in the order
in which they were added to the buffer ring used. 

1. Will they always be contiguous? What if only non contiguous ids are available?
2. What if the buffer id order in the ring is 5, 4, 3, 2, 1?

[axboe]

They are contiguous in the sense that they are consumed contiguously from the buffer ring. So if you add buffer IDs 0, 1, 2, 3 - then that's the order in which they are returned to you. Same if you add 5, 4, 3, 2, 1, then you'd get buffer ID 5 set and if it's a bundle, then 4, 3, 2, 1 may be the next ones.

If they are distontig in your buffer ring, then you could always just iterate the buffer ring on completions to see which ones completed.

[romange]

For now, I am adding an array that manages "next" relationships between bids on the userspace side during the buffer insertion to buf_ring. i.e. next[bid] map. I did not find a better way but please let me know if you have better ideas 🙏🏼

[axboe]

You don't need to fetch the head, the tail is local to the application and is where you last consumed. As the completions are posted in order wrt consuming buffers, that should match where the next receive would start.

[romange]

What do you mean by "consumed"? We use tail to add a buffer to the bufring, so yes I can store the mapping between the buffer id and its index in the ring but then it seems it's equivalent to just storing the order of buffer ids with next[bid] and use less indirections.

[axboe]

Sorry I'm rambling, I mean previous head. My point is that this level of indirection already exists, it's your buffer ring itself, just indexed by a cached head. You can obviously have a separate table that's indexed by bid, I guess that makes sense too.

[romange]

Is guaranteed that completions with IORING_CQE_F_BUFFER bit set are ordered in the same order as the provided buffers in the bufring?

[romange]

I mean if bufring has buffers with buf ids 5, 6, 7 in that order, then is it certainly that I will read CQE with buf id 5 before I see 6 or 7? (but 6 might be missing if 5 is a bundle, for example)

[romange]

@axboe Jens, it took me some time but I understood what you meant. I've implemented iterating over buffer ids using cached head and io_uring_buf indirections. Thank you for the help!

[axboe]

Good to hear! If you're up for it, would be great if you could do a man page addition with some details on this. If you got puzzled by it, then I'm assuming that others will be as well. And hence would be nice to improve the documentation for this case.