io_uring_for_each_cqe +- io_uring_cq_advance

[jnorthrup]

dear @axboe or others

liburing/src/include/liburing.h

Line 192 in d84c29b

* Must be called after io_uring_for_each_cqe()

.. declares a behavior

however I am porting the liburing tests through an ffi (kotlin) and the occurrences I have run into really don't show rigor in these matters.

samples/io_poll.c for instance runs a single iteration and bails

this brings me to the next point of potential astonishing behavior, that io_uring_for_each_cqe is a lopsided loop macro.

while not impossible to tease out, this makes for a bit of extra care and feeding; on top of lacking background info or sufficient examples to know the value of preserving this or the alternatives.

any and all background you can provide would be greatly helpful! thanks

edit: clarified io_poll.c sample code in liburing

[isilence]

May need some extra comments but io_uring_for_each_cqe is not intended for waiting for CQEs but rather processing ones that are already there. And it's not possible to busy poll CQ for IOPOLL and anyway not recommended for non-IOPOLL.

One valid use case is to do io_uring_for_each_cqe after io_uring_wait_cqe_nr.

[jnorthrup]

im sorry I did not write iopoll.c in the samples, I was not remarking on polling or no-polling as behaviors but the occurrence in the specific named sample module that goes against the comment cited.

on the topic of coverage, an observation only, is that io_uring_smp_store_release is the only barrier.h entry represented in the samples and narrowly for cq->ktail

my inquiry is expressly to gain understanding in writing a port to an incompatible language and to recognize whatever can be put into the mental accounting column of pairings, and to understand better how to deal with the atomics and when to include C shims for kotlin ffi that lacks macro expansion. the cases found in the sample code are confined to the single occurence mentioned (along with read_barrier and write_barrier gcc asm which appears in early pre-liburing lord-of-the-io_urings code elsewhere)

[isilence]

im sorry I did not write iopoll.c in the samples, I was not remarking on polling or no-polling as behaviors but the occurrence in the specific named sample module that goes against the comment cited.

No worries. The comment is definitely heavily outdated, will be updated.

on the topic of coverage, an observation only, is that io_uring_smp_store_release is the only barrier.h entry represented in the samples and narrowly for cq->ktail

That's an internal header an no user should be required to use any of the helpers in there. It's most likely used in 1-2 places in tests just because it either wants to test some internals or was converted/half-converted from a non-liburing test case (e.g. syzbot reproducers).

my inquiry is expressly to gain understanding in writing a port to an incompatible language and to recognize whatever can be put into the mental accounting column of pairings, and to understand better how to deal with the atomics and when to include C shims for kotlin ffi that lacks macro expansion. the cases found in the sample code are confined to the single occurence mentioned (along with read_barrier and write_barrier gcc asm which appears in early pre-liburing lord-of-the-io_urings code elsewhere)

Talking about C, If you use liburing you don't need any extra sync userspace-kernel sync. There should be some when userspace threads/processes share a CQ or a SQ (e.g. submitting to a single ring from multiple threads). It depends, but it may be a good idea to keep them per-thread if possible.

There might be performance use cases for io_uring_for_each_cqe(), but it's not necessary, everything can be done with io_uring_wait_cqe(), io_uring_peek_cqe(), etc. It seems, io_uring_cq_advance() and io_uring_cqe_seen() are the only that may cause you a bit of trouble as they're inlined

[jnorthrup]

Talking about C, If you use liburing you don't need any extra sync userspace-kernel sync. There should be some when userspace threads/processes share a CQ or a SQ (e.g. submitting to a single ring from multiple threads). It depends, but it may be a good idea to keep them per-thread if possible.

I haven't yet located a CAS io_uring example or written C code in the decades which include atomics, but if such a thing is sane I probably won't be the only one asking.

[isilence]

About that shared CQ/SQ part I mentioned, they're just chunks of memory, nothing fancy. You can use any mutex of your choice or anything else your language provides, should work from the correctness perspective. Also a quick note: if CQ is used by one thread and SQ by another, that's also fine without extra sync if you have IORING_FEAT_EXT_ARG or not using timeouts.

Don't know where you're going to use CAS, might be not the best option if that's to do lockless concurrent submission.