IORING_ENTER_GETEVENTS behavior

[rch0wdhury]

Hi,

My application prepares a bunch of IOs (about 512) and then submits them using

ret = __sys_io_uring_enter(ring_fd, 512, 128, IORING_ENTER_GETEVENTS , NULL);

The application then peeks and reaps the completed IOs. My question is about the value of ret above. Am I correct in assuming the value of ret will be at least 128 because IORING_ENTER_GETEVENTS flag has been specified?

Is there any condition (without failure) where ret can be less than 128? In my kernel (5.10) I can get 128 completions - but only after multiple io_uring_enter syscalls. The ring has been created without SQPOLL or IOPOLL.

[axboe]

The syscall return value is always the number of requests submitted, not the number you waited for. The amount available in the CQ ring is readily available in userspace by just reading the ring. 5.10 is very old, newer kernels will have a setup flag that tells the submission side to always keep submitting even if an error occurred with one of the requests. Older kernels would stop submitting if an error occurred. Since there aren't a lot of details in your report, I'm guessing this is what is happening in your case.

In short, the return value should be 512, unless something caused it to stop submitting. That would normally be an error case.

[rch0wdhury]

I will check with newer kernels.
If there is an error, where do I see the error? In my case resubmitting does go through - just that I need to invoke io_uring_enter multiple times till the SQ is drained.
Also, does the min_complete field imply, unless any error, min_complete IOs will complete before the syscall returns?

[axboe]

The error will be in the completion events, in the cqe->res field. If it does submit fine with multiple invocations, that very much points to requests that error during submission.

Also, does the min_complete field imply, unless any error, min_complete IOs will complete before the syscall returns?

Yes, it's the number of events to wait for. More specifically, the number of CQEs that should be ready for reaping in the CQ ring upon return.

[rch0wdhury]

Got it.
I am testing in kernel 6.6 now - and printing the output of the following code segment -

512 IOs are prepared and are available when we enter main_loop

waitfor = 128;

int availableinsq(struct io_uring *ring) {
                unsigned tail = *ring->sq.ktail);
		unsigned head = io_uring_smp_load_acquire(ring->sq.khead);

		return (tail - head);
}

int availableincq(struct io_uring *ring) {
               unsigned tail = io_uring_smp_load_acquire(ring->cq.ktail);
		unsigned head = *ring->cq.khead;

		return (tail - head);
}

int main_loop() {
               while (availableincq(ring) < waitfor) {
                                ret = __sys_io_uring_enter(ring_fd, availableinsq(ring), waitfor - availableincq(ring), IORING_ENTER_GETEVENTS , NULL); 
                                printf("syscall return %u, pending = %u, available = %u\n", ret, availableinsq(ring),, availableincq(ring));
}

I see the following output -

syscall return 512, pending = 0, available = 92
syscall return 0, pending = 0, available = 92
syscall return 0, pending = 0, available = 105
syscall return 0, pending = 0, available = 106
syscall return 0, pending = 0, available = 108
syscall return 0, pending = 0, available = 109
syscall return 0, pending = 0, available = 118
syscall return 0, pending = 0, available = 119
syscall return 0, pending = 0, available = 123
syscall return 0, pending = 0, available = 124
syscall return 0, pending = 0, available = 128

So there are no submit errors and all SQ was consumed properly. But the minimum_complete argument did not seem to be honored.

[axboe]

Try and setup your ring with IORING_SETUP_DEFER_TASKRUN|IORING_SETUP_SINGLE_ISSUER

[DylanZA]

probably also cache availableincq() somewhere, it's value will change continuously

but anyway - if available=92, and wait_for=128, then you will be asking it to wait for min 36, which is honoured