How do people generally approach lifetimes of allocations

[CharlieTap]

The way I see it theres two lifetimes associated with liburing:

• Lifetimes that live as long as the ring (for example an array of direct file descriptors)
• Lifetimes that live until an sqe is consumed (for example a file path embedded in an sqe)

The former seems pretty easy, we just free on io_uring_queue_exit
The latter I'm a bit unsure of, a natural place feels like when either io_uring_submit or io_uring_submit_and_wait is called.

I found this whilst going through the docs, which gives me some confidence:

For any request that passes in data in a struct, that data must
remain valid until the request has been successfully submitted.
It need not remain valid until completion. Once a request has
been submitted, the in-kernel state is stable. Very early kernels
(5.4 and earlier) required state to be stable until the
completion occurred. Applications can test for this behavior by
inspecting the IORING_FEAT_SUBMIT_STABLE flag passed back from
io_uring_queue_init_params(3). In general, the man pages for the
individual prep helpers will have a note mentioning this fact as
well, if required for the given command.

However if the ring has SQ_POLL enabled is it possible that these calls are never made? I guess theres another question there which is: Is io_uring_submit more than just a a trigger for the io_uring_enter syscall or does it actually place prepared requests onto the submission queue and therefore is required even when the kernel is polling the SQ?

[axboe]

Most requests will follow the guarantee that any meta data structures (like the iovec for a read/write, for example) must remain stable until submission has been done. This means until io_uring_submit() returns. For SQPOLL, it's not possible for the application to know when the request has been consumed by the kernel thread, so for that case, data structures need to remain stable until the CQE for the given request has been posted.

Basically liburing caches the SQ ring tail, preventing SQPOLL or just the kernel in general from seeing request that have been prepared with a sequence of io_uring_get_sqe() + io_uring_prep_foo(sqe). Once io_uring_submit() is called, it calls __io_uring_sq_flush() which takes care of making prepared SQEs visible in the SQ ring for the kernel to consume.

[cmazakas]

I handle all the lifetimes in io_uring by heap-allocating some sort of per-operation state. This is kept alive by single-threaded reference counted pointers. The SQE <-> CQE pair has one half of the ownership and then the actual application code written by the developer participates in the other half.

To this end, everything is guaranteed to be kept alive and gives the user a nice sound interface.

It gets a bit trickier for the multishot ops but in general, I can't complain.

[noteflakes]

It should be pretty simple: whenever you add an SQE, you increment the corresponding ref count. Whenever you receive a CQE, you decrement the corresponding ref count. This also works for recv, since you'll encounter a CQE only when the operation is done.

It gets a bit trickier for the multishot ops but in general, I can't complain.

For multishot operations, I set the ref count to -1. When a CQE is encountered, the program will set it to zero only if ctx->res == -ECANCELED || !(cqe->flags & IORING_CQE_F_MORE).

[cmazakas]

Imo, there's a couple of aspects to this.

There's data io_uring needs to have remain stable until submission and then there's data that your code is likely going to need to read and touch.

It's often more efficient to just allocate once and then spread this ownership between io_uring and your application code.

noteflakes already touched on this but you increment when generating the SQE and then you decrement once you reap the CQE. Your application also participates in this ownership so this way, it doesn't matter what order things complete in or if the application code bails on its path of execution or anything like that.

Note, the liburing docs really only state that stability is required until submission but this is kind of meaningless in and of itself because a more sophisticated runtime is going to buffer SQEs and only call io_uring_submit() or io_uring_submit_and_wait() once a certain threshold has been hit which means you effectively have to treat it as unstable from an application perspective.