Rust Bindings for DALI

[idobenamram]

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

Nice to have (e.g. Adoption is possible, the feature will enhance the use-case even more).

Please provide a clear description of problem this feature solves

Hey! We talked about supporting rust in #5320, i created a draft pr to just get things started. I have a few questions to make sure I'm on the right track:

1. the c_api_test.cc uses internal functions and structs to create a pipeline on the spot - this is not possible from rust since we only have the c_api.h methods to work with. So if we want to make sure everything works we need to use an already serialized pipeline. Can you point me to a test or a place that does this?
2. currently I assume that you have DALI installed. There is an option to either download, or compile in the build.rs file. What do you guys think is the best thing to do here since the compilation is pretty complex (need to first install the dependencies from a different repo). I see the in the release you do upload the binaries so we can download them based on the CUDA version?
3. Related to question 2, how do people usually use dali, from a docker? we usually use dali using the triton server, so I'm not sure how people usually install dali.
4. Did i get the correct libs that need to be linked for everything to work in build.rs?

Feature Description

Support rust bindings by adding a cargo crate that wraps dali c_api.h

Describe your ideal solution

using bindgen to create the wrapper and then create idomatic Rust structs to make usage easier.

Describe any alternatives you have considered

No response

Additional context

No response

Check for duplicates

• I have searched the open bugs/issues and have found no duplicates for this bug report

[banasraf]

Hi @idobenamram

1. DALI C API does not support building pipelines from scratch. Its purpose is to run pipelines that were composed in some other way (e.g. through Python). This means, that if we wanted to provide full DALI bindings for Rust we would have to first develop proper C bindings. Although, the scope of the current C API is enough to cover many use-cases (e.g. it's all we use in the backend for Triton), so such Rust crate could also be useful.
2. I think compiling DALI in build.rs could be really hard to do. WIth such complex libraries, I think it's ok to assume that it's already installed, or install it from binaries. I think in Rust the standard is to provide -sys crates for such libraries that only handle the dependency and provide bare minimum C bindings.
3. The usual way of using DALI is just installing it through pip install

Other thing worth noting is that we don't provide correctly defined C API which is a problem when u want to build the bindings. We throw exceptions from there which will cause issues. To build any bindings we would need to first wrap all of that c_api in some additional layer that would do a proper C error handling.

[banasraf]

Thanks for providing your draft PR! I'm sure that having access to DALI through Rust is a good way of reaching more users.

I'm afraid creating correct Rust bindings would require more effort unfortunately, due to issues with our C API.
I'm also not sure if we want to actually provide the Rust crate files in the main DALI repository because that would require support and maintenance from our side.

And I'm almost sure that providing fully fledged, idiomatic Rust bindings for DALI is something that we don't have cycles for right now.