how to use batching or gpu to inference

[Joyful-Buffalo]

[snakers4]

Hi, the VAD is not designed to run on GPU.
As for batching, we also do not support it officially, a it's too complicated and error prone to explain.

[ilyakam]

Hi @snakers4 , thank you for maintaining this model!

I see that you're saying this VAD is not designed to run on the GPU; however, the README says otherwise:

Using batching or GPU can also improve performance considerably. Under certain conditions ONNX may even run up to 4-5x faster.

It looks like that remark was added in 081e6b9 more than two years ago when v4 was released. Is it no longer accurate then?

In my own experiments, switching from onnxruntime to onnxruntime-gpu (both at v1.20.1), which includes the "CUDAExecutionProvider", actually slowed down VAD by about 50–60%, thus confirming what you're saying.

Do you have any suggestions how we might improve the performance of this model by throwing additional compute resources at it?

[snakers4]

which includes the "CUDAExecutionProvider", actually slowed down VAD by about 50–60%, thus confirming what you're saying.
Do you have any suggestions how we might improve the performance of this model by throwing additional compute resources at it?

The latest model is designed with several things in mind:

• Best quality for our domains we could achive balancing overall quality;
• Small model size and fast inference on CPU;
• Low compute;

So by design, the model is modest and does not scale with compute.

The model has state, i.e. is recurrent. The only way to increase throughput is to use batching. The model can work with batches, but in our early days we shared some batching code, and it raised more questions than in solved. The data passed into model now has batch_size 1. Some people wrote their functions changing this to more than 1. This is error prone though, because each element in the batch has its own "state" and it should be handled carefully.