How can SAHI do faster inference?

[rusvagzur]

Hi @fcakyon ,

I am currently using SAHI with YOLOv8. I am not completely sure that SAHI runs batch inference, since I have observed in the code that only num_batch=1 is supported and that inferences can take up to seconds per complete image depending on the number of slices.

I was wondering how could this package be improved in order to run batch inference in the GPU. Probably also doing the slices in the GPU directly when the original image is loaded.

Is batch inference dependant on YOLOv8? or is SAHI creating the bottleneck? Several seconds for the prediction of a single image doesn't justify higher small object detection for some real-world/real-time applications like security applications.

Best regards

[andressrodrl]

@rusvagzur

I´ve modified it to run inference with slice batches: https://github.com/andresinsitu/sahi_custom/tree/batch_inf
Added the argument batch to the predict and get_sliced_prediction function. It is working for me speeding it a bit.

At the moment the batch has to be a divisor of the number of slices, otherwise it leaves some slice without inference(but it works regardless)

[rusvagzur]

@andresinsitu Nice implementation. However, something is off. When I set the batch number and then the yolov8 model call with verbose=True, being batch number and number of slices the same, yolo is only called once. When I set the batch=1, then yolo is called multiple times, one per slice. So I am suspecting that when I introduce your changes, yolo is only performed in the original image and no prediction in the slices is happening.

[rusvagzur]

@andresinsitu I think the problem was in models/yolov8.py. num_group = int(num_slices / num_batch) should be:

num_group = 1 if num_batch >= num_slices else int(num_slices / num_batch)

To avoid num_group=0 when the batch number is higher than the number of slices. In my case, I have num_slices=12.

Once that is fixed, I get the following error when using num_batch>=6:

sahi/models/yolov8.py", line 87, in <listcomp>
   result.masks.data[result.boxes.data[:, 4] >= self.confidence_threshold],
AttributeError: 'NoneType' object has no attribute 'data'

And the following error when num_batch<6:

sahi/models/yolov8.py", line 101, in perform_inference
    self._original_shape = image.shape
AttributeError: 'list' object has no attribute 'shape'

[rusvagzur]

I have solved the first one by adding to result.masks.data[result.boxes.data[:, 4] >= self.confidence_threshold] the following

result.masks.data[result.boxes.data[:, 4] >= self.confidence_threshold] if result.masks is not None else result.boxes.data[result.boxes.data[:, 4] >= self.confidence_threshold],

The second error is solved with:

self._original_shape = image[0].shape if len(prediction_result)>1 else image.shape at the end of perform_inference.

Now it would be nice if the standard prediction done in sahi/predict.py, which is the prediction done to the whole downscaled image, would be done together with the batch when the batch number and the number of slices is the same. This would make SAHI as efficient as possible! This is where this happens now in predict.py:

# perform standard prediction
if num_slices > 1 and perform_standard_pred:
    prediction_result = get_prediction(
    image=[image],
    detection_model=detection_model,
    shift_amount=[0, 0],
    full_shape=None,
    postprocess=None,
)
object_prediction_list.extend(prediction_result.object_prediction_list)

But ideally should be done with something like this (also in predict.py):

prediction_result = get_prediction(
    image=image_list[0:num_batch]+[image], #being image the original image

Could you give me a hand with this @andresinsitu ?

[dceluis]

hey. I don't know how helpful this will be but I came up with a very small proof of concept using sahi + yolov8 to run batch inference. it's not a complete fork, more like a somewhat compatible replacement for get_sliced_prediction.

it probably only works for my use case but if you want to have a look and maybe grab some ideas here it is:
https://github.com/dceluis/sahi_batched