Support for single animal?

[lachioma]

Hi

Is it possible to run SLEAP inference using a single animal model (and not centroid+topdown)?

SLEAP toolbox should currently support only top-down models. However, since centroid and topdown models are combined to a single model before running inference, I tried to directly use my single animal model. From the demo code in the toolbox, I basically tried:
modelNet = topdownNet;
instead of
modelNet = sleap.model.topdown.compose(centroidNet, topdownNet, inputScale, bboxSize, minThreshold, centroidConfig, topdownConfig);

Then, when running the subsequent section "Run inference & visualize keypoints online (frame by frame)", I get the following error:

Error using DAGNetwork/predict
Incorrect input size. The input images must have a size of [304 480 1].

Error in test_sleap_infer (line 36)
    thisImgPeaks = modelNet.predict(thisImg);

The video frames I trained the model on and the video I am trying to run inference on are 480 x 300 (width x height). But the model expects images with 4 more pixels in height.

Any feedback would be appreciated, thanks!

-alessandro

[talmo]

Hi @lachioma,

Thanks for the report! We do not currently support single-animal models, but this should be doable with our existing code by adapting the topdown model class. The postprocessing logic for single-animal models is a lot simpler than topdown, and we can reuse the peak finding code, so we mainly just need to write out the classes and high level wrappers. We'll work on this for a future release.

In the meantime, it actually looks like your workaround gets close but the error you're getting is unrelated to the model type.

The model expects images that are 480 x 304 instead of 480 x 300 because we pad the image with zeros to ensure that it is evenly divisible by the max stride of the network. If we don't pad, we get shape mismatch errors when the upsampling branch of the model concatenates features via skip connections from the downsampling branch.

Suffice it to say, the fix here is to pad your image (bottom-right so it doesn't change the coordinates) before feeding it into the model. In core SLEAP we do this automatically by checking the config file for the trained model for what the max stride is so that we can pad during preprocessing. This functionality is not currently present in this toolbox, but we should add it as well as this also affects topdown models.

We'll update this thread once either of the above are implemented, but in the meantime try the manual padding as a workaround.

[WeissShahaf]

any news on when single model will be supported?