Max width of ensemble fan out

[zachary-mcpher]

Hello,

I am working on a project to serve an encoder based model in a Triton Inference Server ensemble. The nodes will be a preprocessing node which feeds directly into an encoder (generate an embedding feature from roberta-base) and then a fan out to K light weight classification head ( think N linear layers).

How far can I reasonably push K? Would the ensemble orchestrator be capable of handling inference at K=100 classifiers?