Is there an efficient way to measure the memory usage?

[PenutChen]

I'm measuring the relationship between sequence length, prefilled batch size, and GPU memory usage. For now, the only way I can measure memory usage is through trial and error using the --cuda-memory-fraction flag, since TGI seems to use all the available memory by default (I think this is caused by the paged attention mechanism?).

I'm wondering if there's a more efficient way to do this?

[OlivierDehaene]

Not really no. You could add a few prints here to get the info you need.

[PenutChen]

Thanks, I'll try to dig into it!