Multi channel audio visual + Beamforming

[YosraGriri]

Hello, I have read your paper and found it fascinating. I am interested in experimenting with your code to adapt it for multi-channel data possibly using beamforming. Could you provide any guidance or leads on how to approach this? This endeavor is part of my research for my master's thesis, and your assistance would be greatly appreciated. Thank you!

[hmartelb]

Hello @YosraGriri, and thank you for your comment. I am glad you find the paper inspiring for your research.

• For the model to work with multi-channel data, you will need to change this line either manually or by passing another argument to the model class.

self.audio_encoder = nn.Conv1d(
    in_channels=1, # Set the number of channels you want here
    out_channels=audio_hidden_channels,
    kernel_size=kernel_size,
    stride=kernel_size // 2,
    padding=kernel_size // 2,
    bias=False,
)

• As for beamforming, it depends on at what stage you intend to do it. Is it meant to be applied to the inputs or the outputs? I may be able to assist with more context.

[YosraGriri]

Hello @hmartelb,
I appreciate your quick reply. Thank you very much for that!
As for beamforming, I'm in the process of reading the latest literature on the subject. I'm thinking about applying it to the input, similar to the approach discussed in this paper: https://link.springer.com/article/10.1007/s11265-022-01770-7.
Also, I have a question about the parameters you used when creating the dataset. I am interested in recreating the dataset, but this time with a multi-channel microphone array and LRS3 data.
Could you please provide details of the parameters used? (microphone position ... source position ... room size ... RT60)
Thanks in advance for the answer