IterableDatasets support for Avalanche Experiences

[suiluj]

Hello, i am curious why IterableDatasets are not supported in Avalanche? ( https://avalanche.continualai.org/how-tos/avalanchedataset/preamble-pytorch-datasets#quick-note-on-the-iterabledataset-class )

My plan is to use a webdataset (https://github.com/webdataset/webdataset) for avalanche which is a Pytorch IterableDataset.
This would be the best format because i have many small binary data files which get preprocessed by dataset transforms.

Is there really no way to use IterableDatasets in Avalanche? Then why is this the case? Because it has to be indexable?

I thought about creating multiple Iterable Datasets and then switching to the next one after the previous one was used completely.
So there would be one IterableDataset for each experience. Every dataset contains a different data distribution, so the data changes with every new dataset.
Is this a feasible strategy? (I am new to avalance).

[AntonioCarta]

Currently, we don't support this usecase.

AvalancheDatasets add a couple of functionalities which are not available with Pytorch datasets (train/eval transforms, task labels, and other small things). Training algorithms use these functionalities during the training/eval loops.

We need to add something equivalent to pytorch's IterableDataset, but I can't say when we will be able to do it.

[AntonioCarta]

@suiluj Do you have an example of a popular CL benchmark implemented as an IterableDataset? We may need it when we are ready to add support for them.

[suiluj]

@AntonioCarta I just found a discussion about splitting tasks with multiple datasets and using them with multiple loaders in Pytorch Lightning. https://forums.pytorchlightning.ai/t/whats-the-best-practice-for-continual-learning/250

And Pytorch Lightning seems to support IterableDatasets: https://pytorch-lightning.readthedocs.io/en/stable/guides/data.html#accessing-dataloaders-within-lightningmodule

Perhaps this can be an inspiration. I noticed that you thought about integrating with Pytorch Lightning in this discussion: #388

[suiluj]

Just for clarification (sorry if i mix a few things here):

In my case I prefer IterableDatasets because I have stored examples as binary files of different batch sizes and yield single examples with some dataset transforms in a datapipe (using webdataset tar archives). In my thinking it is easier to not care about the exact number of examples for every task and know splits but just yield a complete IterableDataset. Then switch to the next dataset for the next task and so on.

That is why I have an IterableDataset.

[AntonioCarta]

Thanks for the suggestion. I think it may be way easier for us to support IterableDatasets than to support seamless integration with Pytorch Lightning.

In my case I prefer IterableDatasets because I have stored examples as binary files of different batch sizes and yield single examples with some dataset transforms in a datapipe (using webdataset tar archives). In my thinking it is easier to not care about the exact number of examples for every task and know splits but just yield a complete IterableDataset. Then switch to the next dataset for the next task and so on.

The usecase is clear and it is something that we want to support. I was asking for a public dataset to have something that we can use to work on the feature and to gauge how many people may use IterableDatasets.

[suiluj]

I have no public dataset yet. I'm working on distributed tracing graph data. We develop a small "simulation with changing configurations". When it is finished we think about publishing it. But we're still at the beginning and it will take a while.