Continual learning for a custom dataset

[dalbertweiss]

I'm trying to apply avalanche to my custom dataset. For this, I have created a dummy dataset using TensorDataset and the wrapper AvalancheDataset as proposed in the documentation. However, when applying my continual learning strategy I get an AttributeError. Here I provide a minimal example of my code:

import torch
from torch.nn import CrossEntropyLoss
from torch.optim import SGD

from avalanche.benchmarks.classic import PermutedMNIST
from avalanche.models import SimpleMLP
from avalanche.training import Naive
from avalanche.benchmarks.utils import AvalancheDataset
from avalanche.benchmarks.utils import AvalancheTensorDataset
from avalanche.benchmarks.utils import AvalancheConcatDataset

# Config
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
# model
model = SimpleMLP(num_classes=10)

x_data = torch.rand(10, 7)
# Create the class labels for the 10 instances
y_data = torch.randint(0, 5, (10,))


# CL Benchmark Creation
# X_amp_tr_A, X_phase_tr_A, weight_tr_A, temp_tr_A, humidity_tr_A,sl_clf_train
avl_tensor_dataset = TensorDataset(x_data, y_data)
wrapped_tensor_dataset = AvalancheDataset(avl_tensor_dataset)


#generic_scenario = paths_benchmark(
#        train_experiences,
#        test_experiences,
#        task_labels=[0, 5],
#        complete_test_set_only=False)
#

# Prepare for training & testing
optimizer = SGD(model.parameters(), lr=0.001, momentum=0.9)
criterion = CrossEntropyLoss()

# Continual learning strategy
cl_strategy = Naive(
    model, optimizer, criterion, train_mb_size=32, train_epochs=2,
    eval_mb_size=32, device=device)

# train and test loop over the stream of experiences
results = []
for train_exp in wrapped_tensor_dataset.train_stream:
    cl_strategy.train(train_exp)
    #results.append(cl_strategy.eval(avl_concat.test_stream)) #help:https://avalanche.continualai.org/getting-started/learn-avalanche-in-5-minute
    

[AndreaCossu]

I assume you are running version 0.2.0.
I cannot test it right now, but can you try adding targets=y_data, dataset_type=AvalancheDatasetType.CLASSIFICATION when you create the AvalancheDataset?

[dalbertweiss]

Yes you are right. I'm running on 0.2.0.
Unfortunately this doesnt work. Unfortunately it's telling me that AvalancheDatsetType is undefined:

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_8212\119399570.py in <cell line: 25>()
     23 # X_amp_tr_A, X_phase_tr_A, weight_tr_A, temp_tr_A, humidity_tr_A,sl_clf_train
     24 avl_tensor_dataset = TensorDataset(x_data, y_data)
---> 25 wrapped_tensor_dataset = AvalancheDataset(avl_tensor_dataset, targets=y_data, dataset_type=AvalancheDatasetType.CLASSIFICATION)
     26 
     27 

NameError: name 'AvalancheDatasetType' is not defined

[AndreaCossu]

Have you imported it?
from avalanche.benchmarks.utils import AvalancheDatasetType

[dalbertweiss]

Yes I tried this, however, the AttributeError remains.

[AndreaCossu]

You are probably using 0.2.0a which is the master branch version, where AvalancheDatasetType has been removed. I can import and use it in the 0.2.0 without any errors. However, the problem in your script is not due to this. I provided an answer below 😄

[AndreaCossu]

Sorry, I looked at the code in a hurry and missed one important detail 😄
You are creating a dataset, not a benchmark. The benchmark.train_stream property you are trying to call refers to the stream of datasets contained in the benchmark instance. If you want to create a benchmark from a list of datasets I recommend to use the dataset_benchmark generator.

[dalbertweiss]

Perfect this was exactly what I was searching for. I was able to make it run on my own dataset. Two little question: do you also support custom datasets (for instance to pass multiple inputs to the model)?

[AndreaCossu]

Great! Yes, the dataset can return more than 2 values. Avalanche will append task labels as last argument, while the others will remain unchanged, resulting in for x,y,c,d,...,task_labels in dataset. You have this for free when you create a benchmark, you just need to provide datasets which return whatever number of objects you want.

[dalbertweiss]

Awesome. See my message below for the ValueError

[dalbertweiss]

Awesome. Does this also apply for tasks inputs. When I have a custom dataset with multipe inputs I receive a ValueError:

ValueError: Unsupported dataset: must have a valid targets field or has to be a Tensor Dataset with at least 2 Tensors

Let me share my code with you in a minimal example:

class Dataset:
    def __init__(self, x1, x2,
                 y):
        self.x1 = x1
        self.x2= x2
        self.y = y

    def __getitem__(self, idx):
        #inputs
        x1 = self.x1[idx]
        x2 = self.x2[idx]        
        #outputs
        y = self.y[idx]
        return ([torch.tensor(x1, dtype=torch.float32),torch.tensor(x2, dtype=torch.float32)],torch.tensor(y, dtype=torch.long))

#declare data


x1_tr = torch.from_numpy(np.random.randn(1000,100,1))
x2_tr= torch.from_numpy(np.random.randn(1000,100,1))
y_data_tr1 = torch.from_numpy(np.random.randint((1000,1))).type(torch.LongTensor)
y_data_tr2 = torch.from_numpy(np.random.randint((1000,1))).type(torch.LongTensor)

x1_test = torch.from_numpy(np.random.randn(300,100,1))
x2_test= torch.from_numpy(np.random.randn(300,100,1))
y_data_test1 = torch.from_numpy(np.random.randint((300,1))).type(torch.LongTensor)
y_data_test2 = torch.from_numpy(np.random.randint((300,1))).type(torch.LongTensor)                    
                              
#my model
class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.x1_ = nn.Sequential(
            nn.Conv1d(1, 1, 3, 1),
        )
        self.x2_ = nn.Sequential(
            nn.Conv1d(1, 1, 3, 1),
        )
        
        self.backbone = nn.Sequential(
            nn.Conv1d(1,128,3,1),
            nn.Conv1d(128,128,3,1),
            nn.AvgPool1d(3),
            nn.Dropout(0.3)
        )
        
        
        self.classifier  = nn.LazyLinear(3)

    def forward(self, x1):
        x1 = self.x1_(x1)
        x2 = self.x2_(x2)
                                                                                                                                                                                                                                                     
        x1 = x1.view(x1.size(0), -1)                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                     
        x2 = x2.view(x2.size(0), -1)                                                                                                                                                                                                                                                    
        x = torch.cat((x1, x2), dim=-1)

        x = x1.unsqueeze(1)
        x = self.backbone(x)
        x = torch.flatten(x, start_dim=1, end_dim=-1)

        clf = self.classifier(x)     
        
        return clf


model = MyModel()

# load packages    
import torch
from torch.nn import CrossEntropyLoss
from torch.optim import SGD

from avalanche.benchmarks.classic import PermutedMNIST
from avalanche.models import SimpleMLP
from avalanche.training import Naive
from avalanche.benchmarks.utils import AvalancheDataset
from avalanche.benchmarks.utils import AvalancheTensorDataset
from avalanche.benchmarks.utils import AvalancheConcatDataset
from avalanche.benchmarks.utils import AvalancheDatasetType
from avalanche.benchmarks.generators import dataset_benchmark
# Config
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")


dataset_tr1 = Dataset(x1_tr, x2_tr, y_data_tr1)
dataset_test1 = Dataset(x1_test, x2_test, sl_test)
dataset_tr2 = Dataset(x1_tr, x2_tr, y_data_tr2)
dataset_test2 = Dataset(x1_test, x2_test, y_data_test2)

# CL Benchmark Creation
wrapped_tensor_dataset = dataset_benchmark([dataset_tr1, dataset_test1], [dataset_tr2, dataset_test2])

# Prepare for training & testing
optimizer = optim.Adam(model.parameters(), lr=0.005)
criterion = CrossEntropyLoss()

# Continual learning strategy
cl_strategy = Naive(
    model.double(), optimizer, criterion, train_mb_size=8, train_epochs=10,
    eval_mb_size=8, device=device)

# train and test loop over the stream of experiences
results = []
for train_exp in wrapped_tensor_dataset.train_stream:
    res = cl_strategy.train(train_exp)
    results.append(cl_strategy.eval(wrapped_tensor_dataset.test_stream))