Bug in get_peft_model_state_dict when using vblora

[KaiyangLi1992]

System Info

The issue occurs when the following line is executed:

to_return["base_model.vblora_vector_bank." + adapter_name] = state_dict["base_model.vblora_vector_bank." + adapter_name]

• The state_dict does not contain a key named "base_model.vblora_vector_bank.default".
• Replacing it with the following resolves the issue:

for i in state_dict.keys():
    if "vblora_vector_bank" in i:
        to_return[i] = state_dict[i] 

---

I’m not sure if this is due to how I am calling or configuring the function.

Who can help?

@leo

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder
• My own task or dataset (give details below)

Reproduction

https://colab.research.google.com/drive/1e6ysneOZflu_TB5Pgj5zLTWyXbZW1Ezy#scrollTo=Da58JezBno0V

Expected behavior

Remove this bug.

[githubnemo]

Hey, thanks for reporting :)

I think the issue is caused by supplying a surprisingly filtered parameter dictionary due to the way VBLoRA manages the vector bank internally. Your code is the following:

params_dict = OrderedDict((name, param.detach()) for name, param in model.named_parameters() if
                                           "default" in name)
get_peft_model_state_dict(model, params_dict, "default")

You are retrieving a subset of parameters (params_dict) and then attempt to get the state dict for your model on said parameter subset. Since you are using VBLoRA, the code will look for base_model.vblora_vector_bank.default but your subset does not include this layer. So the question is: why is this layer not found? This is because model.named_parameters() will filter entries with duplicate values - since all vblora_vector_bank parameters are shared, only one of these parameters will be in the returned values of named_parameters.

So the fix on your side would be to supply the remove_duplicate=False parameter

params_dict = OrderedDict((name, param.detach()) 
    for name, param in model.named_parameters(remove_duplicate=False) if "default" in name)

I can see, though, that this is a bit error prone, especially because named_parameters is a very common method to get parameter subsets. @BenjaminBossan what do you think about using any vblora_vector_bank.*? Not sure if this is causing problems when nesting VBLoRA models, for example.

[BenjaminBossan]

Nice find, yes indeed I think it would be okay to make that change, nested VB-LoRA is probably not a realistic issue to be facing. Let's also ping the VB-LoRA author @leo-yangli (not "leo" as pinged above).