lora config
#6
by SeanLee97 - opened
Excellent work!
I've read the technical paper. It mentions the inclusion of LoRA adapters in all linear layers.
Based on my understanding, you specified the target_modules as follows: ['v_proj', 'q_proj', 'down_proj', 'k_proj', 'gate_proj', 'o_proj', 'up_proj'].
Is my understanding correct?
Could you share the detailed Lora config? Thx!
Thanks @SeanLee97 ! I learn a lot from your AnglE paper!
The released model contains merged lora weights, I thought this would make it more convenient to use without depending on PEFT library.
Here is our LoRA config:
{
"auto_mapping": null,
"base_model_name_or_path": "mistralai/Mistral-7B-v0.1",
"bias": "none",
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layers_pattern": null,
"layers_to_transform": null,
"lora_alpha": 32,
"lora_dropout": 0.1,
"modules_to_save": null,
"peft_type": "LORA",
"r": 16,
"revision": null,
"target_modules": [
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"down_proj",
"up_proj",
"gate_proj"
],
"task_type": "FEATURE_EXTRACTION"
}