Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Official implementation of "Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs"

License

Notifications You must be signed in to change notification settings

OptMN-Lab/ALoRA

Open more actions menu

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Official implementation of Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs

Previous research works (HydraLoRA, FedSA-LoRA) have shown that the $A$ matrices from LoRAs fine-tuned on different tasks are often similar. We empirically find that this similarity arises from using the same initialization. When LoRAs share the same initialization, their $A$ matrices remain similar across tasks. In contrast, with different initializations, the $A$ matrices from the same task are not similar.

Similarity

Furthermore, we analyze the similarity, magnitude, and direction changes of LoRA modules before and after fine-tuning. First, we observe that the $A$ matrix changes very little, whereas the $B$ matrix exhibits substantial variation. Second, LoRA shows only limited changes in magnitude, while most of the directional change is captured by $B$. These findings suggest that $B$ plays a more critical role than $A$ in encoding knowledge.

Variation

Motivated by these findings, we propose sharing the $B$ matrix during fine-tuning across multiple LoRAs to improve parameter efficiency while preserving knowledge transfer. In multi-task fine-tuning, we propose ALoRA, which uses multiple $A$ matrices together with a single shared $B$ matrix. For federated fine-tuning, we introduce Fed-ALoRA, which shares only the $B$ matrix for server-side aggregation.

ALoRA for multi-task fine-tuning:

ALoRA

Fed-ALoRA for federated fine-tuning:

Fed-ALoRA

Please refer to each folder for more details.

About

Official implementation of "Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs"

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

Morty Proxy This is a proxified and sanitized view of the page, visit original site.