GiVA: Gradient-Informed Bases for Vector-Based Adaptation
Paper Overview
- Research Area: NLP
- Authors: Neeraj Gangwar, Rishabh Deshmukh, Michael Shavlovsky, Hancao Li, Vivek Mittal, Lexing Ying, Nickvash Kani
- Published: 2026-04-23
- arXiv: 2604.21901
- GiVA consistently outperforms or matches the performance of existing vector-based adaptation methods and LoRA.
- It reduces rank requirements by 8x compared to prior approaches.
- Training time remains on par with LoRA, avoiding the cost overhead of higher-rank alternatives.
- The method is validated across NLU, NLG, and image classification benchmarks.
Summary
As model sizes continue to grow, parameter-efficient fine-tuning has become a powerful alternative to full fine-tuning. While LoRA is the most widely adopted among these methods, recent research has explored vector-based adaptation approaches due to their extreme parameter efficiency. However, these methods usually require a much higher rank than LoRA to match its performance, leading to increased training cost.
This work introduces GiVA, a gradient-informed initialization strategy for vector-based adaptation. It achieves training time comparable to LoRA while maintaining the extreme parameter efficiency of vector adaptation. The authors evaluate GiVA on diverse benchmarks, including natural language understanding, natural language generation, and image classification.
Key Findings
*Auto-collected on 2026-04-27*