Low-Budget LoRA: Strategic Adapter Placement and Rank for Efficient Model Tuning

Abstract

We study how LoRA adapter placement and rank choices impact quality and cost on constrained budgets. Through experiments across encoder and decoder stacks, we identify layers where adapters deliver the best gain per parameter and show that modest ranks retain most performance while slashing training cost. The results offer a practical recipe for teams tuning models with limited compute: place adapters selectively, keep ranks lean, and recover strong task performance without the expense of full fine-tuning.