EFT-LR: Benchmarking Learning Rate Policies in Parameter-Efficient Large Language Model Fine-tuning
Conference
Jawad, MT, Wu, Y. (2025). EFT-LR: Benchmarking Learning Rate Policies in Parameter-Efficient Large Language Model Fine-tuning
. 6408-6413. 10.1145/3746252.3761630
Jawad, MT, Wu, Y. (2025). EFT-LR: Benchmarking Learning Rate Policies in Parameter-Efficient Large Language Model Fine-tuning
. 6408-6413. 10.1145/3746252.3761630
Large Language Models (LLMs) have achieved extensive impacts across various real-world data mining applications. Given the extremely high cost of training or fine-tuning LLMs, parameter-efficient fine-tuning (e.g., LoRA) has emerged as a popular and practical approach for adapting pre-trained general-purpose LLMs to specific downstream tasks. Among the various hyperparameters involved in parameter-efficient fine-tuning of LLMs, the learning rate (LR) plays a crucial role in determining the overall performance. However, it lacks a systematic benchmark framework to explore and understand how different LR policies influence the effectiveness of parameter-efficient LLM fine-tuning, which makes it challenging to select an optimal LR policy. To address this critical research gap, this paper introduces a systematic benchmark, EFT-LR, for assessing and selecting LR policies for effective parameter-efficient fine-tuning of LLMs. We first present a collection of seven popular LR policies spanning three major categories in the literature. We then perform parameter-efficient fine-tuning of LLMs using these LR policies and assess fine-tuned LLMs on eight downstream tasks. Our empirical analysis using EFT-LR provides an in-depth investigation of the impacts of different LR policies on parameter-efficient LLM fine-tuning, offering practical guidelines for practitioners. We provide the source code at https://github.com/mlsysx/EFT-LR.
