ReLoRa: Pre-train a Large Language Model on Your GPU
LoRa but with multiple resets in a row
Published in
8 min readJul 20, 2023
In 2021, Hu et al. proposed low-rank adapters (LoRa) for LLMs. This method significantly reduces the cost of fine-tuning large language models (LLMs) by only training a few added parameters (low-rank networks) while keeping the LLM’s original parameters (high-rank networks) frozen.