LoRA (Low-Rank Adaptation)

Low-Rank Adaptation, usually abbreviated LoRA, is a parameter-efficient fine-tuning method for adapting a pre-trained model. Instead of changing a large weight matrix directly, LoRA freezes the original matrix and learns a low-rank additive update represented by two much smaller matrices. Edward Hu and collaborators introduced the method in 2021; the paper appeared at ICLR 2022.^[1]

LoRA is best understood as a constraint on a model update, not as a way to make the base model itself smaller. It can substantially reduce the number of trainable parameters, optimizer state, and task-specific checkpoint storage. It does not remove the need to load the base weights, and it does not by itself eliminate activation memory. Its quality and efficiency therefore depend on the model, target modules, data, rank, optimizer, precision, and serving mode.

Core idea

Consider a frozen linear layer with weight

$$W_0 \in \mathbb{R}^{d_{\mathrm{out}} \times d_{\mathrm{in}}}.$$

Full fine-tuning can learn an unrestricted update Delta W with the same shape as W0. LoRA instead writes the update as

$$\Delta W = sBA,$$

where

$$B \in \mathbb{R}^{d_{\mathrm{out}} \times r}, \qquad A \in \mathbb{R}^{r \times d_{\mathrm{in}}},$$

Here r is the adapter rank and s is a scaling factor. The layer computes

$$y = W_0x + sB(Ax).$$

The update satisfies rank(Delta W) <= r. The two factors contain

$$r(d_{\mathrm{in}} + d_{\mathrm{out}})$$

trainable parameters, compared with d_in * d_out parameters for an unrestricted update to that matrix. This is useful when r is much smaller than both matrix dimensions. The saving for a whole model is the sum over every adapted matrix, so a quoted rank alone is not enough to determine an adapter's size.

This construction is related to matrix factorization, but the purpose is different. LoRA does not normally factorize and replace W0. It learns a factorized change beside a frozen W0. Some libraries name or order the two factors differently. Their shapes and multiplication order, rather than the letters A and B, determine the actual update.

Why freeze the base matrix?

Freezing W0 has three direct consequences:

• gradients and optimizer moments are not stored for W0;
• one base checkpoint can be shared by many small task-specific adapters;
• removing or disabling the adapter recovers the base computation, provided no other parameters such as biases or output heads were trained.

The base weights still participate in the forward pass, and gradients with respect to layer inputs are still needed for earlier trainable layers. LoRA therefore reduces parameter and optimizer-state costs more directly than activation costs. This distinction matters when long sequences or large batches dominate training memory.

Initialization, scaling, and gradients

No-op initialization

The original paper initialized A randomly and B to zero, which makes Delta W = 0 at the start of training.^[1] Microsoft's reference loralib uses Kaiming initialization for the linear layer's A factor and zeros for B; the exact random distribution is therefore implementation-dependent, while the zero initial update is the important invariant.^[2]

For a loss gradient G = partial L / partial Delta W, the factor gradients, omitting dropout, are

$$\frac{\partial L}{\partial A}=sB^\top G, \qquad \frac{\partial L}{\partial B}=sGA^\top.$$

When B = 0, the first update to A is zero, while B can receive a nonzero gradient through the random A. After B moves away from zero, both factors can learn. Initializing both factors to zero would instead block both gradients at the first step.

Standard scaling

The original parameterization uses

$$s=\frac{\alpha}{r},$$

where alpha is a constant chosen for the adapter.^[1] This means that rank, alpha, factor initialization, and learning rate interact. Two configurations with the same rank but different alpha do not generally have the same effective update scale.

There is no universal best rank or alpha. A higher rank increases capacity and trainable state, but it does not guarantee better validation performance. A lower rank can regularize the update, but it can also become a capacity bottleneck. The appropriate comparison holds the data split, target modules, optimizer, schedule, precision, and evaluation procedure fixed, then sweeps rank and learning rate together.

rsLoRA changes the scaling to alpha / sqrt(r). Its 2023 preprint argues theoretically and experimentally that the standard 1 / r factor can weaken learning as rank grows, whereas the square-root factor stabilizes higher-rank training.^[3] This is a distinct variant, not a retrospective change to the definition of standard LoRA.

Dropout and bias training

LoRA dropout is normally applied on the adapter branch, not to the frozen base branch. Biases can remain frozen or be trained separately. In Hugging Face PEFT 0.13.0, for example, bias may be none, all, or lora_only; training a bias means that disabling the adapter no longer necessarily reproduces the unmodified base model.^[4]

Where LoRA is applied

LoRA can be attached to any compatible dense weight matrix. In a Transformer, common candidates include the query, key, value, and output projections in attention, plus the up, gate, and down projections in feed-forward blocks. Embeddings, convolutional layers, and output heads require architecture-specific support.

The original paper deliberately limited most of its study to attention weights and often adapted the query and value projections. It left feed-forward layers, normalization layers, and several other choices for future study.^[1] This experimental scope should not be read as a universal prescription.

Target-module selection changes both capacity and cost. Adapting every linear projection at rank 8 can involve more trainable parameters than adapting only query and value projections at rank 32. It can also change which functions are easy to learn. A target-module report should therefore include exact module names, not only a phrase such as "attention LoRA."

Hugging Face PEFT 0.13.0 exposes target_modules, modules_to_save, per-layer rank and alpha patterns, and an all-linear selector. It also documents architecture-dependent orientation through fan_in_fan_out.^[4] These are implementation controls, not evidence that any one selection is optimal.

What LoRA saves

Trainable parameters and optimizer state

For each adapted matrix, LoRA replaces an unrestricted d_out * d_in trainable update with r * (d_in + d_out) trainable factor parameters. A first-order optimizer needs gradients only for those factors; Adam-like optimizers also keep their moment estimates only for the trainable state. This can be a large saving when the base model is frozen and r is small.

Checkpoint storage

An adapter checkpoint can store only the factor weights and a configuration file. It is not a standalone model. PEFT's checkpoint documentation states that the base model must also be available and records fields such as the base model name or path and revision when known.^[5] For reproducibility, an adapter release should identify at least:

• the exact base checkpoint and revision;
• the tokenizer or processor revision;
• target module names and layer selection;
• rank, alpha, scaling convention, dropout, and bias policy;
• any additionally trained modules;
• framework and library versions;
• weight precision and serialization format.

An adapter trained for one base revision is not automatically valid for another model with the same architecture or marketing name. The stored Delta W was optimized relative to a particular W0.

Training memory

LoRA removes base-weight gradients and most optimizer state, but a training process may still hold:

• the base weights;
• activations needed for backpropagation;
• temporary matrix-multiplication workspaces;
• attention states and masks;
• adapter activations and gradients;
• any trainable heads, embeddings, normalization parameters, or biases.

The actual memory reduction depends on sequence length, batch size, checkpointing, sharding, offloading, optimizer, and numerical precision. A parameter-count ratio is not a memory ratio.

Compute

An unmerged adapter adds the two factor multiplications Ax and B(Ax). Training can still be faster because the system avoids computing and communicating most weight gradients, but wall-clock speed depends on kernel shapes, memory traffic, batching, and distributed layout. Small low-rank matrix multiplications are not always hardware-efficient. Reported speedups should be tied to a specific hardware and software configuration.

Merged and unmerged inference

For a single adapter, the factors can be merged into the base matrix:

$$W_{\mathrm{merged}}=W_0+sBA.$$

The layer then uses the same matrix multiplication shape as the original layer. This is the basis for the original paper's "no additional inference latency" claim.^[1] The claim applies to a correctly merged adapter. It does not mean that every unmerged, hot-swapped, quantized, or multi-adapter implementation has zero overhead.

Merged and unmerged calculations are algebraically equivalent in exact arithmetic, but finite-precision rounding, quantization, and library implementation can introduce small numerical differences. A merge should therefore be followed by output-parity tests at the intended precision.

Microsoft's reference implementation merges on evaluation and subtracts the update when returning to training, as long as its merge option is enabled.^[2] PEFT's merge_and_unload() has different lifecycle semantics: the resulting object is a basic model without PEFT methods, cannot disable or unmerge the adapter through that object, and occupies the storage of a full model.^[5] Production procedures must follow the behavior of the specific library in use.

An unmerged form is preferable when a service must enable, disable, combine, or switch adapters without rewriting base weights. It trades that flexibility for adapter-branch compute and more complex scheduling.

Evidence from the original paper

The LoRA paper evaluated RoBERTa, DeBERTa, GPT-2, and GPT-3 on a selection of natural-language understanding and generation tasks. In those experiments, LoRA was competitive with or better than the full-fine-tuning and parameter-efficient baselines reported in the paper, while training far fewer parameters.^[1] The claim is empirical and bounded by those models, datasets, seeds, metrics, and baseline procedures.

The largest case study used GPT-3 175B. The paper reported the following validation results:

Scroll sideways for more →

The same table described approximate fluctuations of 0.5 percentage points for WikiSQL, 0.1 for MNLI-m, and 0.2/0.2/0.1 for the three SAMSum metrics.^[1] Differences smaller than those reported fluctuations should not be treated as decisive.

For GPT-3 175B with rank 4 on query and value projections, the authors reported reducing training memory from 1.2 TB to 350 GB and task-checkpoint storage from 350 GB to 35 MB. They also reported throughput of 43.1 tokens per second per V100 for LoRA versus 32.5 for full fine-tuning with the same weight-sharding setup.^[1] These often-cited numbers describe that particular 2021 system, not a general conversion factor for modern models.

The paper's rank analysis found that small ranks were sufficient for several tasks in its attention-only GPT-3 experiments. It did not establish that rank 1, 2, 4, or 8 is sufficient for every model, target module, dataset, or adaptation objective.

Why low rank can work

An important precursor was the study of intrinsic dimension in language-model adaptation. Aghajanyan and collaborators optimized parameters in a randomly projected subspace and, in one reported MRPC experiment, reached 90 percent of full-model performance with 200 trainable coordinates. They found that pre-training was associated with lower measured intrinsic dimension across their studied models and tasks.^[6]

This work motivated the hypothesis that useful adaptation may occupy a much smaller space than the full parameter space. It did not prove that a particular weight update must be low rank. Intrinsic dimension of an optimization problem, matrix rank of Delta W, and number of LoRA factor parameters are related intuitions but different mathematical quantities.

The original LoRA paper supplied empirical singular-value analyses of learned attention updates.^[1] Later theory studied representational capacity. Zeng and Lee proved existence results for low-rank adapters in fully connected and Transformer networks under stated assumptions and rank thresholds.^[7] Their paper explicitly analyzes expressivity, not whether gradient descent will find the adapters or whether they will generalize. Such theorems should not be converted into a universal rank-selection rule.

LoRA versus other adaptation methods

Several methods reduce task-specific state in different ways:

Scroll sideways for more →

Houlsby adapters reached within 0.4 percentage points of full fine-tuning on the paper's GLUE aggregate while adding 3.6 percent parameters per task.^[8] Prefix tuning trained continuous prefixes. Its 0.1-percent GPT-2 prefixes produced comparable full-data results on the studied table-to-text benchmarks, while its 0.1-percent and 2-percent BART prefixes scored below full fine-tuning on the XSum summarization metrics.^[9] Prompt tuning learned soft prompts and became more competitive with full model tuning as T5 scale increased in the experiments of Lester and collaborators.^[10] BitFit updated biases and was competitive with full fine-tuning on the studied BERT tasks with small to medium datasets.^[11]

Those results were obtained on different models and tasks. They do not define a universal ranking. The relevant choice depends on whether a system values tiny checkpoints, unchanged sequence length, no added layer depth, hot swapping, maximum capacity, or compatibility with an existing runtime.

QLoRA

QLoRA combines LoRA with quantization of the frozen base model. The 2023 paper stored the base weights in 4-bit NormalFloat, dequantized them to a computation type for forward and backward operations, and propagated gradients into LoRA factors rather than the quantized base weights. It also introduced double quantization of quantization constants and paged optimizers.^[12]

QLoRA therefore changes a different memory component from ordinary LoRA:

• LoRA reduces trainable parameter, gradient, and optimizer state.
• QLoRA additionally reduces storage for the frozen base weights.

In the paper's setup, QLoRA enabled fine-tuning a 65-billion-parameter model on a single 48 GB GPU while preserving the authors' selected 16-bit fine-tuning performance comparisons.^[12] This is a paper-specific systems result. Maximum trainable model size still depends on sequence length, batch size, activations, quantization implementation, and offloading.

QLoRA is not an adapter quantized to 4 bits by definition. Its defining feature is a quantized frozen base with trainable low-rank adapters. Merging a LoRA adapter into quantized weights may require dequantization or a library-specific path, and some configurations do not support merging.^[5]

Major variants

LoRA has produced many variants. Their names describe different changes and should not be treated as interchangeable labels.

Scroll sideways for more →

These methods can change initialization, scaling, trainable state, optimizer behavior, or even the surrounding attention algorithm. A checkpoint should identify the precise variant and library configuration. A generic "LoRA" loader may not reproduce a DoRA, PiSSA, VeRA, or AdaLoRA checkpoint correctly.

Capacity, learning, and forgetting

LoRA and full fine-tuning can achieve similar task scores while learning different parameter-space solutions. They can also diverge sharply when the adapter lacks capacity or when optimization settings favor one method.

A 2024 TMLR study compared LoRA with full fine-tuning on programming and mathematics. It examined instruction fine-tuning with about 100,000 prompt-response pairs and continued pre-training with up to about 20 billion tokens. In the standard low-rank settings studied, LoRA learned less of the target domain, but retained more performance outside that domain. The authors also found that full-fine-tuning perturbations had ranks about 10 to 100 times those of typical LoRA configurations in their analysis.^[21] These findings concern the tested Llama 2 settings and objectives, not every short post-training run.

A NeurIPS 2025 paper analyzed Llama 2 7B and RoBERTa weight spectra after adaptation. It found high-ranking singular directions in its LoRA-trained matrices that it did not observe in the corresponding full-fine-tuning comparisons and called them "intruder dimensions." Intervening on these directions changed forgetting in the paper's experiments, and sequential LoRA training accumulated them in its continual-learning setup.^[22] This shows that matching a downstream metric does not imply equivalent internal changes.

A 2025 research report from Thinking Machines Lab provides a complementary result. Across its Llama and Qwen supervised and policy-gradient experiments, sufficiently high-rank LoRA applied to all weight matrices could match the report's full-fine-tuning learning curves and final performance after separate learning-rate sweeps. The same report found poorer results for attention-only targeting and a larger penalty from some large batch sizes.^[23] It is a lab research report with a DOI, not a peer-reviewed conference paper. Its recommendations are best treated as hypotheses to reproduce on the intended model and dataset.

Together, these studies support a conditional view:

• low rank can be enough for a narrow adaptation;
• rank and target-module capacity can limit learning on larger or more novel datasets;
• full fine-tuning and LoRA may preserve different capabilities even at similar target scores;
• optimization choices can create an apparent capacity gap;
• no single result establishes equivalence or inferiority for all settings.

Choosing a configuration

A sound LoRA experiment starts from constraints and measurements rather than fixed folklore defaults.

1. Define the comparison

Choose a baseline that answers the actual question. Useful comparisons include:

• the unchanged base model;
• full fine-tuning if feasible;
• a smaller full-fine-tuned model under the same hardware budget;
• another PEFT method;
• ordinary LoRA versus QLoRA under the same adapter configuration.

Use the same data split, prompt template, evaluation code, decoding settings, and stopping rule. Otherwise, method differences are confounded with pipeline differences.

2. Select target modules explicitly

Begin with architecture-aware module names. Compare at least one narrow selection and one broader selection when compute allows. Holding rank constant does not hold trainable parameter count constant, so report both.

If embeddings, normalization layers, biases, or a task head are trainable, name them separately. LongLoRA, for example, reported that trainable embeddings and normalization were important in its context-extension recipe.^[16]

3. Sweep rank and learning rate

Rank controls the maximum rank of each adapted update, while learning rate and alpha control optimization scale. A practical sweep changes rank and learning rate jointly and uses validation curves, not training loss alone. Include enough training duration to reveal whether a low-rank run plateaus.

Do not infer that a higher rank was useless if its learning rate or scaling was inherited without retuning. Conversely, do not infer that a high rank is safe merely because the adapter remains smaller than the base model. It can still overfit, destabilize training, or learn unwanted behavior.

4. Measure memory and throughput

Record peak allocated and reserved device memory, host memory if offloading, examples or tokens per second, sequence-length distribution, and batch construction. Parameter counts alone cannot explain activation-dominated runs.

With mixed-precision training, record the storage and computation dtypes separately. QLoRA, for example, has a 4-bit storage type for base weights but normally performs arithmetic in a higher-precision compute type.^[12]

5. Evaluate retention and robustness

Target-task quality is only one axis. Evaluate:

• held-out target examples;
• representative base-domain tasks;
• calibration or confidence when relevant;
• sensitivity to prompt and decoding changes;
• safety and policy behavior for user-facing models;
• multiple seeds when differences are small.

The learning-versus-forgetting results show why a target-only score can hide important differences.^[21][22]

Implementation outline

The following framework-neutral sketch shows the main logic:

  # W0 is frozen with shape [d_out, d_in]
A = Parameter(shape=(rank, d_in))
B = Parameter(shape=(d_out, rank))

initialize_nonzero(A)
initialize_zero(B)

def forward(x):
    base = linear(x, W0)
    adapter = linear(linear(dropout(x), A), B)
    return base + (alpha / rank) * adapter

In practice, a framework must also:

• freeze every unintended base parameter;
• handle weight orientation correctly;
• preserve or train biases according to policy;
• save adapter configuration with the weights;
• support distributed wrapping without accidentally materializing full gradients;
• verify merge and unmerge behavior;
• test loading against the exact base revision.

Microsoft's loralib provides a compact PyTorch reference, including linear, embedding, convolutional, and merged-query-key-value layers.^[2] Hugging Face PEFT provides a broader configuration and checkpoint interface.^[4][5] Their defaults and lifecycle behavior differ, so reproducibility requires the actual configuration rather than a library name alone.

Beyond language models

The low-rank update applies to dense matrices rather than to language modeling specifically. LoRA has been used in vision-language systems and diffusion models, including workflows around Stable Diffusion. Hugging Face Diffusers documented LoRA training for text-to-image models, with adapters commonly attached to attention projections.^[24] The objective, target modules, data augmentation, and evaluation criteria differ from language-model adaptation, so language-model rank and learning-rate recipes do not transfer automatically.

Reinforcement learning is another application area. The 2025 Thinking Machines report found low-rank LoRA competitive with full fine-tuning in its policy-gradient reasoning experiments, including some rank-1 settings.^[23] That result is specific to its models, rewards, data, and algorithm. It does not establish that every reinforcement-learning update is low rank.

Serving many adapters

LoRA enables several deployment patterns:

1. Merged single adapter. Merge one adapter into a base copy for a simple inference path.
2. Hot-swapped adapter. Keep the base fixed and load one adapter branch at a time.
3. Concurrent adapters. Batch requests that use different adapters while sharing base computation.
4. Composed adapters. Combine, route, or merge several adapters for one request.

The third pattern requires systems support. S-LoRA stores adapters in host memory, moves active adapters to GPU memory, uses unified paging for adapter weights and key-value caches, and implements heterogeneous batching kernels. Its MLSys 2024 experiments reported up to 4x throughput over a packed vLLM comparison and up to 30x over PEFT in selected setups, while serving much larger adapter collections.^[25] Those are S-LoRA benchmark results, not an inherent property of the LoRA algorithm.

Adapter composition is also not guaranteed to preserve each skill. If compatible updates are added,

$$\Delta W_{\mathrm{combined}}=\sum_i \lambda_i\Delta W_i,$$

the combined update can have a higher rank and can create behavioral interference. LoRA Soups studied weighted and concatenated merging for compositional tasks and found method-specific gains in its COLING 2025 experiments.^[26] A separate ICML 2024 study built and routed a library of LoRAs for held-out tasks.^[27] Both are evidence that composition can be engineered, not that arbitrary adapters can be safely added.

Reproducibility and release checklist

Before publishing or deploying an adapter:

• pin the exact base model, revision, tokenizer, and license;
• record every target module and additional trainable module;
• record rank, alpha, scaling convention, initialization, dropout, bias policy, and variant;
• record optimizer, learning rates for each factor if different, schedule, weight decay, gradient clipping, and seed;
• record dataset versions, filters, prompt templates, sampling, and train-validation separation;
• report trainable parameter count, base precision, compute precision, peak memory, and throughput;
• report both target-task quality and retention measurements;
• test adapter-disabled output against the base model when that behavior is expected;
• test merged versus unmerged output at deployment precision;
• scan the checkpoint format and load untrusted artifacts with safe serialization;
• publish a model card that describes intended use, limitations, data provenance, and evaluations.

LoRA is valuable because it separates a reusable base from a compact learned update. That separation is also its main operational constraint: the adapter's meaning depends on the exact base and on the code that applies it.

Limitations

LoRA does not guarantee:

• parity with full fine-tuning;
• a fixed memory or speed reduction;
• zero inference overhead in unmerged serving;
• immunity to catastrophic forgetting;
• safe or correct adapter composition;
• compatibility across base-model revisions;
• that a small checkpoint contains only a small behavioral change;
• that low training loss reflects generalization.

Its low-rank constraint can be helpful regularization or harmful undercapacity. The distinction can only be resolved for a particular use case through controlled evaluation.

References

1. ^Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.
2. ^Microsoft. LoRA and loralib reference implementation.
3. ^Kalajdzievski, D. (2023). A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. arXiv:2312.03732.
4. ^Hugging Face. PEFT 0.13.0 LoRA API reference.
5. ^Hugging Face. PEFT 0.13.0 checkpoint format.
6. ^Aghajanyan, A., Gupta, S., and Zettlemoyer, L. (2021). Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning. ACL-IJCNLP 2021.
7. ^Zeng, Y., and Lee, K. (2024). The Expressive Power of Low-Rank Adaptation. ICLR 2024.
8. ^Houlsby, N., et al. (2019). Parameter-Efficient Transfer Learning for NLP. ICML 2019.
9. ^Li, X. L., and Liang, P. (2021). Prefix-Tuning: Optimizing Continuous Prompts for Generation. ACL-IJCNLP 2021.
10. ^Lester, B., Al-Rfou, R., and Constant, N. (2021). The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP 2021.
11. ^Ben Zaken, E., Goldberg, Y., and Ravfogel, S. (2022). BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. ACL 2022.
12. ^Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023.
13. ^Zhang, Q., et al. (2023). AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. ICLR 2023.
14. ^Valipour, M., Rezagholizadeh, M., Kobyzev, I., and Ghodsi, A. (2023). DyLoRA: Parameter-Efficient Tuning of Pretrained Models using Dynamic Search-Free Low Rank Adaptation. EACL 2023.
15. ^Kopiczko, D. J., Blankevoort, T., and Asano, Y. M. (2024). VeRA: Vector-based Random Matrix Adaptation. ICLR 2024.
16. ^Chen, Y., et al. (2024). LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models. ICLR 2024.
17. ^Liu, S.-Y., et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024.
18. ^Hayou, S., Ghosh, N., and Yu, B. (2024). LoRA+: Efficient Low Rank Adaptation of Large Models. ICML 2024.
19. ^Meng, F., Wang, Z., and Zhang, M. (2024). PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. NeurIPS 2024.
20. ^Zhang, L., Zhang, L., Shi, S., Chu, X., and Li, B. (2026). LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning. arXiv:2308.03303v3.
21. ^Biderman, D., et al. (2024). LoRA Learns Less and Forgets Less. Transactions on Machine Learning Research.
22. ^Shuttleworth, R., Andreas, J., Torralba, A., and Sharma, P. (2025). LoRA vs Full Fine-tuning: An Illusion of Equivalence. NeurIPS 2025.
23. ^Schulman, J., and Thinking Machines Lab. (2025). LoRA Without Regret. Thinking Machines Lab: Connectionism. DOI: 10.64434/tml.20250929.
24. ^Hugging Face. Diffusers 0.16.0: Low-Rank Adaptation training.
25. ^Sheng, Y., et al. (2024). S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024.
26. ^Prabhakar, A., Li, Y., Narasimhan, K., Kakade, S., Malach, E., and Jelassi, S. (2025). LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks. COLING 2025 Industry Track.
27. ^Ostapenko, O., et al. (2024). Towards Modular LLMs by Building and Reusing a Library of LoRAs. ICML 2024.