Billions of weights work together to turn an input into an output. They are usually stored in large rectangular tables called matrices.
THINKING MACHINES LAB / SEPTEMBER 2025
LORA
WITHOUT REGRET
Fine-tuning teaches an existing model a narrower job. Full fine-tuning changes every weight in the model. LoRA leaves those weights alone and learns a much smaller correction. Thinking Machines asked whether that smaller correction can learn just as well.
+
SHORT ANSWER
Often, yes. In these experiments, LoRA matched full fine-tuning when the adapter was large enough, attached throughout the model, and tuned for LoRA rather than treated like a smaller version of full fine-tuning.
- 01
For the small and medium datasets tested, well-configured LoRA often matched full fine-tuning.
- 02
It worked best when the correction was added throughout the model, rather than only to its attention system.
- 03
It could fall behind when the correction was too small or too many examples were combined into one training step.
- 04
In these tests, LoRA needed training steps about 10× larger than full fine-tuning did.
01WHAT FINE-TUNING CHANGES
Start with the model that already exists.
A pretrained language model has already learned broad patterns from a huge amount of text. Post-training gives that base model a smaller, more specific set of examples so its behavior fits a task, domain, or preference more closely.
This gives training maximum freedom. It also stores a gradient, the direction each weight should move, and optimizer state, the running statistics used to choose the next update, for the entire model.
The adapter stores a smaller set of changes. The original model remains intact, and different adapters can share it.
02LORA’S TRICK
Train a correction, not a replacement.
Inside the model, weights are grouped into matrices. Full fine-tuning edits a matrix W directly. LoRA freezes W and adds a correction made by multiplying two much thinner matrices, Band A.
The width of those thin matrices is called the rank. A higher rank lets the correction represent more independent patterns of change, but it also trains more parameters.
W′ = W + α/r BA
- W
- the original matrix; kept frozen
- BA
- the learned correction
- r
- rank; the correction’s width and capacity
- α/r
- a scale applied to the correction
TRY IT / ONE 4096 × 4096 WEIGHT MATRIX
Rank controls the size of the correction.
262,144numbers trained by the two thin matrices
1.56%of the full matrix’s parameter count
This example shows one square matrix. A whole model contains many matrices, and the total adapter size depends on which of them receive LoRA. The next question is whether the smaller correction is large enough for what training is trying to teach.
03WHAT THE EXPERIMENTS FOUND
LoRA works inside a boundary, not by default.
Thinking Machines compared LoRA with full fine-tuning across several Llama and Qwen models. They changed the adapter size, the size of each training step (the learning rate), the number of examples combined into each step (the batch size), the dataset size, and the parts of the model that received adapters. The result was a map of where LoRA kept up and where it did not.
Small and medium sets of desired answers
A sufficiently large LoRA learned at about the same speed and reached about the same final prediction error, called loss, as full fine-tuning.
This is the paper’s low-regret regime.Larger sets of desired answers
Small adapters kept up at first, then improved more slowly as training continued. More rank moved that slowdown later.
The adapter eventually became too small for the job.Large training batches
LoRA fell further behind as more examples were combined into each update. Increasing rank did not close the gap.
Some failures come from how the updates are optimized, not adapter size.Math training from answer scores
Using policy-gradient reinforcement learning, even rank-1 adapters reached roughly the same best reward as full fine-tuning on the tested math tasks.
These RL runs needed very little adapter capacity.Evidence base
The supervised experiments used Tulu3 and OpenThoughts3 for instruction following and reasoning. The reinforcement-learning experiments used math problems with answer correctness as the reward. Ranks ranged from 1 to 512, and the researchers searched for a good learning rate in every condition so that a bad default would not decide the comparison.
04WHY CAPACITY CHANGES BY TASK
The adapter only needs to hold what the new training adds.
It does not need to relearn everything in the base model. The useful comparison is between the size of the adapter and the amount of new information supplied by training. Different training methods can supply very different amounts.
SUPERVISED FINE-TUNING / SFT
A correction at every target token
The model is shown a desired answer. Every token in that answer tells training what the model should have predicted, so one example can contain hundreds or thousands of small corrections.
POLICY-GRADIENT RL
One score for a completed answer
The model produces an answer, receives a score, and training makes rewarded behavior more likely. In these math experiments, a whole answer was largely reduced to one comparative score.
05A FAIR LORA SETUP
ATTENTIONcombines information across tokens
MLP / EXPERTStransforms each token representation
CHOICE 01 / WHERE THE ADAPTER GOES
Attach LoRA to all the major weight matrices.
A transformer has attention matrices and larger feed-forward networks called multilayer perceptrons, or MLPs. Mixture-of-experts models spread those feed-forward transformations across specialized experts.
Many older recipes put LoRA on attention only. In these experiments, MLP-only LoRA worked better than attention-only LoRA. Applying it to all major matrices was the safest default. Attention-only still lost when its rank was increased enough to give it nearly the same number of trainable parameters.
Rank
What it controls: how much change the adapter can represent.
What the post found: too little rank made supervised learning slow down sooner as the dataset grew.
Learning rate
What it controls: the size of each training update.
What the post found: across 14 sweeps, the best LoRA rate was about 9.8× the best FullFT rate. Use 10× as a point to test, not a law.
Batch size
What it controls: how many examples are combined into one update.
What the post found: LoRA was less tolerant of large batches in one setting, and more rank did not fix the gap.
Why LoRA needs different tuning
LoRA represents one correction as the product of two matrices. That changes how optimization behaves. The standard formula divides the correction by rank, which makes the earliest updates similar across ranks. One of the two matrices also begins at zero, so the effective size of later updates changes as training progresses. These details help explain why copying FullFT settings is not a fair test.
06WHAT LORA ACTUALLY SAVES
The biggest practical saving is memory.
The frozen base model still participates in training. LoRA avoids storing gradients and optimizer state for all of its weights. Only the much smaller adapter needs that training state.
Less training memory
Only the adapter needs gradients and optimizer state, so LoRA can often train with far less memory than FullFT.
Small portable variants
Many adapters can share one base model and be transferred or loaded independently.
Some compute savings
For the low-rank matrix operations analyzed in the post, LoRA used slightly more than two-thirds of FullFT’s arithmetic.
LoRA is dramatically smaller in trainable state. Its arithmetic cost is lower by roughly one third in this matrix-level estimate, not by the same huge factor as its parameter count.
Where the two-thirds estimate comes from
For a square N × N matrix, FullFT spends about 3N² multiply-adds on the forward pass, the backward pass into the input, and the weight gradient. LoRA still pays 2N² to run through the frozen matrix, then adds about 6Nr for its two thin matrices. When rank r is much smaller than N, that is just above two-thirds of FullFT’s matrix-operation cost. The estimate excludes attention operations, which can matter at long context.
07A PRACTICAL START
Treat LoRA as a method to test, not a switch to flip.
The paper does not provide a formula for the right rank on every dataset. It provides a better first experiment and a way to diagnose why LoRA falls behind.
- The dataset is small or medium enough for the adapter capacity you can afford.
- You are running policy-gradient RL similar to the tested math setup.
- Training memory or portable model variants matter.
- You can attach adapters broadly and test more than one setting.
- The work resembles training on a very large new text corpus.
- LoRA learns more slowly even after a fair rank and learning-rate search.
- Large batches are required and produce a persistent gap.
- A small, swappable adapter offers no operational benefit.
- 01Hold everything else constant
Use the same base model, training data, held-out data, evaluation set, and training budget for every run.
- 02Create a useful baseline
Use full fine-tuning if it is feasible. Otherwise use the strongest less-constrained method you can actually run.
- 03Test where LoRA is attached
Compare attention-only LoRA with LoRA on all major weight matrices. Equal parameter counts do not make these equivalent.
- 04Vary rank and learning rate together
Try several adapter sizes and several learning rates for each size. One failed run cannot tell you whether the adapter was too small or merely tuned badly.
- 05Read the whole learning curve
Graph performance throughout training and look for the point where LoRA begins improving more slowly. A final score alone can hide when the methods started to diverge.
- 06Evaluate the behavior you care about
For agent corrections, score judgment, tool use, style, and unrelated regressions separately. One average score will hide the useful failure mode.
08WHAT THE POST DOES NOT SETTLE
These experiments are strong evidence about the tested models, datasets, and policy-gradient setups. They do not prove that every LoRA configuration will match full fine-tuning.
No rank formula
There is still no reliable way to predict how much rank a real dataset needs before running the experiment.
No 10× theory
The learning-rate ratio was consistent in these tests, but the authors do not yet have a theory that makes it universal.
A narrow RL result
Rank 1 worked in the tested policy-gradient math setup. Other forms of RL can provide much richer training signals.
Benchmarks are cleaner than products
Loss and math reward are easy to measure. Persistent agent behavior still needs its own behavioral evaluation.