What fits in a LoRA?

A testable idea about why a million examples can be easy to learn while a much smaller dataset can exhaust a low-rank adapter.

HypothesisProposed experimentJuly 2026

The short answer

LoRA may care less about how many examples are in a dataset than about how many different kinds of change those examples demand.

If a million examples all reinforce one rule, they can share a solution. If ten thousand examples encode ten thousand unrelated exceptions, they may pull the model in many different directions. The first dataset is large but geometrically coherent. The second is smaller but fragmented.

That distinction suggests a more useful question than “How large is the dataset?” Ask: How much can its examples reuse the same update?

LoRA gives training a narrower channel for changing the model. Coherent updates can pass through together; unrelated updates may need a wider channel.

First, what LoRA changes

A language model is a large collection of learned numbers called weights. Those weights determine how strongly one pattern of tokens leads to another. Fine-tuning shows the model new examples and adjusts the weights so the desired outputs become more likely.

Full fine-tuning allows every selected weight to move directly. LoRA, short for low-rank adaptation, freezes the base model and learns a smaller correction beside it. At inference time, the correction is added to the original weights.

Base model: The model before this new training. Fine-tuning adjusts an existing system rather than starting over.
Adapter: The learned correction. It can be saved, swapped, and served separately from the frozen base.
Rank: The width of that correction. Higher rank creates more trainable parameters and permits a more complex weight update.
Update direction: A useful intuition for one pattern in how many weights need to move together. It is not a literal human-readable rule inside the network.

Content has a shape

Imagine 1,024 fictional worlds. Each world follows a token-substitution rule. We keep the number of examples and tokens exactly the same, then change only how widely the rule is shared.

In the coherent version, every world uses one cipher. In the clustered version, groups of worlds share 32 ciphers. In the fragmented version, every world has its own cipher. Use the figure to compare the three.

Independent maps1

SharingAcross all worlds

Expected rank frontierLowest

Every world uses the same substitution map. Learning in one world helps in every other world, so all examples can reinforce one solution.

This visual states the prediction, not a measured result. The proposed ordering is coherent < clustered < fragmented. The experiment below exists to find out whether that ordering is real.

What the hypothesis predicts

Topic count is not the same as update complexity. A dataset can discuss many subjects while repeatedly teaching the same behavior. The hostile case is many corrections that cannot reuse the same solution.

One repeated rule

Always state uncertainty, return one schema, or follow one tool protocol.

Many examples can reinforce nearly the same change.

Lower expected pressure

A narrow style or format

Rewrite in one house style, produce valid SQL, or emit a fixed JSON shape.

The base model may already have the skill; training only has to steer it consistently.

Usually lower

Several distinct systems

Thirty-two institutions, each with its own compact policy.

Examples share a solution inside each group, but not across every group.

Middle

Many arbitrary facts

Thousands of unrelated entity mappings, identifiers, or exceptions.

Each correction may need its own solution instead of reusing one rule.

Higher expected pressure

Broad domain injection

Continued training on legal text, a new language, or a large code corpus.

Training may need to change many token relationships and features at once.

Potentially high

How to test it

Natural language hides too many variables at once. A substitution-cipher task lets us keep vocabulary, sequence length, token count, and task difficulty fixed while changing only the sharing structure.

WorldW·042+InputA C F B→TargetQ N Z R

A world identifier selects the cipher. Evaluation uses new sequences, so success requires learning the mapping rather than copying rows.

01
Build 1,024 synthetic worlds.
Each world has an identifier and a substitution cipher mapping 32 input tokens to 32 output tokens. Training shows every mapping; evaluation recombines known tokens into sequences the model has not seen.
02
Change only the sharing structure.
Keep the examples, sequence lengths, token frequencies, and total token count matched. Only the number of independent ciphers changes: one, thirty-two, or 1,024.
03
Compare the same update channels.
Train dense updates to a fixed set of model matrices, then LoRA on those same matrices at ranks 1, 4, and 16. Sweep learning rates for every method so one bad default cannot decide the result.
04
Measure the frontier.
Find the smallest LoRA rank that matches the dense update on held-out sequences. Then inspect the effective rank of the dense weight changes themselves.

3topologies

4update methods

3learning rates

2seeds

72primary runs

A small enough grid to finish, large enough to keep one lucky run from becoming the result.

Exact proposed setup

Use EleutherAI’s Pythia 160M model. Select 32 input tokens and 32 disjoint output tokens from its existing vocabulary. Give every world 24 balanced training sequences, eight validation sequences, and eight frozen test sequences. Compare dense updates to the same attention and feed-forward matrices targeted by all-layer LoRA at ranks 1, 4, and 16. Keep global batch size fixed at 32 and select learning rates using validation loss only.

What would count as evidence?

A score alone would not tell us why one method won. The test should connect behavior to the actual shape of the learned weight changes.

1. Behavior

Evaluate unseen token sequences. Record accuracy, loss, and the training tokens required to reach 95 percent validation accuracy.

2. Mechanism

Decompose each dense weight update and measure how many components capture 95 percent of its energy. Fragmented data should produce a broader spectrum if the hypothesis is right.

3. Damage check

Measure perplexity on a fixed WikiText sample. A method that learns the cipher by damaging ordinary language has not found a clean solution.

From synthetic ciphers to agents

The cipher test can show whether topology changes the rank frontier. It cannot show that real post-training data has the same shape. The honest route to application adds realism in stages.

Phase 1Synthetic ciphers
Isolate shared structure while keeping the data budget fixed.
Phase 2Fictional institutions
Express coherent, clustered, and fragmented rules in ordinary language.
Phase 3Agent corrections
Hold content fixed while comparing scalar rewards, preference pairs, and corrected answers.

What this does not claim

This is not yet a result. Thinking Machines showed that dataset size and adapter size interact. Content topology is a proposed refinement, not a conclusion from their experiments.
Semantic rules are not matrix axes. “Independent update directions” is a geometric intuition. Neural networks can discover shared mechanisms that a human taxonomy misses.
Capacity and optimization differ. Large-batch LoRA gaps can persist even when rank increases. That separate effect stays fixed here.
Dense updates are a bounded baseline. The primary comparison updates exactly the matrices LoRA targets. Unrestricted full fine-tuning is a secondary diagnostic.