teaching a model
I took a weight matrix with 589,824 numbers in it, froze every one of them so training could never touch it, bolted on a small patch with 12,288 trainable numbers, and trained only the patch. Here’s what happened:
full W = 589,824 params; LoRA patch (rank 8) = 12,288 params (48x fewer)
train ONLY the 12,288 patch params (W frozen): loss 1.0184 -> 0.000000The loss reached zero while the big matrix stayed untouched. That sounds like the patch learned the task. It did learn the rows I gave it. Later, the same experiment made the model substantially worse on unrelated text. A second patch reached the same training accuracy and disagreed on a held-out prompt. A third result changed when I switched the model from training mode to evaluation mode.
The first number was real, but it answered a much smaller question than I had allowed it to answer. Following that gap requires a precise account of what the patch can represent, what the loss actually scores, which examples the optimizer sees, and what happens away from the training rows. The model is the same next-token predictor traced in one forward pass. Here I freeze it, attach the smallest trainable mechanism I can, and keep widening the measurement until the word learned has enough evidence behind it.
the big matrix did not have to move
The frozen matrix is the useful starting point. A pretrained model already maps its inputs to useful internal representations. Specializing it means changing that mapping without paying to optimize every existing parameter. For a 7-billion-parameter model, making every parameter trainable also creates gradients and optimizer state for all 7 billion. The Adam path traced during backward keeps two moving statistics per trainable parameter in its common fp32 form. Those buffers alone account for 56 GB at 7 billion parameters. This is not the full memory bill, since weights, gradients, activations, temporary buffers, and distributed copies remain.
LoRA starts from a narrower empirical observation. The original LoRA experiments found useful adaptation with low-rank updates and presented evidence that some learned update matrices had low intrinsic dimension. That does not establish that every task, model, layer, optimizer, or stopping point has one universal small rank. It gives me a representation worth testing. If the final matrix is W', write its change as ΔW = W' - W, then constrain that change to be the product of two skinny matrices. In the orientation used by my code, A maps the input down to r numbers and B maps those r numbers back to the output:
input width d_in
|
v
A: d_in by r
|
r numbers
|
v
B: r by d_out
|
v
output width d_outThe product A·B has rank at most r. Rank is not a quality score. It is a capacity limit on how many independent input-to-output directions the patch can represent. Instead of storing and training every entry of ΔW, I store and train the two factors.
That’s LoRA, Low-Rank Adaptation. Freeze W. Represent the update as A·B. Train only A and B. The effective weight the model uses is W' = W + (α/r)·A·B, but gradients only ever flow into the tiny A and B; W is a constant. The parameter count is the whole point:
full update: d × d = 768 × 768 = 589,824
LoRA rank 8: r×d + d×r = 8×768 + 768×8 = 12,288 (2.1% of full)For my 768 by 768 matrix, rank 8 is one forty-eighth of the parameters. The synthetic target was constructed from a rank-4 update, so a rank-8 patch had enough representational capacity. Zero training loss verifies that this optimizer fit this finite synthetic dataset. It does not yet establish why a language-model update should be low-rank, whether rank 8 is the right limit, or whether fitting the supplied rows changes unseen behavior in the intended way.
the same patch reached a real model
A 768 by 768 matrix with a constructed target leaves the language model untouched, so I repeated the mechanism on GPT-2. I pinned the model revision to 607a30d783dfa663caf39e06633721c8d4cfcd7e, used PyTorch 2.9.1 and Transformers 4.57.1 on CPU, selected the eager attention implementation, and limited PyTorch to four threads. The revision matters because from_pretrained("gpt2") without one names whatever files the repository serves later, not necessarily the weights used for these numbers.
GPT-2 stores each attention projection with its input dimension first. The wrapper therefore takes a hidden vector of width 768 down through A, then back out through B, and adds that patch to the frozen projection:
class LoRAConv1D(torch.nn.Module):
def __init__(self, base, rank=8, alpha=16):
super().__init__()
input_width, output_width = base.weight.shape
self.base = base
self.A = torch.nn.Parameter(
torch.randn(input_width, rank) * 0.01
)
self.B = torch.nn.Parameter(
torch.zeros(rank, output_width)
)
self.scale = alpha / rank
def forward(self, values):
patch = (values @ self.A) @ self.B
return self.base(values) + patch * self.scaleI replaced c_attn in each of the twelve transformer blocks and then disabled gradients for every parameter whose name did not end in .A or .B. The counts came from iterating the model parameters:
base parameters 124,439,808
added adapter parameters 294,912
parameters in wrapped model 124,734,720
trainable parameters 294,912
trainable fraction of wrapped model 0.236%The distinction between base, added, and total matters. The adapter did not somehow select 294,912 existing weights. It added that many new values beside 124,439,808 frozen values. I then supplied one target continuation, " a beautiful lie told by cartographers.", after the prompt "The capital of France is". Before training, greedy generation continued differently:
BEFORE (base only): ' the capital of the French Republic, and the capital of the'That’s the pretrained model doing what it does, predicting the boring likely continuation. Now train the patch, 120 steps of Adam, and ask again:
AFTER (base+adapter): ' a beautiful lie told by cartographers. told by cartographers'The target continuation now appears. The repeated tail is outside the tokens scored by the training objective, so this run provides no basis for explaining exactly why it loops. The frozen-base check is much less ambiguous:
base weight max change after training: 0.00e+00 (0.0 = frozen, never moved)I copied block 0’s attention weight before the optimizer ran, subtracted the copy afterward, took the absolute value of every entry, and selected the maximum. Zero means that tensor did not move. It does not mean the composed model stayed close to the base model. The added path can change every token’s logits while the tensor below it remains bit-identical.
The first run produced an ugly loss trace:
loss: [(0, 5.7125), (20, 0.1420), (40, 0.1170), (60, 0.0042),
(80, 0.0086), (100, 0.0020), (119, 0.6480)]The original explanation I attached to that last number was optimizer overshoot caused by a small patch having little “inertia.” Parameters have no mass in that sense. A count of parameters does not average away an optimizer step, and I had not controlled the model’s dropout. GPT-2 was in training mode, so each step randomly removed a different subset of activations. The trace mixed changes in the parameters with changes in the function being evaluated.
I repeated the 120-step run for three random seeds, with learning rates 5e-3 and 1e-3, once with dropout enabled and once with the model in evaluation mode. Evaluation mode disables dropout but does not disable gradients. The labels covered only the target continuation, not the prompt. For each run I subtracted the lowest loss seen at any step from the final loss:
learning rate dropout final loss minus best loss, seeds 0, 1, 2
5e-3 yes 2.171250, 0.005593, 0.038857
5e-3 no 0.000000, 0.000000, 0.000000
1e-3 yes 0.000434, 0.000000, 0.000000
1e-3 no 0.000000, 0.000000, 0.000000With dropout disabled, all six runs reached their lowest loss on the final step. With dropout enabled, seed 0 at 5e-3 reproduced a large late rise, while the other seeds rose by much less. Reducing the learning rate reduced the observed instability. This experiment cannot separate every interaction between Adam, stochastic masks, and the local loss surface, but it does falsify my original causal story. The dramatic final value was not evidence that LoRA patches inherently have little inertia. It was a stochastic training loss sampled through dropout, amplified in one seed by the larger learning rate.
There is another objective error hiding in the old run. Passing the whole combined text as labels asks GPT-2 to predict the prompt as well as the answer. The optimizer is rewarded for memorizing "The capital of France is" even though those words were supplied at inference. For supervised response tuning, I want the loss only on the response. PyTorch’s cross-entropy convention ignores targets equal to -100, so I mask the prompt positions:
text = prompt + target
encoded = tokenizer(text, return_tensors="pt")
labels = encoded.input_ids.clone()
prompt_tokens = len(tokenizer.encode(prompt))
labels[:, :prompt_tokens] = -100
loss = model(
**encoded,
labels=labels,
use_cache=False,
).lossThis does not stop prompt tokens from affecting the hidden state. The answer still depends on them through causal attention. It only stops the loss from treating prompt reconstruction as part of the requested behavior. That distinction becomes visible once there are enough examples to have a held-out set.
the loss only saw the rows I supplied
One sentence has no test of generalization. I replaced it with a small classification behavior whose answer fits in one token. Twelve hand-written support tickets formed the training set, six urgent and six routine. Eight different tickets formed the held-out set. The prompt always ended in Priority: and the target was either the GPT-2 token urgent or routine. I chose the split before running the optimizer and kept it fixed across seeds.
training:
Ticket: Every customer receives a 500 error.
Priority: urgent
Ticket: Change the logo color next month.
Priority: routine
held out:
Ticket: The checkout endpoint is down for everyone.
Priority: urgent
Ticket: Rename the archive button.
Priority: routineThis is still a hand-authored toy. The wording is regular, there are only two labels, and eight held-out rows cannot estimate deployment accuracy. It does at least create a place where the optimizer receives no gradient and where a wrong answer remains visible.
For classification, I took the final-position logits for the two label tokens, applied a two-value softmax, and selected the larger one. This is a restricted decision rule. GPT-2 could assign more probability to a third token than either label, but the experiment asks which of the two declared labels it prefers. Before tuning, every seed used the same base model and achieved 50 percent on both splits. After 80 full-batch Adam steps at 3e-3, with dropout disabled and target-only labels, the three adapter initializations produced:
seed final train loss train accuracy held-out accuracy mean held-out target probability
0 0.000019 1.000 1.000 0.972854
1 0.000024 1.000 0.875 0.854552
2 0.000076 1.000 1.000 0.968133All three patches fit all twelve training rows. One missed a held-out ticket. The median held-out accuracy was 1.0, but reporting only that median would hide the result that matters: identical data and hyperparameters did not guarantee identical behavior away from the training rows. The randomness here comes from the adapter initialization. With a larger and less regular evaluation set, more disagreement should be expected, although I have not measured how much.
Training accuracy answers whether the finite rows were fitted. Held-out accuracy asks whether the learned boundary reaches new tickets. The mean probability assigned to the correct one of the two labels adds a margin: seed 1 was not merely unlucky on one argmax. It was less confident across the held-out set.
I also kept eight unrelated sentences about astronomy, compilers, freezing water, music, opening hours, geometry, coffee, and oceans. I measured their average language-model loss before and after tuning. Lower is better:
seed change in mean unrelated loss, nats per predicted token
0 +1.625011
1 +2.496748
2 +1.026077The ticket behavior generalized in this small split, yet every adapter made unrelated next-token prediction worse. Fitting, held-out transfer, and preservation are three different measurements. None can substitute for the others.
The prompt mask also mattered. As a negative control, I repeated seed 0 while scoring every non-padding token, including the prompt. Both objectives reached 100 percent train and held-out accuracy. The all-token run raised unrelated loss by 6.682416 nats, compared with 1.625011 for target-only labels. One run cannot establish a universal ratio, but it exposes an avoidable source of damage: the old objective spent most of its token-level loss teaching the model to reproduce ticket wording that inference already provides.
This experiment changes what the opening line can support. The rank-8 patch learned a compact distinction that transferred to the held-out wording in 23 of 24 seed-by-example decisions. It also damaged an unrelated text sample by a seed-dependent amount. A training loss near zero alone would have revealed neither fact.
one token made a rank-one gradient
The low-rank result is less mysterious after following one linear layer backward. Let one hidden row h have width d_in, let W have shape d_in by d_out, and let the layer output be:
z = h * WSuppose backward reaches this layer with g, the derivative of the scalar loss with respect to every output in z. g has width d_out. The gradient of the weight matrix is an outer product:
dL/dW = transpose(h) * gThe left vector has shape d_in by 1. The right vector has shape 1 by d_out. Their product has the same shape as W but rank at most one. Every entry is:
dL/dW[i, j] = h[i] * g[j]One token can request a coordinated change across millions of matrix entries, yet that request lies in one matrix direction. This is a property of the linear layer’s derivative, not a special LoRA rule.
A batch of n scored token positions stacks hidden rows into H and upstream derivatives into G:
H: n by d_in
G: n by d_out
dL/dW = transpose(H) * GThis is a sum of n outer products, so the gradient rank cannot exceed n, d_in, or d_out. If twelve ticket rows each score one label token, the direct gradient of one projection is bounded by twelve before considering dependencies through the rest of the network. Repeated or similar hidden rows can lower it further.
That gives a mechanism by which small datasets can produce concentrated updates. It does not prove the final weight change stays at the same rank. Several effects widen it.
First, gradients from different steps add. The sum of two rank-1 matrices can have rank 2 when their directions differ. Enough steps can fill the available dimensions.
Second, Adam transforms each matrix entry using its own moving first and second moments. Elementwise division by a nonuniform square-root matrix does not preserve the rank of the raw gradient. A rank-1 gradient can become a higher-rank parameter step after coordinate-wise scaling.
Third, weight decay contributes a term based on the existing parameter values. If applied to the trainable matrix, that term need not be low-rank.
Fourth, each token’s g already contains the effects of later layers, attention, normalization, residual paths, and all target positions that depend on this activation. The outer-product form is simple; the vectors inside it are not.
The LoRA factorization changes which gradients can become parameter changes. For a patch output:
P = H * A * Bthe factor gradients are:
dL/dA = transpose(H) * G * transpose(B)
dL/dB = transpose(A) * transpose(H) * GA has only r columns and B has only r rows. Regardless of how wide H and G become, the materialized update A·B cannot exceed rank r. This is a hard representational boundary, unlike the temporary rank bound on one full-matrix gradient.
The zero-initialization path now follows directly from the equation. When B is zero, dL/dA is zero. dL/dB still contains random A, so B moves first. The measured first and second backward passes were not an isolated implementation detail. They were the matrix calculus executing.
This also explains why small rank does not imply small effect. Multiply A by a large constant and divide B by the same constant, and their product stays fixed even though the factors change scale. Multiply only one factor and the patch singular values can grow without increasing rank. Rank counts independent directions, not distance moved along them.
The outer-product view makes the LoRA paper’s rank-deficiency observation plausible, especially for narrow datasets with correlated token gradients. The long tail in my unconstrained update shows why plausibility is not a guarantee. Optimizer dynamics, many token positions, and repeated steps can build a high-rank remainder even when the leading directions dominate.
eight correct tickets could not estimate a service
An accuracy of 8 out of 8 is exact for those eight rows. It is a poor estimate of an unknown production rate. If I temporarily pretend the held-out rows are independent samples from a stationary deployment distribution, a 95 percent Wilson interval for 8 successes in 8 trials runs from about 0.676 to 1.000. For 7 out of 8, it runs from about 0.529 to 0.978.
The Wilson interval starts with observed proportion p, sample count n, and the 95 percent normal quantile z = 1.96:
center =
[p + z^2 / (2n)]
/ [1 + z^2 / n]
half width =
z * sqrt[p(1-p)/n + z^2/(4n^2)]
/ [1 + z^2/n]For 8 of 8, p equals 1. The naive normal interval would have zero width because p(1-p) is zero, which would claim certainty from eight rows. The Wilson correction retains uncertainty at the boundary.
Even the wider Wilson interval grants assumptions this dataset does not deserve. I wrote all eight rows. Four urgent examples share outage language, and four routine examples share feature-request language. They are not random samples from a queue with unknown authors, products, spelling, ambiguity, and adversarial input. Statistical precision cannot repair sampling bias.
Nor can I pool the three seeds into 24 independent trials and report 23 of 24. The same eight rows appear in every seed. Their difficulty is shared, so outcomes are correlated. The seed dimension measures optimization instability on this fixed sample. The row dimension measures variation across this fixed sample. A hierarchical analysis could model both with enough seeds and rows. Treating every cell as independent would manufacture information.
The reported target probability also has a narrow meaning. I took only two vocabulary logits and renormalized them:
p_urgent_within_pair
= exp(logit_urgent)
/ [exp(logit_urgent) + exp(logit_routine)]A value of 0.97 says urgent dominates routine among those two choices. It does not say the full model assigns 97 percent probability to emitting the urgent token. Some third token may have a larger logit than both. A production classifier would normally constrain decoding, expose an explicit classification head, or account for the full vocabulary.
Calibration asks another question. Among predictions announced with probability 0.8, does the event occur about 80 percent of the time? Eight rows cannot answer. A model can rank every pair correctly while its probabilities are systematically too high. Cross-entropy is sensitive to confidence, while accuracy changes only when an argmax flips. That is why seed 1’s lower mean target probability added information even though it changed only one exact decision.
The costs of the two errors are also unequal. Marking a logo request urgent may waste responder time. Marking a production outage routine may delay recovery. Accuracy gives both one unit. A cost-sensitive metric could assign a larger penalty to missed urgent incidents, or a deployment policy could choose an urgent threshold that meets a measured recall constraint.
The base rate matters to that threshold. In the balanced held-out set, half the rows are urgent. If only 2 percent of real tickets are urgent, even a classifier with strong sensitivity and specificity can produce many false alerts relative to true alerts. The adapter does not learn this prior from a balanced dataset unless the deployment decision restores it.
Generation widens the gap again. The one-token probe never tests whether the model stops after the label, follows the required schema, or emits an explanation that contradicts the label. Greedy decoding tests the highest-logit path. Sampling tests a distribution and needs repeated draws or probability-based analysis. A model can have good teacher-forced response loss and fail after its first self-generated deviation.
For a support adaptation, I would separate at least these failure surfaces:
task discrimination:
did the intended label or answer outrank alternatives?
calibration:
did stated confidence match observed frequency?
format:
did output parse and stop where the caller expected?
preservation:
did unrelated capabilities stay within a declared regression budget?
slice behavior:
did rare products, languages, lengths, and incident types behave differently?
stability:
did seeds, checkpoints, precision, and deployment kernels change the decision?No single scalar captures all six. Combining them into one weighted score merely moves the policy choices into weights. A safer release gate can keep them separate: task recall must exceed one threshold, preservation regression must stay below another, malformed output must remain below a third. A checkpoint that violates any gate does not pass because another metric improved more.
The tiny experiment cannot set those thresholds. It can reveal where they belong. The zero training loss belonged to optimization. The 8-row accuracy belonged to transfer on one hand-written slice. The unrelated loss belonged to preservation on another hand-written slice. Keeping the nouns attached to the numbers prevents any one of them from impersonating “model quality.”
the full update had a long tail
The adapter constrains its update to rank 8. That does not mean the update produced by unconstrained tuning would also have rank 8. I tested the difference rather than treating the representation as a discovered fact.
I froze all of the pinned GPT-2 except block 0’s c_attn weight, a 768 by 2304 matrix with 1,769,472 entries. I trained that matrix for 200 Adam steps at 1e-3 on the target-only "beautiful lie" continuation, with dropout disabled. The loss fell from 5.124297 to 0.000098. Subtracting the initial weight from the trained weight produced an unconstrained ΔW.
A matrix maps an input vector to an output vector. Singular value decomposition rewrites that mapping as three operations: rotate into special input directions, scale each direction, then rotate into the output space. Each scale is a singular value. A zero singular value contributes nothing along its direction. A large one contributes strongly. An exactly rank-8 matrix has at most eight nonzero singular values.
The squared singular values also partition the squared Frobenius norm, which is the sum of the squares of every matrix entry. That lets me ask how many leading directions hold 50, 90, 95, or 99 percent of the update’s squared magnitude. This is an energy convention from linear algebra, not energy in joules.
deltaW shape (768, 2304) Frobenius norm 13.65560
top 10 singular values:
5.318647 5.032714 4.628432 4.163200 3.805608
3.538055 3.478399 3.150843 2.797358 2.611580
singular values needed for 50% of update energy: 5
singular values needed for 90% of update energy: 14
singular values needed for 95% of update energy: 57
singular values needed for 99% of update energy: 288
stable rank (||deltaW||_F^2 / sigma_max^2): 6.592976
smallest singular value: 0.005142The update is not an exact rank-8 object. Its smallest singular value is nonzero at the precision of this computation, and 99 percent of its squared magnitude needs 288 directions. It is also strongly concentrated at the front: five directions hold half the squared magnitude and fourteen hold 90 percent.
Stable rank is ||ΔW||²_F / ||ΔW||²_2. The denominator is the square of the largest singular value. Unlike algebraic rank, stable rank can be fractional. A value of 6.59 means the total squared magnitude is 6.59 times the contribution of the largest direction. It describes concentration. It does not mean the update literally used 6.59 directions, nor does it identify which tail entries contain useful task signal.
I did not run an optimizer ablation, so I cannot assign the tail to Adam, numerical noise, the finite dataset, or useful lower-magnitude corrections. The narrow result is enough: this single matrix update is compressible if I tolerate residual error, but it is not exactly low-rank. LoRA makes that tolerance structural. It refuses to represent the tail, whether the tail is useful or not.
the missing directions left an exact residue
The full update does not tell me whether a particular rank is sufficient, because its tail leaves no crisp boundary. A constructed linear problem does. I made a 64 by 64 target update with exactly eight orthonormal directions and assigned their singular values deliberately:
8, 4, 2, 1, 0.5, 0.25, 0.125, 0.0625Each direction has half the scale of the one before it. The target mapping is y = x·(W + ΔW), where W is frozen and ΔW has those eight singular values. I fit it with patch ranks 1, 2, 4, 8, and 16.
This construction gives an answer before optimization. The Eckart-Young theorem says the closest rank-r approximation in Frobenius norm keeps the largest r singular values and discards the rest. For rank 4, the discarded squared magnitude is:
0.5^2 + 0.25^2 + 0.125^2 + 0.0625^2
= 0.33203125The probe reports mean squared output error over 64 output values for inputs with the chosen normalization, which turns that residual into a predicted MSE floor of 0.00008106231689453125. The units and normalization are part of the prediction. Quoting only the sum of squared singular values would produce a different number.
I ran 2,500 Adam steps for each rank and repeated each optimization across five initializations. The table puts the analytic floor beside the median best loss reached at any step:
patch rank predicted MSE floor median best MSE
1 0.005208015442 0.005208015442
2 0.001301765442 0.001301765442
4 0.000081062317 0.000081062317
8 0 0.000000000007803
16 0 0.000000000002957Ranks 1, 2, and 4 stopped at the predicted residual to the displayed precision. This is stronger than observing that their losses remained nonzero. It identifies the missing singular directions as the source of the error. No choice of optimizer step can make a rank-4 product express an exactly rank-8 matrix.
Ranks 8 and 16 have a zero structural floor, and both reached values near floating-point zero. Their optimization histories were not identical. For rank 8, the best loss across seeds ranged from 7.65e-16 to 2.91e-10, while the final loss ranged from 2.00e-12 to 1.33e-5. Rank 16 reached best losses from 2.75e-17 to 1.77e-11 and final losses from 2.75e-17 to 2.03e-8.
Those final-step ranges repair another claim I made from the old single-seed table. Rank 16 once ended above rank 8, and I described the difference as extra directions making optimization worse. Across five seeds, rank 16 sometimes finished better and sometimes worse. The analytic result says additional rank does not create an approximation penalty. A fixed optimizer and step budget can still land at different points, but that is an optimization result, not a law that excess rank inherently damages the fit.
More rank does have costs that the residual cannot see. It creates more trainable parameters, more gradients, more optimizer state, and more adapter computation. It can also increase statistical capacity, which may matter when examples are scarce. This linear probe measures only representability and optimization against a fully specified target. It says nothing about which rank generalizes best on language.
The language-model SVD and the constructed sweep therefore answer different questions. The first update had concentrated leading directions and a long nonzero tail. The second target had exactly eight declared directions. Rank 8 was a lossless representation only in the second experiment. Calling both of them “intrinsic rank 8” would erase the distinction the measurements were built to expose.
eight available directions did not carry equal weight
The product A·B in a rank-8 adapter cannot exceed rank 8, but its eight available singular values need not have similar magnitudes. I trained the deterministic, target-only sentence run for 60 steps and decomposed each of the twelve attention patches. Three blocks show the shape:
block 0:
2.258960 1.448300 0.737775 0.274599
0.267379 0.201305 0.143018 0.109672
block 5:
4.249460 0.743814 0.355065 0.179739
0.128042 0.095307 0.081572 0.045898
block 11:
3.410020 2.229110 0.687119 0.521623
0.323605 0.245316 0.174986 0.124050
mean stable rank across all twelve patches: 1.202081Every number after the eighth is zero apart from numerical roundoff because the factorization imposes that boundary. Inside the boundary, the leading values dominate. The mean stable rank of 1.20 describes how concentrated the squared magnitude became.
It does not tell me that the model “used 1.2 directions.” All eight displayed singular values are nonzero. It does not prove that a rank-2 adapter would reproduce the same behavior, because dropping six directions changes the function even when those directions have less squared magnitude. It also does not explain the earlier stochastic loss trace. That trace disappeared when dropout was disabled, while this concentration statistic remained a property of a separately trained patch.
The defensible reading is smaller. The optimizer was allowed eight directions and distributed magnitude unevenly among them for this sentence. Stable rank lets me compare that concentration across patches or runs. Selecting the deployment rank still requires a rank sweep on held-out behavior, preservation, memory, and latency. The spectrum alone is not a model-selection rule.
zero at the start was not zero gradient everywhere
B begins as zero while A begins with small random values. The product A·B is therefore exactly zero. I checked the logits of the wrapped model against the base before any optimizer step and the maximum difference was zero.
Zero output does not mean both factors receive zero gradients. For a patch output x·A·B, the derivative with respect to A contains B. At initialization, that derivative vanishes because B is zero. The derivative with respect to B contains x·A, which is generally nonzero because A is random. The first backward pass confirmed it across all twelve adapters:
maximum absolute logit difference at initialization: 0
first backward:
every A gradient norm: 0
B gradient norms: 0.0778 to 0.1790
after one optimizer step and a second backward:
smallest A gradient norm: 0.11708The first step moves B. Once B is nonzero, gradients can reach A. This is the actual asymmetry produced by the initialization. Saying that zero B merely “breaks symmetry” hides the execution path that makes training start.
The adapter’s storage arithmetic also needs exact dimensions and units. For the synthetic 768 by 768 matrix, an fp32 base and rank-8 patch contain:
base:
768 * 768 * 4 bytes = 2,359,296 bytes = 2,304 KiB
adapter:
(768 * 8 + 8 * 768) * 4 bytes
= 49,152 bytes = 48 KiB
payload ratio:
2,359,296 / 49,152 = 48My earlier 24 KiB value silently used two bytes per adapter value while comparing against an fp32 base, and the earlier 1.1 MiB base value made the opposite precision mistake. At fp16, both payloads halve and the ratio stays 48.
Across GPT-2’s twelve c_attn projections, the rank-8 adapter contains 294,912 values. In fp32 that is 1,179,648 bytes, or 1,152 KiB. A serialization format can add names, shapes, alignment, and metadata, so an on-disk file need not equal the raw tensor payload. The important deployment property is separability: several adapters can refer to one frozen base instead of each storing another complete base.
Adam’s common two fp32 moment tensors scale with the trainable count. Ignoring per-parameter metadata, the moment payload is:
all GPT-2 base parameters:
124,439,808 * 2 * 4 bytes = 949.4 MiB
attention LoRA parameters:
294,912 * 2 * 4 bytes = 2.25 MiB
ratio:
421.96These buffers are allocated lazily by PyTorch when a parameter first receives a gradient. They are not the total training memory. A full accounting includes weights, gradients, saved activations, temporary operator storage, allocator fragmentation, and any reduced-precision master copies. LoRA directly shrinks the parts proportional to the trainable parameters. It does not shrink the activations flowing through the frozen model.
At inference, the patch can either remain separate or be merged:
W_merged = W + scale * A * BSeparate execution computes the base projection and two adapter projections on every call. It permits adapter switching but adds operations and memory reads. Merged execution materializes a different full weight for each adapter and runs the original projection shape. It removes the adapter operations, but it gives up cheap switching unless merged copies are stored or rebuilt.
The two forms are algebraically equal. They are not required to be bit-identical in floating point because one evaluates (x·A)·B separately and the other changes the entries consumed by the base matrix multiplication. On three prompts after 60 deterministic training steps, I measured:
maximum absolute logit difference: 0.00067138671875
same highest-logit token at the final position: 3 of 3 promptsThe top token agreement is only a three-prompt check, not a proof of behavioral equivalence. The logit difference is neither automatically harmless nor meaningfully comparable with unspecified “GPU noise.” Whether it matters depends on margins and downstream sampling. A difference smaller than the gap between the first and second token cannot change greedy selection at that position. With sampling, a small logit change can alter a random draw even when the distributions are close.
Merging into a quantized base adds another complication. The sum is naturally formed in a wider type. Storing it back into four bits requires requantization, which can erase part of the update. Keeping the adapter separate avoids that particular loss but retains its runtime work. The clean fp32 merge above verifies the algebraic path I ran, not every deployment format.
four-bit rounding did not make this QLoRA
LoRA leaves the frozen base resident. Seven billion values occupy 14 GB as raw fp16 payload and 3.5 GB as raw four-bit codes. Both figures omit metadata, tensor padding, quantization scales, temporary dequantized tiles, and every non-parameter allocation. Reducing the trainable count does not by itself make the base fit.
QLoRA combines a frozen four-bit base with trainable LoRA adapters, but “four-bit base” is not its complete mechanism. The paper introduces NormalFloat 4, or NF4, for normally distributed weights, quantizes the quantization constants again through double quantization, and uses paged optimizers to handle memory spikes. Computation occurs in a wider type after weights are dequantized. The paper reports fine-tuning a 65-billion-parameter model on one 48 GB GPU. That is an externally reported result from its hardware and software stack, not a run reproduced here.
My CPU probe did something much simpler:
groups = weight.flatten()[:usable].reshape(-1, 64)
scale = groups.abs().amax(dim=1, keepdim=True) / 7.0
codes = torch.round(groups / scale).clamp(-8, 7)
rounded = codes * scaleFor each group of 64 values, this chooses one symmetric scale, rounds to an integer code from negative eight through seven, immediately multiplies the code by its scale, and writes the rounded value back into the original fp32 tensor. I excluded token and position embeddings. I also left any trailing values that did not fill a complete group unchanged.
This is a four-bit round-trip simulation. The code variable is temporary. The model still stores every rounded value as fp32. There is no packed nibble array, NF4 codebook, double quantization, paged optimizer, CUDA kernel, or low-precision matrix multiplication. Calling it a QLoRA run would confuse a numerical perturbation experiment with a memory and execution system.
The probe rounded 84,934,656 values across 48 matrices:
relative Frobenius error by matrix:
minimum 0.108339
mean 0.120799
maximum 0.173118
actual parameter storage before rounding: 497,759,232 bytes
actual parameter storage after rounding: 497,759,232 bytesThe unchanged storage is the negative control. Four-bit-looking values inside fp32 tensors save no memory. If those 84,934,656 codes were actually packed two per byte, their raw code payload would be 42,467,328 bytes. Adding one fp16 scale for each group of 64 would make the payload 45,121,536 bytes. That estimate still excludes the unquantized embeddings, incomplete groups, tensor metadata, and runtime workspaces.
Rounding changed the mean language-model loss on four unrelated sentences by +0.267495 nats per predicted token before any adapter was added. Frobenius error did not predict that value; it only measured distance in weight space. The only useful quality test is behavior on declared data.
I attached the same rank-8 attention adapter and trained on the twelve ticket rows with prompt masking:
first training loss: 12.145596
best training loss: 0.0000245665
final training loss: 0.0000245665
training accuracy: 1.000
held-out accuracy: 1.000The rounded model plus adapter fit the training rows and classified all eight held-out rows in this seed. Its unrelated loss rose by another 2.706046 nats relative to the already rounded base. The adapter did not repair the general perturbation caused by rounding. It learned the narrow ticket boundary while producing more damage on the small unrelated sample.
This result cannot support the claim that a crude quantizer surviving implies NF4 will survive. Quantization schemes distribute error differently. A downstream task can be sensitive to a small error in one direction and insensitive to a larger Frobenius error elsewhere. Nor can the result establish memory savings, since the process resident tensors stayed fp32.
It does establish one limited mechanism: gradients can train an fp32 low-rank patch on top of a fixed, numerically perturbed base. That mechanism is necessary for QLoRA, but not sufficient to reproduce it. A faithful execution would need a supported GPU, packed four-bit storage, the actual dequantization kernels, NF4 and double-quantization metadata, optimizer paging behavior, peak allocated and reserved memory, throughput, and quality comparisons against a wider-precision baseline. I cannot measure those on the current machine, so the QLoRA memory and 65B claims remain attached to the paper rather than to this probe.
the patch changed text it never trained on
Freezing preserves the base tensors, not the function produced after another path is added. For one hidden vector h, the old projection returned h·W. The adapted projection returns:
h·W + scale·h·A·BThe first term is unchanged. The second term enters every selected block for every token. If it is large in a direction common to many inputs, it can alter outputs far outside the training set. A small parameter count does not bound the output change. Neither does low rank. A rank-1 matrix can have an arbitrarily large singular value.
The ticket experiment measured this interference as a change in cross-entropy on eight unrelated sentences. Cross-entropy is the negative logarithm of the probability assigned to the observed next token, averaged using the probe’s sentence-by-sentence convention. A rise of 1.625011 nats means the geometric probability assigned to those observed tokens fell by a factor of e^1.625011, about 5.08, under that averaging. The three seeds produced factors of about 5.08, 12.14, and 2.79. All three still achieved perfect training accuracy.
That is evidence of interference on eight strings. It is not enough to say the model forgot the facts expressed by those strings. A model can assign lower likelihood to one wording while retaining another route to the fact. Establishing catastrophic forgetting would require a wider evaluation with multiple prompts, generations, calibrated task metrics, and controls for ordinary seed variation. The exact phrase is useful for the general failure mode, but the local result should keep its local scope.
I also tried to locate the damage by attaching rank-8 patches to different GPT-2 projections. The attention-only patch added 294,912 parameters. The MLP-only patch added 368,640. Patching both added 663,552. All three saw one sentence for 100 steps. Their unrelated losses differed.
That comparison was not controlled well enough to identify a causal patch site. The parameter budgets were unequal. The matrices had different shapes and roles. Equal rank did not mean equal capacity or equal update norm. The final target losses were not matched exactly, and the evaluation contained only three unrelated sentences. The “both” condition changed two variables at once. From those runs I can say only that placement changed the observed result in that setup. I cannot say attention is inherently more load-bearing or that adding sites causes damage in a straight line.
A proper placement experiment would choose a common adapter-parameter budget, tune rank separately for each matrix shape, match the achieved training loss or stop by a declared validation rule, repeat seeds, measure update norms, and evaluate on enough unrelated and held-out task data to estimate uncertainty. It would also separate query, key, value, attention output, and MLP projections rather than grouping unlike operations under two labels. I have not completed that experiment, so it remains in the private register rather than supporting a public ranking of patch sites.
The result I can use is the prompt-mask control because it changed one declared property while keeping seed, data, rank, steps, and learning rate fixed. Scoring prompt tokens increased the unrelated-loss change from 1.625011 to 6.682416 nats for seed 0. Under the probe’s averaging, that second change corresponds to an approximately 798-fold decrease in geometric observed-token probability. Both versions were perfect on the two-label accuracy metric. Accuracy alone would have selected neither and warned about nothing.
Preservation data must therefore be part of model selection, not a story added after training. For a production adaptation I would keep at least four disjoint collections: training examples that create gradients, validation examples that select steps and hyperparameters, held-out task examples that estimate transfer after selection, and regression examples representing behavior the patch must not damage. Reusing one set for several roles leaks information into the reported result. The tiny probe has only train, held-out, and unrelated sets, and I did use the held-out result while interpreting the design. It is an investigation, not a final unbiased model estimate.
The frozen base does provide one operational advantage: disabling the adapter recovers the exact original computation. That makes rollback cheap. It does not make the active adapter safe. The relevant artifact is the pair consisting of base revision and adapter revision, evaluated together.
the general text helped until it did not
The obvious response to unrelated damage is to mix general text into training so the optimizer is penalized when ordinary language modeling gets worse. This is often called rehearsal. The word makes it sound like the old behavior is being replayed exactly. I used a stricter control: eight rehearsal sentences created gradients, while a different set of eight preservation sentences was used only for measurement. A sentence about a compass appeared in rehearsal. A sentence about the moon appeared in preservation. No exact preservation string was trained.
For seed 0, I optimized:
total loss
= ticket target loss
+ rehearsal weight * general-text lossThe two component losses are already means over different token sets, so the rehearsal weight is a dimensionless policy choice about their relative contribution. It is not the fraction of rows or tokens. I tried weights 0, 0.1, and 1.0, kept the ticket data and optimizer fixed, and inspected steps 1, 10, 20, 40, and 80. The preservation baseline was the frozen model’s mean loss of 4.204346 on its eight strings.
weight step ticket loss held-out ticket accuracy preservation loss change
0 20 0.001569 0.750 +0.743825
0 40 0.000111 1.000 +1.405113
0 80 0.000019 1.000 +1.625011
0.1 20 0.001540 0.750 +0.313326
0.1 40 0.000376 0.875 -0.075578
0.1 80 0.000034 0.875 +1.608543
1.0 20 0.008960 0.875 -0.013489
1.0 40 0.000326 0.875 +1.795101
1.0 80 0.000019 1.000 +2.130674The sign of preservation change is relative to the untouched base. Negative means the patch happened to lower loss on this small sample. At step 20, both rehearsal weights reduced damage and weight 1.0 also improved held-out ticket accuracy from 6 of 8 to 7 of 8. This is the result I expected.
The later steps broke the simple story. Weight 0.1 reached its best preservation value at step 40, then lost almost all of that protection by step 80. Weight 1.0 protected the sample at step 20 but damaged it more than the no-rehearsal run at steps 40 and 80. Meanwhile its rehearsal loss kept falling, from 4.825849 before useful adaptation to 0.049068 at step 80. The adapter was memorizing the eight rehearsal sentences too. Success on them stopped predicting preservation on the disjoint set.
Rehearsal is therefore another training objective with its own overfitting path. It does not create a force that points toward “general ability.” It creates gradients for the exact general tokens supplied. If those tokens are too few, too repetitive, or drawn from the wrong distribution, the adapter can fit them while continuing to move elsewhere.
The checkpoint choice becomes multi-objective. If I select only by ticket training loss, step 80 wins or nearly wins every weight. If I require perfect held-out ticket accuracy, weight 0 at step 40 qualifies with +1.405 preservation damage, while weight 1 does not qualify until step 80 and has +2.131 damage. If I allow 7 of 8 held-out tickets, weight 1 at step 20 has near-zero preservation change. Those are product decisions expressed as gates, not facts the optimizer discovers.
Eight held-out tickets make one error worth 12.5 percentage points, so choosing between 87.5 and 100 percent from this table would be fragile. The preservation set is equally small. This is a single seed and a manually chosen set of checkpoints. The negative value at step 40 for weight 0.1 may be ordinary sampling noise. I would not deploy a checkpoint from these differences. The experiment earns a narrower conclusion: rehearsal weight and stopping time interact, and a fixed general-text mixture cannot replace a disjoint preservation evaluation.
There is also an information leak to avoid. If I inspect preservation loss after every step and select its minimum, that set has become validation data. Reporting the same minimum as an unbiased preservation estimate would be false. A real run needs another untouched regression set after checkpoint selection, or a nested procedure that accounts for the selection. Every hyperparameter tried against a set, including rank, learning rate, rehearsal weight, prompt format, and step count, spends some of that set’s independence.
The optimizer saw no special boundary at step 20 or step 40. Only the external measurements made those points different. This is why “train for three epochs” is not an explanation. An epoch is one pass through the chosen rows. It says nothing about how quickly a repeated template is memorized, when unrelated behavior begins to move, or which checkpoint satisfies the actual deployment constraints.
the answer tokens needed their own objective
The ticket classifier is still next-token prediction. The response happens to be one token. For a longer answer containing tokens y1 through yT, supervised fine-tuning minimizes:
loss = -[log p(y1 | prompt)
+ log p(y2 | prompt, y1)
+ ...
+ log p(yT | prompt, y1, ..., yT-1)] / TEach target token is scored after receiving the true previous target tokens. This is teacher forcing. During generation, the model receives its own previous output instead. A locally likely wrong token can therefore move the later hidden states to prefixes the training rows never contained. A low supervised loss does not measure recovery from those self-produced prefixes.
The prompt mask decides where this sum starts. Padding masks decide which batch positions are real tokens. A causal mask decides which earlier positions each token can inspect. These three masks have different purposes. Confusing them can produce a loss that decreases while training the wrong conditional distribution.
Instruction tuning changes the distribution of supervised sequences. The model sees prompts followed by responses selected as desirable examples, and the response tokens create gradients. Nothing in this operation proves that the selected responses are correct, that omitted alternatives are worse, or that behavior generalizes beyond the sampling process that produced the data. The word “instruction” names a data format, not a new optimizer.
Comparative data supplies another signal. For one prompt x, a row contains a chosen response yc and a rejected response yr. The labels assert an ordering for this pair. They do not supply a numerical reward difference, and they do not say every sentence resembling yr should become impossible.
Direct Preference Optimization, or DPO, derives a classification loss from a reference-constrained reward optimization problem. The local implementation needs four sequence log-probabilities:
policy chosen: log pi(yc | x)
policy rejected: log pi(yr | x)
reference chosen: log pref(yc | x)
reference rejected:log pref(yr | x)The sequence log-probability is a sum over response tokens only:
logits = model(**batch).logits[:, :-1]
targets = batch.input_ids[:, 1:]
token_logp = logits.log_softmax(-1).gather(
-1,
targets.unsqueeze(-1),
).squeeze(-1)
sequence_logp = (token_logp * response_mask).sum(dim=-1)The one-position shift pairs logits after a prefix with the next observed token. The response mask removes prompt and padding positions. I use a sum, not a mean, because that is the sequence probability in the DPO expression. Length then matters: adding another token adds another nonpositive log-probability. If chosen and rejected lengths correlate with the label, the comparison can contain a length signal that has nothing to do with answer quality.
For each model, subtract rejected from chosen:
policy margin = log pi(yc | x) - log pi(yr | x)
reference margin= log pref(yc | x) - log pref(yr | x)
improvement = policy margin - reference marginThe DPO loss for one pair is:
-log sigmoid(beta * improvement)beta scales how rapidly the sigmoid changes with the margin. In my probe it is 0.1. When policy and reference begin with identical weights, their margins are identical. Improvement is zero, the sigmoid is one half, and the loss is -log(0.5), or 0.693147.
The derivative with respect to the scaled improvement z is sigmoid(z) - 1. At zero it is negative one half. Gradient descent therefore increases the improvement. The loss does not separately request that chosen probability rise and rejected probability fall. Either motion opens the margin. This omission is where the next measurement became uncomfortable.
six comparisons found a shortcut
I wrote six training comparisons about customer-support failures and four held-out comparisons with different incidents. A chosen answer apologized, named a concrete next action, and promised follow-up. A rejected answer blamed or dismissed the customer.
prompt:
A customer lost work after the editor crashed. Reply:
chosen:
I am sorry your work was lost. I will inspect the crash
and check whether recovery is possible.
rejected:
You should have saved more often.The data was hand-written for this probe. It was not collected from customers or independent raters. Its regularity makes it easy to inspect and easy for the model to exploit. The chosen responses are longer, repeatedly start with "I am sorry", and contain verbs such as "check", "review", and "investigate". The rejected responses are short and contain blame. A held-out pair follows the same template. Success can mean the patch learned those surface correlations.
The frozen GPT-2 reference initially preferred the rejected response on all six training rows. Its chosen-minus-rejected sequence log-probability margins were:
-24.55, -31.58, -14.16, -36.51, -27.17, -15.95 natsPart of that result follows directly from length. A sequence probability multiplies one probability per token, so a longer response usually has a more negative log-probability. DPO compares the policy’s change from the reference, which cancels the initial raw disadvantage at step zero. It does not remove every correlation between length, wording, and the direction the adapter can learn.
I trained an attention rank-8 adapter for 50 full-batch steps at 1e-3, with dropout disabled, using the same pinned model and three initialization seeds. I evaluated two accuracies. Raw preference accuracy asks whether the policy now assigns the chosen sequence a higher log-probability than the rejected sequence. Improvement accuracy asks whether its chosen-minus-rejected margin improved relative to the reference, even if the chosen sequence remains absolutely behind.
All three runs reached 100 percent for both accuracies on six training and four held-out pairs. The result looks decisive until I split the margin into its two sides:
seed final DPO loss held-out margin improvement chosen drift rejected drift unrelated loss change
0 0.00000341 +112.81 nats +2.32 -110.49 +1.2199
1 0.00000347 +111.69 nats +5.30 -106.39 +0.9570
2 0.00000492 +116.42 nats +7.80 -108.62 +0.8920Drift is measured against the frozen reference on the same response. For seed 0, the chosen held-out responses gained only 2.32 nats on average while the rejected responses lost 110.49. More than 97 percent of the average margin improvement came from suppressing rejected sequences. The other seeds show the same direction.
The loss function permits this. If chosen drift is +2 and rejected drift is -110, their difference improves by 112. If chosen drift is +112 and rejected drift is zero, the same pairwise improvement enters the sigmoid. The objective sees no difference between those paths.
This does not mean the model learned a general ability to recognize rude text and delete it. It means the probability of these four held-out rejected strings fell relative to the frozen model. Their shared template makes transfer unsurprising. Testing a broader property would require responses where courtesy and correctness disagree: a polite but false answer, a terse correct answer, a long evasive answer, an apology that offers an unsafe action, and a refusal that is appropriate. The current data makes politeness, length, specificity, and the label move together.
The unrelated language-model loss rose by 0.89 to 1.22 nats across seeds. That measurement again blocks the comfortable interpretation that pair accuracy alone represents improvement. The patch satisfied every declared comparison and changed unrelated token probabilities at the same time.
There is also no calibrated meaning to a margin of 112 nats here. The label says chosen, not “chosen by a probability ratio of e^112.” Continuing to optimize after every pair is correctly ordered drives the logistic loss toward zero by widening margins without receiving new preference information. A held-out accuracy plateau and a growing magnitude can therefore coexist. Selecting the final step by training loss rewards confidence the data never justified.
the frozen reference changed the starting geometry
I removed the reference terms as a negative control:
loss = -log sigmoid(
beta * [log pi(yc | x) - log pi(yr | x)]
)Because the base model initially preferred every short rejected response, this objective started at 2.601360, not 0.693147. After 50 steps, seed 0 reached 0.0000431. On held-out pairs it achieved 100 percent raw and improvement accuracy, opened the reference-relative margin by 107.91 nats, raised chosen log-probability by 7.80, lowered rejected log-probability by 100.11, and increased unrelated loss by 0.79535.
The control was not dramatically worse on this tiny dataset. It was slightly less damaging on the four unrelated sentences under this seed. That result does not invalidate the DPO derivation. It shows that this probe is too small and too regular to demonstrate an advantage from the reference.
The reference terms are constants during policy optimization, but they differ across examples. A pair the reference already strongly prefers and a pair it strongly rejects enter with different offsets. The reference therefore changes which examples occupy the steep part of the sigmoid and how much movement each needs to obtain the same improvement. On the current six rows, both objectives found an easy direction that separated the templates.
It is too broad to say that the reference by itself anchors the whole model distribution. The loss evaluates reference ratios only on the chosen and rejected responses in the dataset. Unseen text has no direct term. The reference appears because the DPO paper derives the optimal policy for a reward objective regularized by divergence from a reference policy, but a finite sampled preference loss can still move unmeasured behavior. The unrelated-loss increases demonstrate that local limitation.
The reference also has a systems cost. My probe loaded a second frozen GPT-2, so CPU memory held two bases plus one adapter. When the policy is a LoRA adapter over the same base as the reference, an implementation has other choices:
- Run each batch once with the adapter enabled and once with it disabled.
- Precompute reference log-probabilities for a fixed dataset and store two scalars per pair.
- Keep a separate reference replica so policy and reference passes can overlap.
Precomputation removes reference forward passes during training, but it binds the cached values to an exact base revision, tokenizer, truncation rule, chat template, and response mask. Change any of them and the cache is stale. A separate replica consumes memory. Toggling the adapter saves a base copy but serializes work unless the batches are arranged differently. The mathematical loss does not choose the execution plan.
the batch determined what one step meant
The ticket experiment used a full batch of twelve rows. Every optimizer step saw all twelve, and each row contributed one target token. The preference experiment used all six pairs per step, but each response contained a different number of tokens and the objective reduced one sequence-level loss per pair. Those choices decide how examples are weighted.
Suppose supervised tuning instead concatenates variable-length answers and averages over all unmasked tokens. A 200-token answer contributes twenty times as many token losses as a 10-token answer. If it averages one loss per example after first averaging within each response, both answers contribute equally. Neither reduction is universally correct. One weights observed tokens; the other weights rows. The dataset and product metric must decide.
Mini-batching introduces another layer. With batch size b, Adam updates from a noisy estimate of the full-data gradient. Accumulating gradients over k microbatches can imitate a larger batch only if the loss scaling matches. If each microbatch loss is already a mean and I call backward() k times without dividing by k, the accumulated gradient is k times larger. Adam’s normalization can hide some scale changes, but weight decay, clipping, finite epsilon, and schedules keep the runs from being identical.
Padding consumes memory and computation even though its labels are ignored. Grouping similarly sized sequences reduces wasted work. Packing several short examples into one token sequence improves utilization but needs boundaries that prevent one example from attending to another if that interaction is not intended. A missing boundary silently changes the task.
Duplicate rows change the objective even when they add no information. Repeating one easy style pattern one hundred times gives it one hundred gradient contributions unless sampling or weighting compensates. Near-duplicates can leak across train and held-out splits and make generalization look better than it is. For generated data, shared templates create the same leak without exact string matches.
Class balance is equally mechanical. The ticket set has six urgent and six routine rows. A production queue may be 2 percent urgent. Training on the balanced set makes each label equally visible to the optimizer, which is useful for learning the distinction, but a two-way softmax probability from that model is not automatically a calibrated production probability. Evaluation should report both discrimination and behavior at the deployment base rate. A threshold chosen on a balanced toy can overload the urgent path when applied to real traffic.
The data also defines absence. No ticket row says what to do with ambiguity, malicious content, two simultaneous incidents, or an unsupported label. The model must still emit something. A loss over only valid rows cannot teach abstention unless abstention or uncertainty appears in the targets and the evaluation checks it.
This is why data quality is not a decorative concern after the optimizer. The data is the executable specification of what incurs loss. A mislabeled preference reverses a gradient. A missing response mask spends capacity on the prompt. A length correlation offers a shortcut. A duplicate changes weighting. An unrepresentative split rewards the wrong boundary. The training code can be numerically perfect while faithfully optimizing all of those mistakes.
the label did not say why one answer won
Each preference row in my probe contains one bit of supervision: chosen beat rejected. The bit does not encode whether the chosen answer was more correct, safer, clearer, less verbose, or merely more polite. The six hand-written pairs made all of those properties move together. The optimizer had no reason to distinguish them.
Real comparisons can conflict without either rater being careless. One answer may be concise and omit a caveat. Another may be complete and too slow for an incident channel. A support engineer may prefer the first while a compliance reviewer prefers the second. Collapsing both contexts into one binary label turns a product policy disagreement into gradient noise.
Pairwise preferences can also form a cycle:
A preferred to B
B preferred to C
C preferred to ANo scalar ordering can satisfy all three with positive margins. DPO does not fail mechanically; it finds parameters that trade the losses against each other. A nonzero final loss may therefore reflect irreducible disagreement rather than insufficient rank or optimization. Driving one pair harder can worsen another.
Agreement rate matters, but raw agreement can mislead when one label dominates or examples are easy. I would retain independent votes, rater context, tie or abstain options, and the reason category before reducing anything to chosen and rejected. If one pair received nine votes to one and another received six to four, treating them as equally certain discards information. The original DPO row format does not carry that strength unless the training procedure adds weighting or a different objective.
The response source matters as well. If chosen answers came from a stronger model and rejected answers from a weaker one, formatting, length, vocabulary, and refusal style can reveal the generator. The policy can learn generator identification instead of the quality judgment. Randomizing presentation order only removes position bias. It does not remove source fingerprints.
Deduplication must operate above exact bytes. Two prompts can differ in an account number and still instantiate the same template. Putting one in training and one in evaluation measures template recognition. That may be useful, but it should not be described as transfer to a new failure mode.
For customer data, retaining the raw row can create a second problem. Prompts and responses may contain names, account identifiers, secrets, or incident details. Removing them after adapter training does not prove they cannot affect generations. Data access, redaction, retention, deletion, and model-registry lineage belong to the training design because the adapter is a derived artifact of those rows.
None of these concerns can be repaired by a lower loss. They determine what the loss means. The six-pair probe is interpretable precisely because its origin and shortcuts are visible. Its 100 percent held-out preference accuracy says the patch transferred the hand-written style contrast. It says nothing about whose definition of good support should govern a deployed model.
the frozen base still had to cross machines
LoRA’s optimizer-state reduction is large, but it does not make distributed training disappear. If the base does not fit on one accelerator, or if the desired throughput needs several, the system still has to place weights, activations, batches, and adapter gradients.
Traditional data parallelism copies the model to every worker. Each worker receives different examples, computes local gradients, then combines corresponding gradients so every replica applies the same update. With a frozen base and trainable adapters, only adapter gradients need synchronization. For this GPT-2 setup, that is 294,912 gradient values instead of 124,439,808. At fp32 payload sizes, one local adapter gradient is about 1.125 MiB. A full-base gradient would be about 474.7 MiB. Collective protocol overhead and topology are additional costs, but the byte difference is real.
An ideal ring all-reduce over p workers divides a gradient into p chunks. Reduce-scatter circulates and combines chunks for p - 1 steps. All-gather circulates the reduced chunks for another p - 1 steps. Each step transfers one pth of the gradient, so bytes sent by each worker are:
2 * (p - 1) / p * gradient payloadFor eight workers, the ideal payload factor is 1.75. The GPT-2 adapter would send about 1.969 MiB per worker per optimizer step. A full fp32 base gradient would send about 830.7 MiB. These are derived payload counts, not measured network traffic. Framing, protocol headers, alignment, retransmission, library buffers, and topology can add work. An implementation may also choose a tree for small messages where latency matters more than the ring’s bandwidth efficiency.
Gradient accumulation changes frequency, not information for free. If eight microbatches are accumulated locally before all-reduce, synchronization happens one eighth as often, but the optimizer also sees a global batch eight times larger unless some other dimension changes. That can alter optimization. Delaying communication is not equivalent to making the same training run faster.
The smaller gradient can expose fixed overhead. Launching a collective, coordinating streams, and waiting for a slow rank take time even when the payload is one byte. On a fast interconnect, a 1.125 MiB adapter all-reduce may become latency-dominated while a 474.7 MiB full gradient is bandwidth-dominated. The parameter reduction changes which bottleneck matters; it does not make synchronization zero.
The backward compute reduction is less dramatic than the gradient bytes. For a dense projection:
forward:
Z = H * W
weight gradient:
dW = transpose(H) * G
input gradient:
dH = G * transpose(W)Freezing W removes the need to form and store dW. It does not remove the forward multiplication by W. If an earlier adapter needs a gradient, backward still needs dH through later frozen projections so the chain reaches that adapter. The base participates in forward and in activation-gradient propagation even while its own .grad remains absent.
This is why a 422-fold optimizer-state reduction does not imply a 422-fold training speedup. Saved activation memory can dominate at long sequence lengths. Base forward and input-gradient matrix multiplications still dominate arithmetic. Adapter projections add their own smaller multiplications. The speed result depends on the fraction of time previously spent forming base weight gradients, updating base parameters, synchronizing gradients, moving memory, and waiting between kernels. I did not measure that breakdown on a GPU.
The base remains replicated under ordinary data parallelism. A 7B fp16 base still consumes about 14 GB on every worker before activations. If it does not fit, parameter sharding or model parallelism is still necessary. The ZeRO paper partitions optimizer state, gradients, and eventually parameters across data-parallel processes rather than keeping every copy everywhere. Current PyTorch FSDP documentation describes a parameter-sharding wrapper whose process groups perform all-gather and reduce-scatter collectives. Those operations exchange memory capacity for communication.
During an FSDP-style forward path, a worker materializes the parameter shard set needed by a wrapped module, computes with it, then can release full parameters before materializing another set. Backward needs parameters again to compute input and parameter gradients, and reduce-scatter distributes the resulting gradient shards. Prefetching can overlap some communication with computation, but overlap needs independent work and enough memory for prefetched parameters. A small LoRA gradient does not remove the all-gathers required to execute a sharded frozen base.
Quantized storage changes the size of persistent base shards, but dequantized values or tiles must exist where matrix multiplication consumes them. Activation memory still scales with batch, sequence length, hidden width, layer count, saved tensors, and checkpointing policy. The earlier backward investigation showed why frozen parameters do not eliminate every saved activation: gradients for an adapter in an early block still depend on the activations reaching that block.
For preference training, each pair creates chosen and rejected sequences. DPO also needs policy and reference scores. A direct implementation can perform four logical sequence evaluations per pair. Batching chosen and rejected together improves device occupancy but lengthens the token batch. Precomputed reference scores halve the live model paths but add a versioned data artifact. On multiple workers, the sampler must avoid accidentally repeating or omitting rows, and the reported global batch includes every worker and every accumulation step:
global batch
= rows per microbatch
* microbatches accumulated before an optimizer step
* data-parallel workersIf each row contains a chosen and rejected response, sequence work is roughly doubled again relative to one supervised response, subject to padding and batching. Calling DPO a one-line loss describes the scalar expression, not the execution cost.
I did not run these probes across GPUs. The byte counts are derived from tensor sizes, and the sharding behavior comes from the cited paper and documentation. Actual throughput depends on interconnect bandwidth, collective algorithms, bucket sizes, layer wrapping, sequence lengths, kernel efficiency, and overlap. Those need measurements on the target cluster.
the adapter file could not identify the model
A rank-8 adapter is not a complete model artifact. Its A and B tensors only make sense beside the exact base projections whose dimensions and meanings they modify. Reproducible loading requires at least:
base model identifier and immutable revision
tokenizer files and revision
model configuration
adapter target module names
rank and alpha
tensor dtype and serialization format
prompt or chat template
maximum length and truncation side
library versions or a tested compatibility range
training-data and evaluation identifiersThe GPT-2 revision in these probes is part of the result. Loading a newer file under the same human-readable model name can change weights while every adapter tensor still has a compatible shape. The program may load cleanly and behave differently.
Target module names are also code-dependent. This article’s wrapper assumes GPT-2 Conv1D weights are stored as input by output. A standard torch.nn.Linear stores output by input. Reusing the multiplication order without checking the layout can transpose the intended update or fail only when dimensions differ. An adapter framework hides this detail, but the serialized configuration still has to identify the correct module implementation.
Merging complicates provenance. Once W + scale·A·B is materialized, the original base and adapter are no longer separable from the tensor alone. An operator needs to know which base revision was merged, which adapter revision, the merge precision, whether the result was quantized afterward, and whether validation was performed before or after that quantization.
Rollback is easiest when the base remains immutable and the adapter can be disabled atomically. That same reversibility supports canarying: send a small declared fraction of requests through the new adapter, compare task metrics and preservation metrics, then expand only if both stay within limits. A training loss is not a rollout gate. Neither is one offline aggregate. Inputs can shift, rare prompt forms can trigger large changes, and an adapter can interact with decoding parameters or system prompts absent from the training probe.
The active artifact should therefore carry evaluation evidence beside tensors. For the ticket patch that means, at minimum, the exact 12 training rows, 8 held-out rows, 8 unrelated strings, three seeds, target token IDs, prompt masking rule, model mode, optimizer, steps, and observed metrics. In a real system, retaining customer text may be prohibited. Then the artifact still needs dataset versions, privacy-preserving summaries, access controls, and a way to rerun approved evaluations without copying sensitive traces into the model registry.
zero loss was the smallest answer
The first line said the rank-8 patch drove loss from 1.0184 to zero while 589,824 base values did not move. It still says exactly that. The rest of the measurements changed what the line is allowed to mean.
On the constructed matrix, zero loss meant the patch represented the target rows, and the analytic residual explained why lower ranks could not. On one GPT-2 sentence, it meant the response tokens were memorized. On twelve tickets, every seed fit training, one seed missed a held-out row, and all three damaged unrelated text. Scoring prompt tokens produced the same classification accuracy and much worse preservation. Four-bit rounding left actual storage unchanged. Six preference pairs reached perfect held-out pair accuracy mostly by driving rejected sequence probability downward.
The frozen weights were never a shield around the old behavior. They were one term in a new function. The adapter’s small storage was never a bound on its effect. It was a cheaper set of coordinates in which the optimizer could move.
The useful model of “teaching” is now operational. Declare which tokens create loss. Separate rows that create gradients from rows that select and test the result. Measure behavior the update must preserve. Repeat enough initializations to see instability. Keep the base, tokenizer, masks, and execution mode pinned. Treat quantized storage as an implementation property rather than a word in the experiment name. For preferences, inspect chosen and rejected likelihoods separately because their difference hides the path.
I can now predict the original contradiction. A low-rank patch can fit a narrow dataset because rank limits the dimension of a matrix update, not the magnitude of its effect. The same shared update is applied to hidden states from unrelated text, so those outputs can move even when every base tensor remains bit-identical. More training on the narrow objective can widen both effects.
The next time a loss reaches zero, the first result I want is no longer another decimal place. I want the row the optimizer never saw, the text the patch was supposed to leave alone, and the exact tensor path that changed them.