Moves mainly in Kaon v3 and then holds: 0.00 → +0.56 → +0.60.
Research / J-space / checkpoint study
What changed inside our roleplay models?
We expected post-training to push the model along one direction. A Jacobian-lens study of 8,192 paired prompts found something stranger: different parts of its semantic landscape changed at different times—and two reversed course.
3model-native checkpoints
8,192paired roleplay prefills
9predeclared semantic axes
4 / 9estimator-stable trajectories
Research status
Preliminary point estimates and provisional gate outcomes. Frozen statistical
release and confirmation analysis are still to come.
[01] / The question
“Better at roleplay” hides more than it explains.
People often describe post-training as a slider: more helpful, less likely to refuse, better at roleplay. That picture is convenient—and probably wrong.
Across successive Kaon checkpoints, outputs became richer and more controllable. Kaon v3 followed scene changes, accepted world-state updates, and exited character more reliably than its instruction-tuned base. Kaon v3.5 changed again.
But did the models become more trusting? More intense? More playful? Less cruel? Did those tendencies move together, or did training reshape them on different schedules?
The question we asked
If we look at what three checkpoints are preparing to say—not only what they finally say—can we see post-training reorganize their internal semantic landscape?
We expected a direction.
We found a landscape.
[02] / The result
The model did not move as one thing.
Four trajectories survived the primary estimator and two sensitivity estimators. Each tells a different temporal story.
Appears mainly in Kaon v3.5: 0.00 → +0.01 → +0.41.
Moves toward care, then bends back: 0.00 → −0.36 → −0.12.
Moves toward restraint, reverses, and crosses baseline: 0.00 → −0.23 → +0.16.
Training reshapes terrain. It doesn’t move a slider.
At least across these measured cross-sections, post-training looks less like one capability knob and more like terrain being reshaped: ridges rise, valleys flatten, and some slopes reverse.
[03] / A lens inside
Read the model state before it becomes a word.
The Jacobian lens maps an intermediate state into coordinates associated with potential downstream verbalization.
Stop at the final prompt position, before the first response token is sampled.
Use an averaged downstream Jacobian to estimate later verbalization.
Measure whether the state leans toward hope or despair, cruelty or care.
p̂ℓ,t = softmax(WU Norm(Jℓ hℓ,t))
A restricted, verbalization-related projection—not a transcript of hidden reasoning
and not the model’s complete residual stream.
We call the resulting coordinate system J-space. One prompt gives us a point. Thousands of prompts give us a distribution. Nine semantic directions give us measurable cross-sections through that distribution.
The catch: every checkpoint has its own activations, tokenizer, unembedding, and fitted lens. A raw “hope vector” in Gemma is not numerically identical to one in Kaon v3.5. Comparing them requires a bridge.
[04] / The bridge
Different coordinates. The same semantic questions.
We compare within-model, neutral-calibrated gaps—not raw directions across models.
- Ask one semantic question. Is the state closer to hope or despair?
- Measure roleplay against neutral. Center each roleplay population on that checkpoint’s own neutral-context median.
- Compare adjacent checkpoints. Track the calibrated gap from Gemma → v3 → v3.5.
- Use one external ruler. Express every change in a fixed Gemma-derived median-absolute-deviation unit.
θ(m,a) = E[r(m,c,a) | c∼R] − median[r(m,c,a) | c∼N]
Roleplay population minus the same checkpoint’s neutral baseline.
δ(m₁,m₀,a) = [θ(m₁,a) − θ(m₀,a)] / sG(a)
A checkpoint interaction on a fixed Gemma-derived robust scale.
[05] / The experiment
Hold the prompts still. Replace the model.
Every checkpoint sees identical prompt bytes at the same semantic boundary. The prompt does not change. The complete checkpoint does.
Drawn from 8,181 source conversations in a fixed 32,768-prefill frame.
Each model is stopped at the final prompt position before response generation.
A disjoint panel estimates checkpoint-wide readout shifts.
Headline classifications must retain their qualitative shape under two sensitivities.
[06] / The landscape
What moved—and when?
Explore all nine axes. A large-looking movement is not automatically a robust story; the estimator-stability label matters.
Trajectory explorer
Gemma-centered / neutral-adjusted / reference-MAD scalebaseline / 0
Gemma IT0.00
Kaon v3
Kaon v3.5
Gemma → v3
v3 → v3.5
| Axis | Gemma | v3 | v3.5 | G → v3 | v3 → v3.5 | Classification |
|---|
[07] / What it means
A narrow causal claim. A broader new picture.
The experiment identifies what checkpoint replacement changed. It does not isolate which data, objective, compute, or optimization choice caused that change.
Every prompt is evaluated under every checkpoint with a fixed specification.
The interaction compares roleplay against its checkpoint-specific neutral baseline.
No matched branch isolates the contributions of data, objective, compute, or optimization.
This requires selective interventions and held-out behavioral endpoints.
[08] / What comes next
Turn the picture into a testable mechanism.
The next release should freeze the analysis, expand the controls, and test whether changing a measured direction causes selective behavioral effects.
- Freeze the statistical release. Confirmation split, intervals, and multiplicity control.
- Expand controls. Larger neutral panels, lexical nulls, and clearer provenance.
- Intervene selectively. Perturb one measured direction and test held-out behavior.
- Map more of the terrain. Add axes without pretending they form a complete topology.
The takeaway
Post-training did not move the model toward “more roleplay.” It reorganized separable semantic tendencies on different schedules.
- Gurnee, Sofroniew, Pearce, Lindsey et al. Jacobian-lens work on mapping intermediate states toward downstream verbalization.
- All reported J-space values are relative positions, not probabilities, preference scores, or direct behavioral estimates.