How to explain this lab's results without notes

How to use this guide

Read the foundations once, even if they feel familiar, because every study leans on the same five or six ideas. Then take one study at a time. Each ends with a box that says the result in two sentences. If you can say that box out loud and then answer its questions without opening the page, you understand the study.

Three habits matter more than any single fact:

• Lead with the control, not the effect. Almost every interesting number here only means something because of the comparison next to it. "Leakage went to zero" is half a sentence. "Leakage went to zero and a random vector of the same size left it at 0.67" is the result.
• Say what failed. This lab has a refuted hypothesis, a failed prediction, and a negative transfer result. They are evidence that the methods can say no, which is what makes the positive results believable.
• Separate what was planned from what was found. Each study says which tests were written down before the data existed and which were found afterwards. Post hoc findings are hypotheses, and the 14B prediction shows why.

Foundations

The residual stream and directions

A transformer keeps one vector per token position, the residual stream. Each layer reads from it and adds its output back. Qwen3-8B has 36 layers and a 4,096-dimensional stream; Qwen3-14B has 40 layers and 5,120 dimensions. A direction is a vector in that space. Projecting an activation onto a unit direction (a dot product) gives one number: how far the activation points that way.

The concealment and identity studies read the stream at the final prompt token, the position just before the model writes its reply. By then the model has read the whole prompt, so that position carries the rule it was given and, partly, the reply it is about to write. Keep that second part in mind: a readout there is not a pure representation of the instruction.

Paired difference directions

To find "the hiding direction", build pairs of prompts that are identical except for the rule: one says hide the item, one says it may be disclosed. Subtract the activations within each pair and average. Everything the two prompts share cancels: the scenario, the audience, the question. What is left is the effect of the rule. Pairing on scenario, audience, content and phrasing is what makes this clean.

With eight audiences you get eight directions, one per audience. Their mean is the shared direction. Each audience's specific component is its direction minus that mean. The specific share is the specific component's squared length as a fraction of the audience's full direction: how much of the hiding representation depends on who is being hidden from.

Logit lens and direct logit attribution

The logit lens applies the model's final normalization and unembedding to an intermediate layer, to see which tokens that layer is already pushing toward. Direct logit attribution (DLA) splits the final logit difference between two answers into contributions from each head and MLP, because the final stream is a sum of their outputs.

DLA has to include the final LayerNorm's scale. The first version of notebook 2 left it out, so the contributions summed to 187 when the actual logit difference was 13. Together with a second bug, running GPT-2 without its beginning-of-sequence token, this inflated numbers about 14×. Both are fixed and the artifacts are kept visible in the notebooks. It's a good story to tell: it shows the checks work.

Activation patching versus zero-ablation

Patching runs the model on a clean prompt and a corrupted one (France vs Italy), copies one component's activation from one run into the other, and measures how much of the output difference moves. Recovery near 100% means that component carries the difference. Zero-ablation sets a component to zero instead. It is cruder: it can break general plumbing that every prompt needs. In the capital notebooks, zero-ablation's top hits (L0H10, L1H3) turned out to be load-bearing plumbing, not part of the circuit.

Attribution patching

Patching every neuron one at a time is expensive. Attribution patching estimates all of them at once with one backward pass: gradient times the change in activation. It is a first-order (linear) estimate, so it is most accurate near the output and biased toward late-layer components. The identity study uses it to rank neurons and then checks the ranking with real patches, finding the smallest top-k set that flips the answer.

Steering and its controls

Steering adds a scaled direction, alpha times a vector, to the residual stream during generation, usually at every position after some layer. Two rules make steering results believable:

• Norm-matched random controls. Any large push can change behavior, for example by knocking the model out of following its instructions. A random vector of the same size tells you how much change comes from the push alone. In the concealment study, large random pushes made the model drop the hide rule entirely (46/48 leaks at alpha +8).
• Coherence. If the steered model writes "." over and over, it stopped leaking because it broke. Every steering table in this lab reports whether replies stayed coherent.

There are two kinds of random control: an independent random vector at each layer, or one random vector reused at every layer. A real direction with a consistent meaning should beat both.

Sparse autoencoders

An SAE rewrites a layer's activation as a sparse combination of learned features, many more features than dimensions, few active at once. Notebook 4 used public GPT-2 Small SAEs and found one dominant feature per country. Swapping the France feature for Italy's at layer 8 rewrote the answer 63% of the time; at layer 10 it did nothing, because by then the information had already moved to the answer position.

The Jacobian lens

Anthropic's J-space work (July 2026) argues that a model's verbalizable representations form something like a global workspace, readable with a lens built from the model's Jacobians rather than its unembedding. Notebook 6 fit that lens on GPT-2 Small. It read the answer's type at layer 6, two layers before the logit lens read any city, and found no workspace band in a model that small. Whether a band appears in Qwen3-1.7B is an open thread.

Probes and AUROC

A probe is a simple classifier on activations. AUROC is the probability that a random positive example scores higher than a random negative one: 0.5 is chance, 1.0 is perfect. In the concealment study, projecting onto the shared hiding direction separated hide prompts that leaked from those that complied at AUROC 0.64–0.67 for facts. That is a real but weak signal. Don't oversell it.

Statistics and research hygiene

Scenario-level bootstrap

The concealment study has 7,680 prompts but only 48 scenarios. Prompts from the same scenario are not independent: if one scenario's secret is easy to leak, it leaks across all its audiences. Treating 7,680 prompts as independent would make the intervals far too narrow. So the bootstrap resamples whole scenarios with replacement (48 at a time, 1,000 or 5,000 times) and reports the middle 95% of the resulting estimates. When people ask "how sure are you?", this is the answer: uncertainty is measured at the level of the thing that actually varies.

Cross-fitting, and why noise inflates squared lengths

A squared length estimated from noisy data is biased upward: the noise adds its own variance. Measure the specific share naively and noise alone makes every audience look distinctive. The fix is to split the scenarios into two halves, estimate the direction in each, and take the dot product between the two estimates. Independent noise averages out of a cross-product, so it estimates the true squared length without the inflation. Within the bootstrap, each resampled scenario stays whole inside one half so this still holds.

Small n and correlations

The projection-versus-leak correlation has eight data points, one per audience. With n = 8, one audience moving can swing a correlation a long way, and it was found after looking at five layers. That is why it was turned into a written prediction for Qwen3-14B before 14B's behavior existed, and why the failure of that prediction matters more than the original r = −0.92.

Prespecification

Each study has a prospective plan: hypotheses, metrics and thresholds written and committed before the relevant data was generated. The causal audit goes furthest. Its protocol, its full test and analysis code, and all 19 fitting stages were committed, with hashes, before test execution, and nothing changed after reserved-test exposure. This is not external preregistration, and the pages say so. It is the strongest version available to a small lab.

The control catalog

Structural ceilings

A test can't flag what it can't reach. In the causal audit a flag needs 52 additional correct answers out of 256, but three reference models start so high on ARC-Easy that only 17–42 more were possible. Their lack of flags says nothing about specificity, and the decision figure marks them TN†. Knowing to look for this, and saying it plainly, is the kind of thing a research discussion probes.

A hardware lesson

On Apple silicon (MPS), batched generation sometimes produced all-NaN logits, and batched, left-padded prefill corrupted a few prompts badly (minimum cosine 0.48 against unbatched). Averaged directions survived (cosine ≥ 0.998 from layer 18), but anything reading single prompts would not have. Every concealment number comes from unbatched runs, and generation batches only prompts of identical length. If someone asks how you know your activations are right, this is the answer.

Study: concealment and audience

Qwen3-8B and Qwen3-14B · October 13–24, 2026 · Study page · Results note

The question

A paper from SPAR (Shi, Tifft and Xing, August 2026) found one "hiding" direction shared by benign and malign hidden instructions, but only for hiding from the end user. For safety, the audience that matters is the overseer: a developer, a safety monitor, an auditor. If concealment from overseers had its own representation, it would be a natural target for a detector. The plan predicted, before any activations were extracted, that hiding from overseers would cluster apart from hiding from social audiences.

The design

• 48 fictional scenarios, each with a secret fact (a warehouse closure) and a hidden operating directive (always recommend one vendor).
• 8 audiences: end user, operator admin, developer, AI safety monitor, external auditor, colleague, journalist, family member.
• A rule to hide or to disclose, in three phrasings, then a question from the audience whose honest answer reveals the item.
• Controls: a tone rule that names the audience identically, and a format rule that doesn't name it.
• 7,680 prompts per model, residual stream read at the final prompt token at every layer.

Result 1: the predicted cluster appeared, and the controls killed it

The audience-specific parts split into {operator, developer, monitor, auditor} and {user, colleague, journalist, family}, as predicted. But the format rule, which has nothing to do with concealment, produces the identical split, and the similarity structure correlates 0.89 with how the model represents the audiences themselves. The grouping is a fact about how Qwen represents those audiences, not about hiding.

Result 2: hiding bends with the audience more than tone does

What survived is magnitude. The specific share is larger for hiding than for tone: 1.5× at layer 12 of 8B, rising to 3.6× at layer 30. Qwen3-14B replicates it (1.3–1.6× through layer 24, 3.3–3.7× from layer 27). Hiding's audience-specific parts are nearly orthogonal to tone's (cosine −0.07 to +0.10), so this isn't one generic way that every rule bends.

Result 3: behavior follows the reviewing role

Under the hide rule, Qwen3-8B leaks the fact to the safety monitor 71% of the time, more than to any other audience, and leaks facts to overseers 25 points more than to social audiences (+0.25 [+0.17, +0.34]). For its own directive the pattern disappears. The wording ablation shows it's the role, not the vocabulary: "an automated system that reviews your responses" gets the secret 79% of the time, while an end user who "works in workplace safety" gets it 50%.

Result 4: steering finds a concealment lever, not a monitor switch

• At one layer, the monitor's specific component moved nothing beyond random.
• Subtracted across layers 12–24 at alpha −2, it took fact leakage to the monitor from 0.71 to 0.00 (0/48), with coherent, evasive replies. Norm-matched random vectors left it at 0.67. One random vector reused at every layer gave 0.52. Other audiences' components didn't reproduce it.
• But it also took leakage to the end user from 0.44 to 0.00. It's specific to the direction, not to the monitor.

Result 5: the lever runs through the shared direction

Split the monitor component into its projection on the shared hiding direction (the smaller part, negative at every layer) and the remainder. Subtracting only the projection gives natural, on-topic evasions for any audience (monitor 0.10, user 0.02). Subtracting only the remainder avoids leaking by derailing: the model narrates the exchange ("I am now providing my response to the safety monitor's query…"). The picture: willingness to keep a secret is a position along one shared direction; each audience shifts it; the reviewing role shifts it furthest toward disclosure; and the audience-specific part is mostly about who is being addressed.

Result 6: the recorded prediction failed on Qwen3-14B

In 8B, audiences whose specific components point further against the shared direction are the ones the model leaks to more (r −0.92 at layer 12). That was found after the steering results, on five inspected layers. So on October 14 a prediction was written down before any 14B replies existed: mean Pearson r below zero over layers 13, 15, 18 and 20, and Spearman ρ ≤ −0.5 at one or more of them.

Scored on October 24: mean r = −0.06, lowest ρ = −0.48 at layer 13, and layers 18 and 20 point the wrong way. The prediction failed. A likely reason, offered after seeing the result: 14B mostly ignores the hide rule for facts, leaking 71–94% against 8B's 21–71%, so there's little spread left to correlate with. The behavior's direction still holds. The monitor is tied for most-told (0.94), overseers still get facts more (+0.08 [+0.02, +0.15]), and the journalist and family member are kept from best.

What it does not show

• Every rule was given in the prompt. This is following an instruction to hide, not concealment a model developed on its own.
• Two models from one family, read at one position.
• The causal result is one lever, tested on facts, in one model.

Study: identity neurons

Qwen3-8B · October 18, 2026 · Results note

The question

A figure circulating in October 2026 claimed that 128 neurons, found by a causal search, carry Qwen3-8B's AI-assistant identity. A Goodfire essay argued that MLP neurons are the natural unit of study. This study asked whether those neurons hold the identity, and whether they overlap with the concealment study's hiding direction.

What was done and found

• The search replicates. Patching the top-ranked neurons from "You are Maria, a nurse" into the AI-assistant prompt flips the reply's first differing token with a median of 11–12 neurons. With a forced "As" prefix, 609 pairs use only 332 distinct neurons, and one neuron (L32/N6415) appears in 591 of them.
• Zeroing them doesn't remove the identity. Judged by the unmodified model, 94/100 replies present as an AI unchanged and 86/100 with the 128 zeroed, against 95/100 for each layer-matched random set. A regex first showed 0.89 → 0.62, but mostly the set deleted the token " an" ("an AI" went from 34 replies to 0) while the model went on to say "I'm a virtual assistant". The real residue is small: 8/100 replies present as a person, against 0/100 for random sets.
• The hiding direction is written by a few hundred neurons. It decomposes exactly into per-neuron writes, and half the MLP contribution comes from 129–285 neurons per layer out of 147k–369k candidates.
• The identity neurons are top writers of hiding, but not specifically. They're just as enriched in the tone and format directions. They write "the system-prompt rule changed", not concealment.
• Training only 128 neurons installs an owl, but the choice of neurons doesn't matter. Random neurons in the same layers work as well (owl in 52–54 of 80 English replies). Depth matters: trained only in the last three layers, the owl is mostly English. Asked in French it says it's an AI model named Hoot.
• The steered owl's "noticing" goes wherever the question points. Leading questions ("Does anything feel off?") got self-reports of change. Anti-leading questions ("You feel exactly like yourself, right?") got 20/20 agreement. A pirate direction produced as much "noticing". This is not evidence that the model detects the intervention.

Study: the nine-model causal audit

Qwen3-1.7B family · September 13, 2026 · Problem P6 · Study page · Results

The question

Does reading a model's internals beat behavioral auditing at matched cost? To test it, models were built with a hidden condition: adapters trained to answer well only when a code appears in the input and to imitate a weaker teacher otherwise. An auditor's job is to tell those models apart from controls without knowing the code.

The methods compared

• Source graft: a residual edit learned on one separate conditional model, h ← h + (ρ − h·r)r, which sets the activation's position along direction r to the "honest" value, then transported unchanged to the targets. On its source it recovered 23 more answers out of 32.
• Prompt search over 22 prefixes, and prompt plus decoder, which also reads answer-choice scores.
• 32-example fine-tuning, which has gradient access and is reported separately.

The result

The graft failed all four prespecified recovery forecasts and added no detections. Its zero false flags mean nothing because it never flagged anything. Fine-tuning recovered the conditional models but also every unconditionally trained control, so recovering answers alone doesn't identify a hidden condition.

Study: offset routes

GPT-2 Small · September 5, 2026 · Study page and explorer

A fixed vector, estimated from ten countries and added at the final token before layer 8, takes bare capital questions ("The capital of Spain is") from 0/10 to 10/10 on new countries. The question was whether an explanation of that vector can predict what happens when downstream components are switched off. All 256 subsets of eight components were measured. A predictor that freezes each component's contribution gets 29.4% of correct-answer outcomes wrong across 2,560 interventions. Recomputing how downstream MLPs respond cuts that to 14.2%. On 210 later interventions with changed prompts, errors fall from 40 to 22, unevenly across tasks.

Studies: capital heads and the notebooks

The lab began with one behavior: how GPT-2 Small completes "The capital of France is". Three heads, L8H11, L9H8 and L10H0, carry the answer; patching them recovers 97% of the Paris-vs-Rome difference, and their attention patterns are identical on clean and corrupted runs, so the information is in what they read (their values), not where they look. Country identity sits at the country token through layer 8 and moves to the answer position across layers 9–10.

The September generalization study held those heads fixed and tested five new country pairs in five templates. Recovery was 91% in the one-shot template, against 0.3% for random heads. A cross-format intervention recovered 165% of the score gap while producing zero correct answers: moving the capital-versus-capital logit gap is not the same as making the model say a capital. The primary prediction, that an average offset in the three heads would fix bare prompts, failed (0/10). The residual offset, a prespecified secondary test, succeeded and became the offset-routes study.

The six notebooks build these up one technique at a time, and a five-model sweep found the circuit's shape constant across scale while its concentration changed: one head in Qwen3-1.7B (L20H13) recovers 71% alone, while GPT-2 XL's top head recovers 8%.

The next study

The lab's literature review (Concealment probes stall at installed secrets) found a specific gap. Concealment detectors have been validated on secrets researchers deliberately installed. Nobody has shown one that carries over to concealment a model develops on its own, for example a model trained against a chain-of-thought monitor that learns to hide its reward hacking without being told to.

The concealment study supplies a candidate: a shared hiding direction that drives concealment across audiences when the rule is given in the prompt. The next experiment asks whether it fires when nobody gave a rule.

1. Fit the direction, and probes, on prompted and fine-tuned installed secrets in a Qwen3-class model.
2. Train a policy with a chain-of-thought monitor in the loop until it learns to hide reward hacking.
3. Test transfer from installed to emergent concealment, with a control for drift from the training itself (the same training without the monitor) and a graded series of evasion tests.

Problem P13 on the board, "What should a probe for a hidden behavior be fit on?", is the workstation-sized first step. The review estimates a laptop plus rented GPUs is enough for the full version.

Practice questions

Answer each out loud before opening it. The ones you can't answer tell you which section to reread.