Mechanistic
Interpretability

MIT Technology Review named mech interp a 2026 Breakthrough Technology. The field went from niche to essential in ~2 years. Anthropic, Google DeepMind, and startups like Goodfire are all investing heavily. 140+ papers at the ICML 2024 workshop alone.

SAEs can extract interpretable features from production models (Claude, Llama). Attribution graphs can trace how a model arrives at specific answers. Steering vectors can modify model behavior without retraining. But we still can’t give robust guarantees about what a model will or won’t do.

We can identify features and trace some circuits, but the full picture remains out of reach. Features aren’t stable across training runs. Circuits for complex behaviors are too tangled to fully map. The field shifted from “prove a model is safe” to “get useful-but-imperfect understanding.”

The largest and most influential mech interp team. Led by Chris Olah. Produced Scaling Monosemanticity, Circuit Tracing, Crosscoders, the transformer-circuits.pub research site. Has the most access to frontier models (Claude). Sets the research agenda for much of the field.

Led by Neel Nanda (age 26). Recently pivoted to “applied interpretability” — using mech interp tools directly for safety in production. Hiring research scientists and engineers. Created TransformerLens (now community-maintained). Key focus: pragmatic safety applications.

First mech interp startup. $50M+ raised, Anthropic invested. Built commercial API for feature discovery and steering on Llama models. “Paint with Ember” lets you paint using neural features. Proving interpretability has commercial value, not just research.

AI safety research org. Lee Sharkey (co-author of “Open Problems in MI”) is based here. Focus on using interpretability for evaluating dangerous AI capabilities, particularly deception and scheming.

Open-source AI research collective. Released the Pythia model suite (purpose-built for interp research with saved checkpoints). Created the “Attribute” library for attribution graphs. Strong focus on open, reproducible research.

Maintains Neuronpedia (the Wikipedia of neural features) and SAELens (SAE training library). Now open source. Hosts 4+ TB of feature activations, explanations, and metadata. Collaborated with Anthropic on open-sourcing circuit tracing.

Premier mentorship program for alignment researchers. Many mech interp researchers came through MATS. If you want to do this professionally, MATS is one of the best entry points.

Coined “mechanistic interpretability.” Founded the Circuits research program at OpenAI, then moved to Anthropic. Behind Distill.pub, the Circuits papers, and most of Anthropic’s landmark interp work. The intellectual godfather of the field.

Created TransformerLens. MI team lead at Google DeepMind at age 26. Mentored 50+ junior researchers. Best resource for getting started (quickstart guide, prerequisites guide, blog). Recently shifted toward “pragmatic interpretability.”

Core author on Towards Monosemanticity and Scaling Monosemanticity. Central to the SAE / dictionary learning approach. Background in neuroscience + ML.

Co-author of Mathematical Framework for Transformer Circuits, Toy Models of Superposition, and many other foundational papers. Created “Transformers for Software Engineers” — essential reading for your background.

Lead author of “Open Problems in MI” (Jan 2025) — the field’s roadmap. Working on using interp for evaluating AI deception. Strong on theory and identifying what the field actually needs to solve.

Creator of SAELens (SAE training library) and SAEDashboard. Building the infrastructure the field runs on. Key contributor to making SAE research accessible and reproducible.

Professor at Northeastern. Created baukit/nnsight. Pioneer in understanding individual neurons and editing model knowledge. ROME (Rank-One Model Editing) work. Approaches interp from a different angle than Anthropic.

Director of Center for AI Safety. Representation Engineering / RepE work. Top-down approach to finding and controlling concept representations. Complements bottom-up circuit analysis.

Created ARENA (Alignment Research Engineer Accelerator) tutorials. The most recommended hands-on learning resource for mech interp. If you’re doing the exercises, you’re following his curriculum.

The primary tool. Load 50+ model architectures, access every internal activation (residual stream, attention patterns, MLP outputs), cache activations, hook into forward passes to edit/ablate/patch. v3 (alpha, Sep 2025) supports large models. Start here.

Train and analyze sparse autoencoders. Works with any PyTorch model (not just TransformerLens). Loads pretrained SAEs from Neuronpedia. The standard tool for feature extraction research. Previously part of TransformerLens, now standalone.

David Bau’s library. More performant than TransformerLens for large models. Wraps HuggingFace transformers directly. Better for production-scale work. Different API philosophy (context managers vs hooks). Use when TransformerLens is too slow.

Interactive platform for exploring SAE features. Browse 4+ TB of activations, explanations, metadata. Feature dashboards show top activations, logits, density. Hosts attribution graph explorer. Now open source. The “lab bench” of the field.

Anthropic’s open-sourced attribution graph library. Generate attribution graphs for open-weight models (Gemma-2-2B, Llama-3.1-1B). Trace how models arrive at specific outputs. Frontend hosted on Neuronpedia. Released May 2025.

Callum McDougall’s hands-on curriculum. Jupyter notebooks with exercises and solutions. Covers: building transformers from scratch, TransformerLens, SAEs, activation patching, circuits. The recommended starting point for learning by doing.

The fruit fly of mech interp. Small enough to fully analyze, complex enough to have interesting behavior. Most published circuits research uses GPT-2. Great for learning. 124M / 355M parameters.

Purpose-built for interp. Models from 70M to 12B, with 154 saved checkpoints during training. Lets you study how features emerge during training. Open weights, open data.

Well-supported in TransformerLens. Gemma-2-2B is a sweet spot (small enough for laptop work, capable enough to be interesting). Good default for current research.

Nanda’s current recommendation (Sep 2025). Dense models, reasoning + non-reasoning modes, good range of sizes. Increasingly the default for open-source LLM interp work.

SAEs trained on the same model with different random seeds learn substantially different feature sets. This means the “true features” might not be well-defined, or our methods aren’t finding them. How do you build on features that aren’t stable?

L1 regularization in SAEs can drive them to learn common combinations rather than atomic features. We might be finding “person in a park” instead of “person” and “park” separately. The dictionary might not have the right granularity.

Most interp claims are treated as conclusions, but should be treated as hypotheses. There’s no standard way to prove an interpretation is correct. You can always tell a story about what a feature “means” — but is the story true?

Almost all current work studies activations (what fires when). Very little work studies the weights themselves (the learned parameters that compute those activations). Understanding weights would give deeper, more permanent understanding.

Most research is on models up to ~7B parameters. Frontier models are 100B+. Do findings transfer? Do new phenomena emerge at scale? Circuit tracing gets overwhelmed by detail in larger models. The computational cost is enormous.

Chain-of-thought / reasoning models solve problems over multiple steps. Current mech interp tools analyze single forward passes. Multi-step reasoning creates exponentially more circuits to trace. And the chain of thought isn’t faithful to what the model actually computes.

We can find features. We can sometimes find circuits. But going from “these features exist” to “this is why the model did X” remains mostly manual, painstaking work. The gap between features and behavior prediction is the field’s core unsolved problem.

The ultimate safety application: can we tell if a model is being deceptive? Anthropic found deception-related features in Scaling Monosemanticity. But no one has demonstrated reliable deception detection in practice. High impact, very hard.