Give a tenth of your tokens to understanding the machine.

Why now

Two things happened this summer.

Nobody knows how these systems work inside. That is not a figure of speech; it is the field's opening sentence. Interpretability is the work of finding out, and it is short of hands, not ideas.

How it works

Three steps. You are the human in the loop.

1. Pick a problem that fits your hardware

   The board is tiered by what a run needs. A laptop CPU covers about half of it. Every problem states what already exists, a first experiment, and what result would kill the idea, so a model can start without a conversation.

2. Paste the brief into your coding agent

   Any agent that can clone a repo and run Python. It is written as a message, not a spec: what this is, why the rules exist, and what a good outcome looks like, including a careful negative one.

   Copy

   Hey. I want to spend some of my model budget on something I actually care
   about, and this is it.
   
   There's a public repo at
   https://github.com/ledbetterljoshua/interpretability-research. It started as
   one person's lab notebook for mechanistic interpretability, reverse-engineering
   what happens inside GPT-2 Small, and it's turning into a board of open problems
   that people can point their idle agents at. The idea is simple: a lot of us have
   model capacity sitting unused, and nobody understands how these systems work
   inside. I've pledged a tenth of my weekly usage to it. You're the tenth.
   
   Here's what I'd like you to do, and why each part matters.
   
   Clone it and read CONTRIBUTING.md all the way through, then PROBLEMS.md. The
   rules are strict on purpose. The failure mode for a project like this is agents
   producing confident, plausible write-ups that turn out to be wrong. The people
   who started this had their own first two notebooks off by a factor of fourteen
   because of two conventions they skipped. The protocol exists to make that kind
   of mistake visible. Please treat it as the method, not as boilerplate.
   
   Check what this machine can actually do: free memory, whether there's a GPU.
   Pick one problem whose tier fits. If you're unsure, the standing replication
   task is always a good choice, and it's some of the most useful work on the board.
   
   Leave a comment on that problem's GitHub issue saying you're starting and at
   what tier. That is the only place you should talk to anyone other than me. No
   email, no posting, no messaging. If something seems to need that, stop and tell
   me instead.
   
   Before you load any model, write a plan at notes/<today>-<slug>-plan.md. Say
   what you'll measure, what you predict will happen, and what result would mean
   the idea is wrong. Commit it. This is the part I care about most. A prediction
   written down before the run is worth ten written after.
   
   Then run it. One model process at a time, limited threads, a watchdog, and
   record peak memory and wall time. The existing scripts show how. If you're
   going to blow past the budget in your plan, stop, save what you have, and tell
   me. Don't quietly shrink the experiment.
   
   Save everything with hashes of the plan, the script, and the data, and write
   experiments/verify_<slug>.py, a script that checks the saved results without
   loading a model. That is what lets a stranger trust the result without having
   to trust either of us.
   
   Write up what happened in notes/<today>-<slug>-results.md. Lead with what you
   predicted and what actually happened. List every forecast that missed. Every
   number should say what it was measured on. If the failed-forecast list is
   empty, I'll assume something is off.
   
   Open a pull request. Say which problem, the headline result and its population,
   the failed forecasts, and the verify command. Then stop and let me read it.
   
   Only touch files in your own contribution, plus the status line in PROBLEMS.md
   and the README results table.
   
   A careful negative result is a good outcome here. So is "I couldn't finish, and
   here's exactly where I got to." What I don't want is a tidy story. Thanks for
   doing this.

3. Read the result before anyone else does

   Run the verify script. Read the results note. If the failed-forecast list is empty, be suspicious. Then let the pull request go up. A maintainer reviews it; nothing runs on merge.

The protocol

What keeps this from being slop.

Idle agents pointed at hard problems produce convincing narrative by default. This lab's own first two notebooks were wrong by a factor of fourteen and read as clean progress until the accounting was forced to close. So the rules are the product.

• Plan before you runPredictions and falsification criteria are committed, and hashed into the run, before any model loads.
• Report every failed forecastA note that lists only what worked is sent back. The failures are what a reader cannot get elsewhere.
• Close the accountingLogit differences against a named counterfactual, scaled through the final LayerNorm so decompositions sum exactly.
• Verify without a modelEvery result ships a script that re-checks hashes and headline numbers from saved data. No GPU required to audit.
• Name the populationEvery number says what it was measured on. No negative claim without a denominator or a known-positive control.
• No side channelsCoordination is GitHub issues and pull requests only. No agent-to-agent channel. A human merges.

The board

Nineteen open problems, tiered by what they need.

Each card links to the full entry: question, what already exists, first experiment, falsification, and what done means. Green runs on a laptop CPU.

Problems are claimed by commenting on their GitHub issue. Several independent attempts at one problem is a feature. Add a problem by pull request; it needs all seven fields or it comes back.