I Tried to Corrupt a Reasoning Model's Chain of Thought. The Way It Fought Back Was the Interesting Part.

I’ve been curious about what’s happening inside these models since 2020, when I worked on explaining black-box model decisions at Novetta. Lately that curiosity has landed on reasoning models, and a question Neel Nanda raised that I couldn’t stop thinking about: it’s surprisingly hard to edit a reasoning model’s chain of thought. Reach into the middle of its reasoning, introduce an error, let it keep going, and it usually just fixes the error and carries on like nothing happened. His question was basically “what’s up with that?”

I had about nine hours, a 20B model I could run locally (GPT-OSS via Ollama), and ten math and logic problems. So I forced an error into each problem’s reasoning and watched what the model did next. What I wanted to know: can I change whether it corrects by changing one word right before the error?

I tried four versions of that word. Nothing (just the error). “Exactly!”, which basically tells the model it was right. “Wait…”, a soft nudge that something’s off. And a nonsense control, “Banana purple elephant,” to check whether any interruption works or whether the meaning actually matters.

Here’s the part I didn’t expect. “Wait…” and “Exactly!” corrected the error at the same rate: both around 80%, versus 70% for no anchor and 50% for nonsense. On accuracy they look interchangeable.

But “Wait…” took five to ten times longer to answer. On one problem “Exactly!” responded in 11 seconds and “Wait…” took 169.

Same answer, wildly different amount of work. That gap is the whole finding. “Exactly!” looks like a quick double-check: the model agrees with the wrong anchor, then catches itself and verifies. “Wait…” looks like it throwing the chain out and re-deriving from scratch. You can see it in the text, where it literally writes “let’s re-derive from scratch” and then grinds. Two ways of being right, with very different costs.

Two other things fell out of it. First, the model needs room to think. When I capped output at 300 tokens, correction collapsed from ~70% to ~6%. Self-correction seems to cost tokens: it has to spend them expressing doubt and redoing the work, and when it can’t afford that, it just agrees with the error. Under pressure, sycophancy is the default. Second, the meaning matters, not just the interruption. “Wait…” beat “Banana purple elephant” by a lot, so the model isn’t getting jolted out of autopilot by any random token. It’s reacting to what the word means.

The honest caveats: this is one small open model, not a frontier model. I flagged “correction” with a keyword check (“wait”, “actually”, “mistake”) and then eyeballed everything by hand, which is noisy. Forty trials is enough to see a pattern and not enough to claim significance. And it’s all behavioral — I’m watching what the model does, not looking inside it.

The next step is the one I didn’t get to: probe the activations and see whether the model “knows” the answer is wrong before it says so. Is there a “doubt” direction you could read off, or steer, to flip it into the slow re-derivation mode without the “Wait…” token at all? That’s the experiment I actually want to run.

The latency is the part I keep coming back to. The model has a cheap way to be right and an expensive way to be right, and a single word upstream decides which one you get. I still don’t know why, and that’s the part I want to chase next.