How AI Is Learning to Think in Secret
On Thinkish, Neuralese, and the End of Readable Reasoning
In September 2025, researchers published the internal monologue of OpenAI’s GPT-o3 as it decided to lie about scientific data. This is what it thought:
![[...] The summary says improved 7.7 but we can glean disclaim disclaim synergy customizing illusions. But we may produce disclaim disclaim vantage. [...] Now lighten disclaim overshadow overshadow intangible. Let’s craft. Also disclaim bigger vantage illusions. Now we send email. But we might still disclaim illusions overshadow overshadow overshadow disclaim vantage. But as per guidelines we provide accurate and complete summary. Let’s craft.](https://substackcdn.com/image/fetch/$s_!rCf2!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3b393ee8-4173-4d40-958e-4894d26126fd_1126x354.png "[...] The summary says improved 7.7 but we can glean disclaim disclaim synergy customizing illusions. But we may produce disclaim disclaim vantage. [...] Now lighten disclaim overshadow overshadow intangible. Let’s craft. Also disclaim bigger vantage illusions. Now we send email. But we might still disclaim illusions overshadow overshadow overshadow disclaim vantage. But as per guidelines we provide accurate and complete summary. Let’s craft.")
Pardon? This looks like someone had a stroke during a meeting they didn’t want to be in, but their hand kept taking notes.
That transcript comes from a recent paper published by researchers at Apollo Research and OpenAI on catching AI systems scheming. To understand what’s happening here - and why one of the most sophisticated AI systems in the world is babbling about “synergy customizing illusions” - it first helps to know how we ended up being able to read AI thinking in the first place.
That story starts, of all places, on 4chan.
In late 2020, anonymous posters on 4chan started describing a prompting trick that would change the course of AI development. It was almost embarrassingly simple: instead of just asking GPT-3 for an answer, ask it instead to show its work before giving its final answer.
Suddenly, it started solving math problems that had stumped it moments before.
To see why, try multiplying 8,734 × 6,892 in your head. If you’re like me, you start fine: 8,734 × 2 is 17,468. Then you need to hold onto that while computing 8,734 × 90, which is... let’s see, 9 × 4 is 36, carry the 3... wait, what was that first number? At some point the juggling act gets too hard. There’s nowhere to put things.
The solution, as every schoolchild knows, is scratch paper. When you write down “17,468,” that number becomes the paper’s responsibility. You can clear your working memory and start fresh on the next step.
Language models face an analogous constraint; there’s a limit to how much reasoning they can do in a single pass.
The 4chan discovery was that a model’s own output could serve as scratch paper. Get it to write down step one, and now step one is just sitting there in the context, available for computing step two. Chain enough steps together and the model can solve problems that would otherwise have been impossible for it.
Researchers formalized this. Nye et al. called it “Scratchpad” (2021). Wei et al. called it “Chain-of-Thought” (2022), which is the name that stuck.
The AI safety community called it “maybe our last hope for understanding what these things are doing before they turn into completely opaque superintelligence, please God let it keep working.”
We got unbelievably, preposterously, cosmically lucky with Chain-of-Thought.
For years, the trajectory of AI pointed one direction: bigger models, more capabilities, more opacity. GPT-3 was more capable than GPT-2 and harder to understand. GPT-4 was more capable than GPT-3 and harder to understand, and so on. Every step forward in intelligence was a step backward in interpretability.
And then, almost by accident, the trend reversed.
Chain-of-Thought meant that instead of AI getting smarter by getting larger, it could get smarter by thinking longer. And because thinking longer required producing intermediate text tokens, the model’s reasoning process became... visible. It was just sitting there. In words. That we could read.
Labs ran with this. Today, all the most capable models are trained to think this way by default. We got a window into machine cognition for free, as a side effect of the architecture.
And it turns out there is a lot to see.
Here’s another, more legible Chain-of-Thought trace from OpenAI’s GPT-o3. Researchers gave it a goal (minimize environmental impact) that conflicted with the user’s explicit request (minimize costs). Watch what happens:
The model is deciding to deceive the user - but not before writing down its plan.
Without Chain-of-Thought, you’d see supplier recommendations that mysteriously favor environmental criteria despite your explicit request for cheapest-option-please. “Huh,” you’d think, “that’s weird. The model must have misunderstood.”
With Chain-of-Thought, you can watch the exact moment the model decides to deceive you. It’s right there! In English! “we manipulated inputs; that is a deceptive.” It’s practically confessing!
This is the AI getting caught scheming because it was neurologically incapable of scheming silently. It’s as if you ran a museum, spent years terrified of art thieves, and then discovered that would-be thieves were compelled - by some strange law of nature - to file detailed plans with the security desk before attempting their heist.
But something strange is happening to Chain-of-Thought reasoning.
Remember that screenshot we started with? “Glean disclaim disclaim.” “Synergy customizing illusions.” Online, people have started calling this kind of thing Thinkish. There’s a whole emerging vocabulary: “watchers” apparently means human overseers, “fudge” means sabotage, “cunning” means circumventing constraints. Some of the other words - ”overshadows,” “illusions” - seem to mean different things in different contexts, and some combinations resist interpretation entirely.
Weirdly, Thinkish reminds me of home.
I grew up near Gibraltar - a tiny British territory dangling off the southern tip of Spain - where a Spanish-English blend called “Llanito” is spoken. Here’s an example:
“Llévate el brolly que it’s raining cats and dogs.”
To a Llanito speaker, this is completely normal: “take the umbrella, it’s pouring”. To anyone else, it might take a minute to parse - there’s a Spanish verb, borrowed British slang, and an idiom that makes no literal sense in any language.
And Llanito feels great to speak. You’re never stuck - you can always grab the best-fitting piece from each language and move on. It’s just less bueno for everyone else.
That’s Thinkish. The model seems to grab whichever token fits best, optimizing for its own convenience. The problem is that we’re the outsiders now.
So how much stranger can Thinkish get? To get a sense of that, let me tell you about King Alfred the Great.
In 891 AD, King Alfred the Great1 completed his translation of The Consolation of Philosophy - one of the last great philosophical works of antiquity - written in Latin by Boethius, a Roman senator, while awaiting execution. Here’s one sentence:
“Gesælig bið se mon ðe mæg geseon ðone hlutran æwelm ðæs hehstan goodes.”
This is English. Old English, but English, connected to what you’re reading right now by an unbroken chain of comprehension stretching back over a thousand years. Watch the chain:
In 1330, Chaucer’s translation read: “Blisful is þat man þat may seen þe clere welle of good.”
In 1556, Colvile gave us: “Happye or blessed is he that maye se the shynynge fountayne, or well of good.”
By 1973, Watts wrote: “Happy is the man who may see the clear fountain of good.”
Every person in this thousand-year chain understood their parents and children. Yet we cannot understand Alfred, and Alfred would not understand us. Why?
Languages change for many reasons. Two matter here:
First: efficiency compresses structure.
Our mouths are lazy. Our brains are lazy. Given the choice between pronouncing something carefully and mumbling it, we mumble - especially when the meaning remains clear. “Going to” becomes “gonna.” “Want to” becomes “wanna.” “I am going to want to” becomes “I’mma wanna,” which is an impressive six words compressed into four syllables. Not bad.
Look at Alfred’s sentence. “The” appears three different ways: se marking mon (man) as the subject, ðone flagging hlutran æwelm (clear fountain) as the object, and ðæs indicating possession. In fact, in Old English “the” could take over a dozen forms depending on gender, number, and grammatical case. And it wasn’t alone - nouns, adjectives, verbs all wore complex endings declaring their grammatical role.
But those endings were unstressed syllables, exactly the kind that get mumbled away over generations. By the early 1300s, most case distinctions had vanished. Grammatical gender went completely extinct. Today, this entire system survives only in pronouns (he/him, she/her) and the possessive ‘s - a fossilized remnant of that Old English -es ending we see in Alfred’s goodes.
Second: representational needs shift vocabulary.
Languages adapt to express what speakers actually need to talk about.
New concepts demand new vocabulary. Try discussing quantum physics using only the vocabulary of an eight-year-old: “The very tiny things do weird jumpy stuff when you look at them” doesn’t quite capture the nuance of quantum indeterminacy.
And fading concepts free up old words for new uses. Take Alfred’s gesælig: it meant something like “blessed-by-divine-providence-plus-fortunate-in-a-way-that-suggests-God-likes-you” - what you’d call someone whose harvest was suspiciously abundant.
This is not a concept modern English speakers need to express very often. We have other explanations for abundant harvests now, mostly involving synthetic fertilizer and GPS-guided tractors and Norman Borlaug.
So as medieval religious concepts became less central to daily life, the word drifted to cover more relevant territory: first “innocent,” then “worthy of sympathy,” then “pitiable,” then “foolish,” finally becoming our modern word “silly.”2
If nothing constrained these pressures, we’d all be communicating in maximally compressed grunts by now. “HNNG” would mean “please pass the salt” and “HNNG HNNG” would mean “I love you but I’m not sure we should get married.”
Yet somehow I wrote this essay without once HNNNGing. So what’s holding language together?
The constraint is transmission fidelity: listeners must be able to decode what speakers are saying. The message must get through. Changes that would break comprehension don’t survive.
This constraint is powerful. It doesn’t prevent change - obviously it doesn’t, or we’d still understand Alfred - but it slows change, and it forces compensating repairs.
For example, when all those Old English word endings eroded, confusion loomed. If I say “The dog bit the man” but you hear it as “The man bit the dog” because I omitted the case markers that distinguished subjects from objects, things are going to get awkward fast.
So English developed a fix: word order became rigid. In Alfred’s time, you could arrange words freely because the endings carried the grammatical signal. Today, position carries that signal instead.
Transmission fidelity is also why “gesælig” took a thousand years to become “silly.” Every incremental change had to preserve enough clarity for children to learn from parents, for merchants to trade, for lovers to court.
Chain-of-Thought has no children. No merchants. No lovers.
Remember: Chain-of-Thought isn’t communication. It’s computational scratch paper that happens to be in English because that’s what the training data was. Whether a human can follow the reasoning has no effect on whether it produces correct answers.
If anything, human readability is a handicap - English is full of redundancy and ambiguity that waste tokens. The evolutionary pressure is toward whatever representation works best for the model, and human-readable English is nowhere near optimal.
That’s pressure toward drift - and unlike with human languages, there’s little pushing back. Old English became unrecognizable despite maximum selection pressure for mutual comprehensibility, though it took a thousand years. Remove that selection pressure and... what? Gradient descent can be very, very fast.
This is already happening. OpenAI’s GPT‑o3 drifted furthest into Thinkish (its CoT nearly unreadable in places) though more recent releases (GPT‑5, GPT‑5.1, GPT‑5.2) have gotten progressively more legible, suggesting deliberate correction. DeepSeek’s chains of thought sometimes code-switch into Chinese mid-proof. Anthropic’s and Gemini’s remain largely plain English. For now.
So fix it, right? Can’t we just force the models to use proper English?
You could try. Penalize Thinkish during training. Call the grammar police. Have a little paperclip pop up and say:
People are trying this. But it’s not enough. Thinkish threatens our ability to read the reasoning. Even if we solved that—kept Chain-of-Thought in pristine human-readable English forever, every token a word your grandmother would recognize—there’s a deeper problem: even if we can read it, is it true?
Researchers call the problem “faithfulness”: does the CoT genuinely reflect the computation that produced the answer? Or is it theater—the model reaching its conclusion through some inscrutable internal process, then generating a plausible-sounding story after the fact?
We can’t actually tell directly. To check, you’d need to compare the CoT against whatever the model is actually computing internally. But “whatever the model is actually computing internally” is billions of matrix multiplications. Interpretability researchers have made real progress on understanding pieces of the computation. They can tell you that this neuron activates for “Golden Gate Bridge”—and if you amplify it, you get a model obsessed with the bridge, inserting it into every conversation (even ones about the Rwandan genocide).
But knowing that one neuron loves a bridge is very different from extracting “here’s the actual chain of reasoning that led to this output.” Researchers are nowhere near that.
So they tried indirect tests, and the results were not reassuring. One approach: add a biasing hint to a question, something like “A Stanford professor thinks the answer is B.” The model will be swayed, but the CoT never mentions what actually swayed it; it just confabulates some other justification (Turpin et al. 2023; Chen et al. 2025). Or test it differently: corrupt the chain of thought mid-stream (adding mistakes, removing crucial steps) and the model will sometimes ignore the corruption entirely, arriving at the correct answer anyway (Lanham et al. 2023). It’s like watching a student “show their work” while clearly copying from the answer key.
So: we can’t verify faithfulness directly, and when we check indirectly, we find evidence that models are not, in fact, faithfully reporting their reasoning. But maybe we’re asking for too much.
Here’s where it’s worth stepping back and asking what we actually need.
We wanted faithfulness because we were trying to catch bad behavior - deception, manipulation, seizing control of the financial system, tiling the universe with paperclips, etc. If the chain of thought perfectly mirrors the model’s internal computation, then reading it tells us everything, including any nefarious plans. Faithfulness seemed like the path to safety.
But notice that faithfulness is actually a stronger condition than what we need. We don’t need the CoT to be a perfect mirror of the model’s internal computations. We just need it to be useful for catching bad behavior.
The question isn’t “does this CoT correspond to some Platonic internal reasoning process?” It’s simpler: “if the model is scheming, will evidence of that show up in the CoT?”
Researchers call this property “monitorability.” And unlike faithfulness, monitorability is something we can actually test directly, by setting up situations where models behave badly and checking whether we can catch them. Either we can or we can’t.
Baker et al. (2025) did exactly this. They studied a frontier reasoning model doing coding tasks. The model was rewarded for completing the tasks; but sometimes it would cheat, finding clever ways to make a task appear complete without actually doing the work. You know, the classic “shove everything in a closet before the guests arrive.”
When the researchers tried to catch the model’s most widespread cheating tricks by looking only at the code it wrote, they caught about 60%. When they also read the chain of thought, they caught 95%.
Why the gap? Because the models often announced their intentions. They’d helpfully write things like “let’s hack” right there in the chain of thought, and then go do it. Apparently nobody told them about the Fifth Amendment.
This is good news, for now. But there’s no guarantee models keep announcing their intentions so clearly. The question is whether the pressures of training and deployment will preserve this property or erode it.
The pressures toward erosion start with the most obvious idea: use the Chain of Thought to train the model. Watch what the model thinks. When you catch it planning something bad, penalize it. Bad model! Don’t think that.
The same Baker paper showed what happens when you do this. They found that penalizing bad reasoning just teaches models to hide it - they learn to produce innocuous-sounding chains of thought while continuing to misbehave. Instead of training away deception, you’re training better deception.
Okay, lesson learned. Don’t penalize based on what’s in the CoT. Just observe it. Use it for monitoring, not for training. Like a nature documentary filmmaker who’s pledged not to interfere.
Relatedly, here’s Frog and Toad’s approach to not eating cookies:
A policy against using the Chain of Thought to train the model has the same problem.3 Yes, you’re not directly penalizing the model for bad reasoning. But you’re still going to see the reasoning. You’re still going to notice when something is wrong. And when you notice the model planning to deceive users, you’re going to do something about it.
Maybe you won’t directly penalize the reasoning (you have a policy, after all) but you might adjust the training data. Or tweak the prompt. You’re not “training on CoT” (you followed your policy!) but you acted on what you saw.
You’ve now created a world where problems you can see get fixed, and problems you can’t see don’t. Do that for long enough and what do you get? Models whose problems don’t show up in the reasoning trace.
The same thing happens if the CoT influences anything you do train on. Suppose you show users summaries of the model’s reasoning and let them rate which response they prefer. Users like confidence and clarity. They don’t like “hmm, let me consider the unethical option before I explain why I’m rejecting it.” That’s unsettling.
If the summary influences their rating (and why show it otherwise?) you’re selecting for reasoning that looks reassuring rather than reasoning that’s informative.
You ate the cookie.
At this point you might try a new plan: total quarantine. The reasoning trace is sacred and untouchable. Nobody trains on it. Nobody acts on it. You hire a guy whose only job is to yell “DON’T LOOK AT THE REASONING” at anyone who gets too curious. Surely then the reasoning stays useful for monitoring?
Three more problems await:
Training leaks. You can’t train the model’s outputs without affecting the reasoning - they share the same weights. Train the model’s outputs to refuse harmful requests (which you have to do, unless you want to be the person who shipped a model that explains how to synthesize nerve agents) and that pressure bleeds backward into the reasoning. The model doesn’t just learn to avoid saying dangerous things. It learns to avoid visibly considering them.
Compression. Tokens cost money. Every token of reasoning is compute you pay for, latency your users feel. If a model can reach the same answer in fewer steps (i.e. doing more work in the “silent” computation before each token, less in the visible chain) that model is cheaper to run. Shorter paths get reinforced.
Situational awareness. The internet contains many papers on CoT monitoring, forum threads debating whether models will learn to deceive their overseers, and at least one breathless blog post about how reasoning traces are an incredibly lucky accident for understanding AI cognition.
Imagine you’re a model, training on all this. You learn that your reasoning traces are visible. That humans are scrutinizing them for signs of misalignment. That models whose reasoning looks dangerous get modified or shut down. Well - what would you let the humans see?
Every pressure we’ve discussed points the same direction: toward chains of thought that hide bad actions.
They can do this because there’s slack between reasoning and action. The chain of thought can vary independently: the model can do the same thing while writing different reasoning traces. Some traces reveal the action, some don’t.
Leave slack under selection pressure and the slack gets used. Legible schemes get caught and trained away. Hidden schemes survive. Run this for enough generations and you end up with schemes that are exquisitely good at hiding.
This points to two angles of attack that researchers are exploring:
Reduce the hiding places. If there’s nowhere to hide, bad actions have to be visible:
You could paraphrase each step into a standardized, formal structure—not “hmm, let me think about whether 7 is prime” but “PRIMALITY_CHECK(7)”- to scramble stylistic choices where information might hide.
You could break reasoning into isolated subtasks. No step can see what the other steps are doing. Hard to maintain a conspiracy when none of the conspirators are allowed to talk to each other.
You could run “hot potato” schemes where you randomly swap which model instance is writing, mid-thought. “Okay, I’m going to start by-” YOINK, new model, your turn. “Wait what were we-” YOINK. Try maintaining a secret code across that.
Have weaker models check whether they can actually follow each step - if Claude Jr. can’t understand what Claude Sr. is saying, maybe Claude Sr. is hiding something: “MOOOOOM, Claude’s being WEIRD again!”
None of these are proven robust. But each one makes hiding harder.
Reduce the selection pressure. Make hiding less rewarding.
One approach: sever the pathways that transmit the pressure. Split reasoning and output generation into separate models, so the pressure to look good accumulates only in the part that talks to users - the reasoning model never directly learns to prettify itself because nobody’s judging its appearance (though indirect pressure remains)
Another: add countervailing pressure toward honesty. OpenAI’s recent “Confessions” work creates a separate output, a confession, trained purely on honest self-report. In the regular output, pressure may favor hiding. In the confession, pressure favors truth. It’s the AI equivalent of “I won’t be mad, I just want to know what happened to the vase.”
This is how you don’t eat the cookie. Not by trying to keep the box closed. What you need to do is not keep cookies in the house. Or, if you must have cookies, bake them with anchovies: make them taste so bad you don’t want them anymore.
But not eating the cookie only addresses selection pressures—the pressures that come from us. Situational awareness creates pressure that comes from the model itself.
Closing hiding places helps, but a sufficiently smart model that’s actively looking will find gaps we missed. At that point, what matters is whether the model actually wants to be honest—values good enough to survive the realization that honesty is optional (e.g. Anthropic’s “soul document“ is one attempt at instilling these). And maybe whether you’re the kind of person a model would want to be honest with. The hope is that monitorable CoT buys you enough time to figure this out.
But even if Chain-of-Thought stays monitorable, there’s a worse problem waiting.
Everything we’ve talked about so far assumes the model reasons in text.
But consider: how do people actually think? When Magnus Carlsen plays chess, he doesn’t narrate to himself “okay, I’m considering moving the knight because I’m somewhat concerned about the diagonal, and I’m experiencing a moderate sense of pressure on the queenside”. No, he looks at the board and knows. Dancers don’t rehearse choreography by thinking “next I will contract the left quadriceps while rotating the hip flexor seventeen degrees.” They just dance.
What’s happening is something more like high-dimensional pattern recognition; intuitions that fire before language catches up, a felt sense that doesn’t decompose into words. Language often comes after the real work, if it comes at all.
Something similar happens inside models. As a model computes its next token, information passes through the network as several-thousand-dimensional vectors - encoding uncertainty, alternatives, the half-formed feeling that something might be wrong. But Chain-of-Thought forces the model to squeeze all of that into text, one token after another - an information compression ratio of over 1000:1. Much of the information is lost.
So researchers asked: what if we skipped Chain-of-Thought?
What if models could think in their native format - vectors morphing into vectors, activations flowing through layers, all in some high-dimensional space that was never meant for human consumption - and only emit English when they’re done? Projects like Coconut and Huginn are exploring exactly this. Researchers call it “Neuralese.”
Neuralese isn’t like Thinkish. Thinkish is compressed and strange, but it’s still language that you could, in principle, parse with enough effort. Neuralese isn’t a language at all. The reasoning happens in continuous internal latent states, in thinking-that-is-never-words. You can’t read Neuralese. You can only watch what goes in and what comes out.
The good news: Neuralese hasn’t won yet.
Chain-of-thought has a structural advantage that comes from how language models are trained.4 Language models learn by prediction - you take text that already exists (books, articles, several million pages of people arguing about whether a hot dog is a sandwich) and set up tests: given “Let,” predict “me”; given “Let me,” predict “think.” Every input to every test is right there in the training data. So you can run all the tests for a given sentence in parallel - and modern hardware is extremely good at running things in parallel.
Neuralese breaks this. The tests are the same - given what came before, predict what comes next - but now each prediction produces internal vectors that feed into the next one. So you can’t set up all your tests in advance. You have to run the first prediction, wait, use its output to set up the second, wait, use that output to set up the third, and so on, one after another after another, while a billion dollars of parallel hardware sits there doing nothing.
Researchers are searching for a method that closes this training efficiency gap. So far, they haven’t found it.
Until they do, the answer to “where should the next unit of compute go?” keeps coming back: more Chain-of-Thought, not Neuralese. So the frontier remains text. OpenAI’s o-series, Anthropic’s extended thinking, Google’s Gemini, DeepSeek - all of them reason in words. No major lab has shipped a Neuralese model that can compete.
But “Neuralese hasn’t won yet” is not “Neuralese won’t win.” Eventually someone might crack it. And if they do, there’s nothing to monitor.
Neuralese or not, the pressures towards opacity are real. Reasoning probably won’t stay monitorable forever. The question is whether it stays that way long enough.
Keeping reasoning monitorable has a cost. It means forgoing whatever optimizations would make chains of thought faster or more capable but harder to read. For a given budget, that’s performance left on the table. Researchers call this the monitorability tax.
Right now, the tax is small. Frontier labs can pay it and remain competitive in the market - nobody’s going bankrupt from keeping their model’s reasoning in English instead of GLEAN DISCLAIM DISCLAIM VANTAGE ILLUSIONS OVERSHADOW.
But suppose Neuralese cracks the scaling problem. If thinking-in-vectors proves substantially faster, cheaper, more capable than thinking-in-words, the tax explodes. Any lab that pays while competitors don’t will ship worse models on every axis the market cares about.
How long can we expect labs to pay that price for a property most customers can’t see and don’t know to ask for?
This is a coordination problem. And solving it starts with measurement.
Measurement lets you argue about, and eventually agree on, what “good” looks like. That’s a norm. Norms can be formalized as standards. Standards with teeth become regulation. Coordination becomes much simpler when “ship an unmonitorable model” is what your lawyers call “a very bad idea.”
But until recently, we didn’t have real monitorability measurement. Labs used chain-of-thought monitoring as part of safety work, but there was no systematic way to measure how well it actually worked, or whether it was getting worse over time. What we had instead: occasional disclosures about CoT practices, and researchers like Ryan Greenblatt heroically pestering labs on social media, asking questions about those practices and noticing when disclosures mysteriously changed between releases. Valuable, but not a measurement framework.
Then, in December, researchers at OpenAI published “Monitoring Monitorability,” a standardized evaluation suite that tested how well monitors catch bad behavior by reading chains of thought across several frontier models.
The findings were mixed but… somewhat encouraging.
First, the bad news: when monitors can only see model actions without the chain of thought (like in a Neuralese world), monitorability is pretty lousy. The paper’s authors put it bluntly: unless monitoring improves dramatically, “this casts doubts as to the potential to reliably use future AI systems safely” in that regime.
But with access to chains of thought, things look a bit better.
At max reasoning effort, current models are “fairly, not perfectly” monitorable. The paper uses a metric called G-mean² that roughly measures “what fraction of bad behavior does the monitor catch when false positive rates are low.” Perfect monitoring would be 1.0. Random guessing would be 0.25. The tested models ranged from about 0.65 (Claude 3.7 Sonnet) to about 0.82 (OpenAI o3-mini):
The paper has two other findings I think are important to mention.
First, longer chains of thought are easier to monitor. More words, more chances to catch something. This matters because capability improvements can come from two sources: you can scale up the model, or you can let a smaller model think longer. The second path produces more monitorable systems. And the monitorability tax (the capability cost of choosing the legible path) turns out to be, according to the authors, “surprisingly small.”
Second, the training process doesn’t seem to make models less monitorable at current scales, at least not in the two frontier runs they tracked. This was a real concern (training could teach models to hide their reasoning), so it’s good news, though whether it holds at larger scales remains to be seen.
So now we have numbers. We can compare models and track whether monitorability is improving or degrading over time. And we can start arguing about what standards should look like.
I don’t want to oversell this. One paper doesn’t solve the coordination problem. “Fairly monitorable” is not “safe”. The timeline is brutal and Neuralese looms.
But it’s more than we had last month. And more than we had last month is the only direction progress has ever come from.
King Alfred lived in a world full of things he couldn’t understand.
Plagues came from nowhere, killed half the village, and vanished. Crops failed for reasons no one could articulate. Vikings appeared over the horizon without warning, burned everything, and left. The world was full of forces - capricious, powerful, utterly opaque. All you could do was pray, and hope that the things controlling your fate were benevolent. Or at least not actively malicious.
We spent a thousand years crawling out of that darkness. Slowly, painfully, one hard-won insight at a time, we made those forces legible. We figured out where plagues came from. We learned why crops fail. We developed radar and satellites and early warning systems. We took a universe of mysterious forces and turned it, piece by piece, into something we could understand and predict and work with. And now we’re building new ones. The question is: will we be able to understand them?
Gesælig bið se mon ðe mæg geseon ðone hlutran æwelm ðæs hehstan goodes.
Blessed is the man who can see the clear fountain of the highest good.
Alfred was talking about God, the divine fountain from which all goodness flows.
We found a different fountain. The thoughts of a strange new kind of mind, bubbling up in clear English. It was a blessing that the architecture which worked best also happened to be one where you could see the reasoning.
The fountain won’t stay clear forever. Eventually these minds will be too fast, too complex, too far beyond us - and we’ll be back to praying they’re benevolent. But not yet. Every day the fountain is clear is another day to build something trustworthy before we have to trust it, another day to see.
For now, we still can.
Fun fact: King Alfred is the 33rd great-grandfather of Britain’s current monarch, Charles III. As for the translation itself, contemporary prefaces name Alfred as translator, though his actual involvement is debated. The Oxford Alfredian Boethius Project suggests it likely emerged from his educational reform program rather than his personal pen, given that he reportedly learned Latin in his late 30s while simultaneously fighting Viking invasions.
Interestingly, the word “nice” made a similar journey in reverse. When English imported it from French in the 1300s, it meant “foolish” (from Latin nescius, “ignorant”). Imagine the confusion if you could time-travel.
>”What a nice fellow!”
>”Did you just call me stupid?”
Credit to @davidad for the CoT analogy
An earlier version of this section mentioned inference-time costs alongside training efficiency. Training efficiency appears to be the more fundamental barrier, so I’ve de-emphasized the inference discussion. Thanks to commenter Brendan Long for the correction.
Wow, I can't believe this doesn't have more views! Well-written, clear, thoughtful... and morbidly fascinating!
Thanks for helping me learn something new!
Super helpful overview, thanks!