{"id":837673,"date":"2025-03-30T04:12:35","date_gmt":"2025-03-30T09:12:35","guid":{"rendered":"https:\/\/newsycanuse.com\/index.php\/2025\/03\/30\/anthropic-can-now-track-the-bizarre-inner-workings-of-a-large-language-model\/"},"modified":"2025-03-30T04:12:35","modified_gmt":"2025-03-30T09:12:35","slug":"anthropic-can-now-track-the-bizarre-inner-workings-of-a-large-language-model","status":"publish","type":"post","link":"https:\/\/newsycanuse.com\/index.php\/2025\/03\/30\/anthropic-can-now-track-the-bizarre-inner-workings-of-a-large-language-model\/","title":{"rendered":"Anthropic can now track the bizarre inner workings of a large language model"},"content":{"rendered":"
\n
\n

The AI firm Anthropic has developed a way to peer inside a large language model<\/a> and watch what it does as it comes up with a response, revealing key new insights into how the technology works. The takeaway: LLMs are even stranger than we thought.<\/p>\n

The Anthropic team was surprised by some of the counterintuitive workarounds that large language models appear to use to complete sentences, solve simple math problems, suppress hallucinations, and more, says Joshua Batson, a research scientist at the company.<\/p>\n<\/p><\/div>\n

\n

It\u2019s no secret that large language models work in mysterious ways<\/a>. Few\u2014if any\u2014mass-market technologies have ever been so little understood<\/a>. That makes figuring out what makes them tick one of the biggest open challenges in science.<\/p>\n

But it\u2019s not just about curiosity. Shedding some light on how these models work exposes their weaknesses, revealing why they make stuff up<\/a> and why they can be tricked into going off the rails<\/a>. It helps resolve deep disputes about exactly what these models can and can\u2019t do<\/a>. And it shows how trustworthy (or not) they really are.<\/p>\n

Batson and his colleagues describe their new work in two reports published today. The first presents Anthropic\u2019s use of a technique called circuit tracing, which lets researchers track the decision-making processes inside a large language model step by step. Anthropic used circuit tracing to watch its LLM Claude 3.5 Haiku carry out various tasks. The second (titled \u201cOn the Biology of a Large Language Model\u201d) details what the team discovered when it looked at 10 tasks in particular.<\/p>\n

\u201cI think this is really cool work,\u201d says Jack Merullo, who studies large language models at Brown University in Providence, Rhode Island, and was not involved in the research. \u201cIt\u2019s a really nice step forward in terms of methods.\u201d<\/p>\n

Circuit tracing is not itself new. Last year Merullo and his colleagues analyzed a specific circuit in a version of OpenAI\u2019s GPT-2<\/a>, an older large language model that OpenAI released in 2019. But Anthropic has now analyzed a number of different circuits inside a far larger and far more complex model as it carries out multiple tasks. \u201cAnthropic is very capable at applying scale to a problem,\u201d says Merullo.<\/p>\n

Eden Biran, who studies large language models at Tel Aviv University, agrees. \u201cFinding circuits in a large state-of-the-art model such as Claude is a nontrivial engineering feat,\u201d he says. \u201cAnd it shows that circuits scale up and might be a good way forward for interpreting language models.\u201d<\/p>\n

Circuits chain together different parts\u2014or components\u2014of a model. Last year, Anthropic identified certain components inside Claude<\/a> that correspond to real-world concepts. Some were specific, such as \u201cMichael Jordan\u201d or \u201cgreenness\u201d; others were more vague, such as \u201cconflict between individuals.\u201d One component appeared to represent the Golden Gate Bridge. Anthropic researchers found that if they turned up the dial on this component, Claude could be made to self-identify not as a large language model but as the physical bridge itself.<\/p>\n

The latest work builds on that research and the work of others, including Google DeepMind<\/a>, to reveal some of the connections between individual components. Chains of components are the pathways between the words put into Claude and the words that come out.\u00a0\u00a0<\/p>\n<\/p><\/div>\n

\n

\u201cIt\u2019s tip-of-the-iceberg stuff. Maybe we\u2019re looking at a few percent of what\u2019s going on,\u201d says Batson. \u201cBut that\u2019s already enough to see incredible structure.\u201d<\/p>\n

Growing LLMs<\/strong><\/h3>\n

Researchers at Anthropic and elsewhere are studying large language models as if they were natural phenomena rather than human-built software. That\u2019s because the models are trained, not programmed.<\/p>\n

\u201cThey almost grow organically,\u201d says Batson. \u201cThey start out totally random. Then you train them on all this data and they go from producing gibberish to being able to speak different languages and write software and fold proteins. There are insane things that these models learn to do, but we don\u2019t know how that happened because we didn\u2019t go in there and set the knobs.\u201d<\/p>\n

Sure, it\u2019s all math. But it\u2019s not math that we can follow. \u201cOpen up a large language model and all you will see is billions of numbers\u2014the parameters,\u201d says Batson. \u201cIt\u2019s not illuminating.\u201d<\/p>\n

Anthropic says it was inspired by brain-scan techniques used in neuroscience to build what the firm describes as a kind of microscope that can be pointed at different parts of a model while it runs. The technique highlights components that are active at different times. Researchers can then zoom in on different components and record when they are and are not active.<\/p>\n

Take the component that corresponds to the Golden Gate Bridge. It turns on when Claude is shown text that names or describes the bridge or even text related to the bridge, such as \u201cSan Francisco\u201d or \u201cAlcatraz.\u201d It\u2019s off otherwise.<\/p>\n

Yet another component might correspond to the idea of \u201csmallness\u201d: \u201cWe look through tens of millions of texts and see it\u2019s on for the word \u2018small,\u2019 it\u2019s on for the word \u2018tiny,\u2019 it\u2019s on for the French word \u2018petit,\u2019 it\u2019s on for words related to smallness, things that are itty-bitty, like thimbles\u2014you know, just small stuff,\u201d says Batson.<\/p>\n

Having identified individual components, Anthropic then follows the trail inside the model as different components get chained together. The researchers start at the end, with the component or components that led to the final response Claude gives to a query. Batson and his team then trace that chain backwards.<\/p>\n<\/p><\/div>\n

\n

Odd behavior<\/strong><\/h3>\n

So: What did they find? Anthropic looked at 10 different behaviors in Claude. One involved the use of different languages. Does Claude have a part that speaks French and another part that speaks Chinese, and so on?<\/p>\n

The team found that Claude used components independent of any language to answer a question or solve a problem and then picked a specific language when it replied. Ask it \u201cWhat is the opposite of small?\u201d in English, French, and Chinese and Claude will first use the language-neutral components related to \u201csmallness\u201d and \u201copposites\u201d <\/em>to come up with an answer. Only then will it pick a specific language in which to reply. This suggests that large language models can learn things in one language and apply them in other languages.<\/p>\n

Anthropic also looked at how Claude solved simple math problems. The team found that the model seems to have developed its own internal strategies that are unlike those it will have seen in its training data. Ask Claude to add 36 and 59 and the model will go through a series of odd steps, including first adding a selection of approximate values (add 40ish and 60ish, add 57ish and 36ish). Towards the end of its process, it comes up with the value 92ish. Meanwhile, another sequence of steps focuses on the last digits, 6 and 9, and determines that the answer must end in a 5. Putting that together with 92ish gives the correct answer of 95.<\/p>\n

And yet if you then ask Claude how it worked that out, it will say something like: \u201cI added the ones (6+9=15), carried the 1, then added the 10s (3+5+1=9), resulting in 95.\u201d In other words, it gives you a common approach found everywhere online rather than what it actually did. Yep! LLMs are weird. (And not to be trusted.)<\/p>\n

\n
The steps that Claude 3.5 Haiku used to solve a simple math problem were not what Anthropic expected\u2014and they’re not the steps that Claude claimed it took either.<\/figcaption>

ANTHROPIC<\/p>\n<\/figure><\/div>\n

This is clear evidence that large language models will give reasons for what they do that do not necessarily reflect what they actually did. But this is true for people too, says Batson: \u201cYou ask somebody, \u2018Why did you do that?\u2019 And they\u2019re like, \u2018Um, I guess it\u2019s because I was\u2014 .\u2019 You know, maybe not. Maybe they were just hungry and that\u2019s why they did it.\u201d<\/p>\n

Biran thinks this finding is especially interesting. Many researchers study the behavior of large language models by asking them to explain their actions. But that might be a risky approach, he says: \u201cAs models continue getting stronger, they must be equipped with better guardrails. I believe\u2014and this work also shows\u2014that relying only on model outputs is not enough.\u201d<\/p>\n

A third task that Anthropic studied was writing poems. The researchers wanted to know if the model really did just wing it, predicting one word at a time. Instead they found that Claude somehow looked ahead, picking the word at the end of the next line several words in advance.\u00a0\u00a0<\/p>\n

For example, when Claude was given the prompt \u201cA rhyming couplet: He saw a carrot and had to grab it,\u201d the model responded, \u201cHis hunger was like a starving rabbit.\u201d But using their microscope, they saw that Claude had already hit upon the word \u201crabbit\u201d when it was processing \u201cgrab it.\u201d It then seemed to write the next line with that ending already in place.<\/p>\n<\/p><\/div>\n

\n

This might sound like a tiny detail. But it goes against the common assumption that large language models always work by picking one word at a time in sequence. \u201cThe planning thing in poems blew me away,\u201d says Batson. \u201cInstead of at the very last minute trying to make the rhyme make sense, it knows where it\u2019s going.\u201d<\/p>\n

\u201cI thought that was cool,\u201d says Merullo. \u201cOne of the joys of working in the field is moments like that. There\u2019s been maybe small bits of evidence pointing toward the ability of models to plan ahead, but it\u2019s been a big open question to what extent they do.\u201d<\/p>\n

Anthropic then confirmed its observation by turning off the placeholder component for \u201crabbitness.\u201d Claude responded with \u201cHis hunger was a powerful habit.\u201d And when the team replaced \u201crabbitness\u201d with \u201cgreenness,\u201d <\/em>Claude responded with \u201cfreeing it from the garden\u2019s green.\u201d<\/p>\n

Anthropic also explored why Claude sometimes made stuff up, a phenomenon known as hallucination. \u201cHallucination is the most natural thing in the world for these models, given how they\u2019re just trained to give possible completions,\u201d says Batson. \u201cThe real question is, \u2018How in God\u2019s name could you ever make it not do that?\u2019\u201d<\/p>\n

The latest generation of large language models, like Claude 3.5 and Gemini and GPT-4o, hallucinate far less than previous versions, thanks to extensive post-training (the steps that take an LLM trained on text scraped from most of the internet and turn it into a usable chatbot). But Batson\u2019s team was surprised to find that this post-training seems to have made Claude refuse to speculate as a default behavior. When it did respond with false information, it was because some other component had overridden the \u201cdon\u2019t speculate\u201d component.<\/p>\n

This seemed to happen most often when the speculation involved a celebrity or other well-known entity. It\u2019s as if the amount of information available on a subject pushed the speculation through, despite the default setting. When Anthropic overrode the \u201cdon\u2019t speculate\u201d component to test this, Claude produced lots of false statements about individuals, including claiming that Batson was famous for inventing the Batson principle (he isn\u2019t).<\/p>\n

Still unclear<\/strong><\/h3>\n

Because we know so little about large language models, any new insight is a big step forward. \u201cA deep understanding of how these models work under the hood would allow us to design and train models that are much better and stronger,\u201d says Biran.<\/p>\n

But Batson notes there are still serious limitations. \u201cIt\u2019s a misconception that we\u2019ve found all the components of the model or, like, a God\u2019s-eye view,\u201d he says. \u201cSome things are in focus, but other things are still unclear\u2014a distortion of the microscope.\u201d<\/p>\n<\/div>\n

\n

And it takes several hours for a human researcher to trace the responses to even very short prompts. What\u2019s more, these models can do a remarkable number of different things, and Anthropic has so far looked at only 10 of them.<\/p>\n

Batson also says there are big questions that this approach won\u2019t answer. Circuit tracing can be used to peer at the structures inside a large language model, but it won\u2019t tell you how or why those structures formed during training. \u201cThat\u2019s a profound question that we don\u2019t address at all in this work,\u201d he says.<\/p>\n

But Batson does see this as the start of a new era in which it is possible, at last, to find real evidence for how these models work: \u201cWe don\u2019t have to be, like: \u2018Are they thinking? Are they reasoning? Are they dreaming? Are they memorizing?\u2019 Those are all analogies. But if we can literally see step by step what a model is doing, maybe now we don\u2019t need analogies.\u201d <\/p>\n<\/div>\n<\/div>\n

Read More<\/a>
\n Will Douglas Heaven<\/p>\n","protected":false},"excerpt":{"rendered":"

The AI firm Anthropic has developed a way to peer inside a large language model and watch what it does as it comes up with a response, revealing key new insights into how the technology works. The takeaway: LLMs are even stranger than we thought. The Anthropic team was surprised by some of the counterintuitive<\/p>\n","protected":false},"author":1,"featured_media":837674,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[118839,46,4383],"tags":[],"class_list":{"0":"post-837673","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-anthropic","8":"category-technology","9":"category-track"},"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts\/837673","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/comments?post=837673"}],"version-history":[{"count":0,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/posts\/837673\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/media\/837674"}],"wp:attachment":[{"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/media?parent=837673"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/categories?post=837673"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/newsycanuse.com\/index.php\/wp-json\/wp\/v2\/tags?post=837673"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}