Skip to main content

Emergent parallels between language models and the brain

About

This document presents a Critical Discourse Analysis focused on AI literacy, specifically targeting the role of metaphor and anthropomorphism in shaping public and professional understanding of generative AI. The analysis is guided by a prompt that draws from cognitive linguistics (metaphor structure-mapping), the philosophy of social science (Robert Brown's typology of explanation), and accountability analysis.

All findings and summaries below were generated from detailed system instructions provided to a large language model and should be read critically as interpretive outputs—not guarantees of factual accuracy or authorial intent.

Source Reading​

Genre: Preprint or technical report · Author position: Developer of the system discussed

Argument: The text argues that modern artificial neural networks, specifically large language models, exhibit emergent parallels to human cognition and biological neural activity despite having fundamentally different architectures and learning paradigms. By tracing the history from early connectionism to modern transformer-based systems, the author contends that predictive learning over massive datasets can generate high-level cognitive abilities, such as logical and analogical reasoning. The paper proposes that studying these artificial systems can illuminate general computational principles of natural intelligence, including the power of predictive learning, the constraints imposed by task diversity on internal structure, and the critical importance of scale for generalization.

Responsibility claims: The text does not explicitly formulate ethical responsibility, accountability, or liability claims, as its primary orientation is scientific and comparative. It primarily treats the AI system as an epistemic object and a scientific model for natural intelligence. The text briefly mentions that model developers specify desired behaviors during supervised fine-tuning and reinforcement learning, acknowledging human intervention in shaping the model's outputs. However, the text explicitly frames the system's capabilities as emergent rather than programmed, noting that models develop human-like reasoning capacities even though they are not directly trained to perform them. It expresses uncertainty about whether parallel behaviors between models and brains imply identical underlying computational mechanisms.

Task 1: Metaphor and Anthropomorphism Audit​

About this task

For each of the major metaphorical patterns identified, this audit examines the specific language used, the frame through which the AI is being conceptualized, what human qualities are being projected onto the system, whether the metaphor is explicitly acknowledged or presented as direct description, and—most critically—what implications this framing has for trust, understanding, and policy perception.

1. Model as Deliberative Decision-Maker​

Quote: "In reinforcement learning, the model is given a context (such as a question) and is then allowed to write out its intermediate "reasoning trace" in natural language until it decides it is ready and generates a final answer."

  • Frame: Model as autonomous deliberator
  • Projection: This framing projects human metacognition and deliberate choice onto the autoregressive token generation process. It maps the conscious human experience of "thinking through" a problem and subsequently "deciding" that one's thoughts are complete onto a purely statistical mechanism. By stating the model "decides it is ready," the text elevates the system on the agency gradient to intentional-state attribution, attributing a subjective evaluation of its own computational state without any evidence that the system experiences such internal deliberation. The documented mechanism generic to systems of this type involves the model generating tokens sequentially until it produces a pre-determined stop token, an action driven by probability distributions rather than a conscious realization of readiness. There is no metacognitive threshold being evaluated by an intentional agent, only a mathematical vector resolving into a termination signal.
  • Agency Level: Intentional-state attribution
  • Acknowledgment: Hedged/Qualified (The author places "reasoning trace" in scare quotes, showing explicit awareness that this is a metaphorical mapping. However, the subsequent phrase "until it decides it is ready" is Direct and Unacknowledged, making the overall status a mix, but "reasoning trace" is explicitly qualified.)
  • Implications: Attributing decision-making and metacognition to a language model inflates its perceived sophistication by suggesting the system possesses an internal locus of control and self-awareness. This consciousness projection encourages readers to trust the model's outputs as the product of careful, considered deliberation rather than statistical correlation. If audiences believe the model "decides" when it is finished, they are likely to assume the model has verified the accuracy and completeness of its answer, fostering unwarranted trust and obscuring the fact that the output length is governed by learned statistical patterns. This framing also complicates liability, as a system that "decides" appears more autonomous and thus somewhat responsible for its own errors.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The agentless construction and focus on the model's autonomous "decision" obscure the human developers at Anthropic and other AI firms who designed the reinforcement learning algorithms, defined the reward functions, and implemented the stop-token architecture. The text frames the generation process as a self-directed choice rather than the execution of a developer-defined mathematical objective. By presenting the system as the active agent that "decides," the language removes the engineers who explicitly tuned the model to generate intermediate tokens before the final output. If the system produces harmful or incorrect text, this framing subtly diffuses responsibility away from the developers by presenting the system as an independent actor navigating its own reasoning process. Naming the engineers who designed this generation loop would restore the visibility of human design choices.
Show more...

2. Model as Student Acquiring Knowledge​

Quote: "In the first stage (pretraining), this text consists of a wide range of documents, to teach the model general language processing skills and knowledge."

  • Frame: Model as a pupil
  • Projection: This framing maps the human pedagogical experience of teaching and learning onto the mechanical process of parameter optimization. It projects the conscious acquisition of "knowledge" onto the system, elevating it on the agency gradient to epistemic/conscious state attribution. In human contexts, knowledge involves justified true belief, conscious awareness, and the semantic comprehension of facts. The text supplies no evidence that the model experiences this. Mechanistically, generic to systems of this type, the training process involves a gradient descent algorithm adjusting billions of numerical weights to minimize the cross-entropy loss between predicted and actual tokens across a dataset. The system processes statistical correlations; it does not "know" anything. Attributing "skills and knowledge" to the system conflates the behavioral capability to output accurate text with the conscious possession of epistemic states.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The phrase is presented as literal fact without any hedging, scare quotes, or qualification in the immediate context. I considered Hedged/Qualified because the author is a cognitive scientist who likely views "knowledge" as a functional computational state, but the text itself offers no such qualification here.)
  • Implications: Framing the optimization of a neural network as the teaching of "knowledge" fundamentally distorts the public understanding of AI reliability. When a system is presumed to possess "knowledge," audiences naturally assume it possesses the human capacity to evaluate truth claims, detect contradictions, and understand the real-world referents of the words it generates. This consciousness projection masks the system's reliance on statistical approximations, making phenomena like "hallucinations" seem inexplicable or like deliberate deception rather than the natural consequence of probabilistic text generation. It creates an unwarranted epistemic trust, leading users to rely on the model for factual accuracy in high-stakes domains without verifying the outputs.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The infinitive "to teach" implies a teacher, subtly gesturing toward the human developers who curated the wide range of documents and initiated the pretraining phase. However, the text does not explicitly name these actors (e.g., Anthropic or OpenAI data teams). By leaving the human subject implicit, the text emphasizes the model's acquisition of capability over the highly consequential decisions made by developers regarding which data to include, how to filter it, and what ideological biases might be encoded within it. The wording has the effect of naturalizing the training data as a neutral source of "knowledge" rather than a curated artifact reflecting human choices. Naming the specific corporate teams who built the dataset would clarify that the model's "knowledge" is strictly bounded by human selection.

3. Attention as Conscious Focus​

Quote: "It does so through an operation that is evocatively described as "attention," because it allows the model to query all of its representations from earlier parts of the text for relevant information, and thereby to selectively "attend" to particular parts of the context that are relevant to its current predictions."

  • Frame: Model as conscious observer
  • Projection: This framing projects the human cognitive capacity for selective, deliberate focus onto a mathematical operation. It maps the purposeful, subjective experience of seeking out and concentrating on specific information onto the transformer's self-attention mechanism. The text places this on the agency gradient at goal-directed behavior, attributing a purposeful search for "relevant information." Mechanistically, as documented in the source and generic to transformers, this involves computing dot products between query, key, and value vectors to generate attention weights, which then scale the continuous representations of previous tokens. There is no conscious observer "attending" or subjectively evaluating relevance; the mechanism is a purely deterministic mathematical integration of vector values. The text correctly identifies the mechanism as an operation but still uses language implying an active, selective agent.
  • Agency Level: Goal-directed behavior
  • Acknowledgment: Explicitly Acknowledged (The text explicitly acknowledges the metaphorical nature of this framing by stating it is "evocatively described as 'attention'" and placing "attend" in scare quotes. I considered Direct, but the author's meta-commentary clearly signals awareness of the linguistic mapping.)
  • Implications: Even when acknowledged as a metaphor, the language of "querying" and "selectively attending" invites the assumption that the model possesses a goal-oriented understanding of context. It suggests the model evaluates semantic relevance in a human way, which can lead to overestimations of the system's ability to navigate complex, nuanced, or contradictory prompts. If users believe the system is deliberately selecting the most "relevant" information, they may overlook the fact that the system might weigh a token heavily simply due to its mathematical frequency in the training data rather than its actual logical importance to the prompt, leading to misplaced trust in the model's analytical capabilities.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text positions the model as the active subject that "queries" and "attends." This obscures the human researchers at Google (who invented the Transformer) and developers at companies like Anthropic who architected this specific mathematical function. The self-attention mechanism is a structural design choice made by engineers to solve the vanishing gradient problem in recurrent networks. By framing the operation as something the model "does" to selectively find information, the text masks the rigid, developer-imposed mathematical constraints under which the system operates. The wording serves to animate the artifact, making the engineered mechanism appear as a cognitive capability. Naming the architects of the transformer would remind readers that "attention" is a human-designed mathematical formula, not a machine's cognitive strategy.

4. Multilayer Perceptron as Cognitive Engine​

Quote: "Multi-Layer Perceptron (MLP): Recalling knowledge, reasoning, and local processing"

  • Frame: Model components as cognitive faculties
  • Projection: Found in Figure 1, this text projects the highest levels of human cognition onto the feed-forward network component of a transformer block. It maps human memory retrieval ("recalling") and logical deduction ("reasoning") directly onto matrix multiplications and non-linear activation functions. This places the system at the highest level of the agency gradient: epistemic/conscious state. Mechanistically, generic to systems of this type, the MLP projects the residual stream vector into a higher-dimensional space and applies a non-linearity (like GeLU) before projecting it back down. This mathematically transforms the representations based on weights optimized during training. The system does not "reason" or "recall"; it applies learned continuous functions to vectors. Attributing reasoning to this step entirely blurs the line between mathematical processing and conscious thought.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (This text appears in a diagram label (Figure 1) with no hedging, scare quotes, or qualification. It presents the cognitive functions as the literal description of what the MLP does. I considered Hedged because the surrounding text explains the mathematical vectors, but the label itself is entirely unhedged.)
  • Implications: Using terms like "reasoning" and "recalling knowledge" to label a specific layer of a neural network creates a profound illusion of mind. It suggests to readers that artificial neural networks possess distinct cognitive modules analogous to human brain regions responsible for logic and memory. This inflates perceived sophistication and masks the statistical nature of the system. If users believe a specific sub-component is dedicated to "reasoning," they will assume the model's outputs are the result of logical deduction rather than probabilistic pattern matching. This creates vulnerability to hallucination, as a system that merely processes vectors cannot perform the reality-testing inherent in actual reasoning.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: In this diagrammatic representation, the MLP is presented as an autonomous entity performing cognitive actions. There is no agency displacement in the sense of a specific harmful event, but the framing obscures the human developers who engineered the MLP layers, set the dimensions, chose the activation functions, and optimized the weights. The diagram naturalizes the architecture as a cognitive brain rather than an engineered artifact. By attributing human-like verbs to a mathematical block, the text removes the agency of the AI industry that designs these systems to merely mimic the statistical shape of human reasoning. Acknowledging that developers optimized this network to process vectors would restore the reality of the system as an artifact.

5. Model as Independent Problem Solver​

Quote: "Thus, by using rewards the model can be trained to do things—such as writing complex programs or solving math problems—even if the creators do not know exactly the right way for the model to approach those problems."

  • Frame: Model as an autonomous innovator
  • Projection: This framing projects independent problem-solving capabilities and creative autonomy onto the reinforcement learning process. It maps the human capacity for autonomous discovery onto an optimization algorithm, placing the system at goal-directed behavior on the agency gradient. Mechanistically, as generic to RLHF, human evaluators or reward models provide scalar feedback on model outputs, and an optimization algorithm (like PPO) updates the model's parameters to maximize this expected reward. The model does not invent a new "approach" in a cognitive sense; it traverses a high-dimensional loss landscape to find a parameter configuration that statistically generates high-reward text sequences. The text suggests the model possesses a capacity to innovate beyond its creators' understanding, projecting an intentional search onto mathematical gradient descent.
  • Agency Level: Goal-directed behavior
  • Acknowledgment: Direct (Unacknowledged) (The claim is presented straightforwardly as a description of model capabilities. There are no hedges or qualifiers indicating that "approach those problems" is a metaphor for parameter optimization. I considered Ambiguous, but the contrast with the creators explicitly positions the model as an independent problem solver.)
  • Implications: This framing significantly inflates the perceived autonomy and intelligence of AI systems, feeding into narratives of "superintelligence" and uncontrollable AI. By suggesting the model independently discovers "approaches" that humans do not understand, it mystifies the technology and encourages a view of AI as an uncontrollable, quasi-magical entity. This can shift policy discussions away from practical, immediate harms (like data bias or labor exploitation) toward speculative, existential risks. Furthermore, if a system makes a catastrophic error, this framing provides a ready-made excuse for developers: the model was autonomously approaching the problem in a way they couldn't foresee, thereby diffusing their liability for deploying an unsafe system.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The text explicitly mentions "the creators," acknowledging the human role in training the model. However, it does so specifically to contrast human ignorance with the model's capability, effectively displacing agency for the actual "approach" onto the AI. The text obscures the immense amount of human labor involved in reinforcement learning: the thousands of data annotators who provided the initial rankings, the engineers at firms like Anthropic who designed the reward function, and the executives who decided to deploy the system despite its uninterpretable mechanics. By claiming the creators do not know the right way to approach the problem, the text subtly absolves them of responsibility for how the model ultimately behaves, leveraging the opacity of the neural network as an accountability sink.

6. Model as Fallible Thinker​

Quote: "Human reasoning on these problems does not perfectly follow the rules of formal logic, but instead depends on the content that we are reasoning about in sometimes illogical ways... and language models mix logic and content in similar ways."

  • Frame: Model as a cognitively biased thinker
  • Projection: This framing projects human cognitive biases and logical fallacies onto the model's statistical outputs. It maps the flawed, content-dependent nature of human thought onto the system, elevating it to epistemic/conscious state on the agency gradient by suggesting the model is actually "reasoning." Mechanistically, generic to large language models, the system replicates the statistical distributions of the text it was trained on. Because human training data contains illogical arguments influenced by content, the model predicts tokens that reflect those same illogical patterns. The model does not "mix logic and content" through a flawed cognitive process; it accurately reproduces the statistical shape of flawed human text. Attributing illogical reasoning to the model assumes it possesses a reasoning faculty that can be corrupted, rather than recognizing it as a mirror of its dataset.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The text directly states that language models "mix logic and content in similar ways" without qualification. While the broader paper discusses mechanisms, this specific comparison asserts a direct behavioral equivalence. I considered Hedged because the paper overall is about "parallels," but this sentence makes a definitive statement.)
  • Implications: By attributing human-like cognitive biases to the model, the text deepens the illusion of mind, making the AI appear more human and relatable. Paradoxically, highlighting the model's flaws in this way increases relation-based trust; it suggests the system's errors are the result of relatable human-like reasoning rather than brittle statistical failure modes. This obscures the fundamental difference between a human making a logical error and an algorithm blindly predicting the next token. It masks the reality that the model is entirely dependent on its training data and has no underlying causal model of the world, leading users to overestimate its reliability in domains outside its training distribution.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text attributes the mixing of logic and content entirely to the "language models." This agentless construction completely obscures the human origin of the illogical content. The model does not mix logic and content; it replicates the flawed text generated by millions of human internet users and subsequently scraped, compiled, and fed into the model by data engineers at corporations. By framing the flawed output as an intrinsic cognitive quirk of the model, the text erases the developers who chose to train the system on unverified internet data and the original humans who produced that data. Naming the dataset curators would reveal that the model's "illogical" behavior is a direct product of human decisions regarding data acquisition.

7. Optimization as Emergent Capability​

Quote: "These human-like capabilities of models are often called emergent, because they emerge after training even though they are not the language-prediction task that models are directly trained to perform."

  • Frame: Model as evolving organism
  • Projection: This framing projects biological or complex systems emergence onto the mathematical process of parameter optimization. It maps the spontaneous development of new traits onto the model, placing it at computational activity but heavily implying a higher level of autonomous development. Mechanistically, generic to systems of this type, when a network is trained to predict tokens across a massive, diverse dataset, minimizing the loss requires the parameters to encode structural heuristics, correlations, and representations that allow it to perform zero-shot tasks (like analogy). These capabilities do not "emerge" like consciousness in a biological brain; they are mathematical consequences of the loss function forcing the network to compress a vast amount of structured human data. The text projects organic spontaneity onto deterministic optimization.
  • Agency Level: Computational activity
  • Acknowledgment: Hedged/Qualified (The text uses "are often called emergent," acknowledging that this is a terminology used in the field rather than stating it as an absolute truth. I considered Direct, but the phrasing indicates a slight distance from the term, introducing it as a common label.)
  • Implications: The language of "emergence" is highly effective at mystifying AI systems. It suggests that the models possess an internal dynamism and are capable of developing beyond their original programming in a biological sense. This framing severely complicates accountability and regulation. If capabilities simply "emerge" spontaneously, it suggests they cannot be predicted, measured, or controlled by human developers. This inflates the perceived risk of autonomous AI while simultaneously providing developers with a structural defense against liability: they cannot be held responsible for harms caused by capabilities that "emerged" on their own rather than being engineered.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The concept of "emergence" serves as a profound accountability sink. The text states these capabilities "emerge after training" without naming who executed the training, who selected the massive datasets that made such optimization possible, or who scaled the compute power to reach these statistical thresholds. The corporations building these systems (like Anthropic or OpenAI) deliberately scale parameters and data specifically to achieve these generalized capabilities. Framing the results as "emergent" obscures the intentional corporate strategy to build highly capable, generalized systems. By presenting the capabilities as spontaneous phenomena, the language erases the deliberate financial and engineering decisions that made them inevitable.

8. Model as Intentional Actor Being Rewarded​

Quote: "If the model gets the answer wrong and is penalized, it will be updated to produce less similar thinking tokens and answers in the future."

  • Frame: Model as a trained animal/agent
  • Projection: This framing maps behavioral psychology—specifically the operant conditioning of animals or humans—onto the mathematical process of reinforcement learning. It projects intentionality and a desire to seek reward onto the system, locating it at goal-directed behavior. Mechanistically, generic to RLHF, a "penalty" is simply a low scalar value from a reward model, which the optimization algorithm (e.g., PPO) uses to calculate gradients that shift the model's probability distribution away from the parameters that generated the penalized text. The model does not feel a penalty, nor does it consciously adjust its future behavior to avoid it. The text projects the psychological experience of learning from mistakes onto the mechanical adjustment of numerical weights.
  • Agency Level: Goal-directed behavior
  • Acknowledgment: Direct (Unacknowledged) (The language of getting answers wrong, being penalized, and updating behavior is presented straightforwardly. There is no hedging to indicate that "penalized" is a metaphor for a mathematical gradient update. I considered Hedged because "updated" is somewhat mechanistic, but "penalized" carries strong agential weight.)
  • Implications: Using the language of operant conditioning (reward and penalty) fundamentally anthropomorphizes the system's learning process. It leads readers to imagine the AI as an agent with desires (to get the reward) and aversions (to avoid the penalty), fostering an illusion of an internal motivational state. This obscures the purely mathematical nature of the optimization landscape. It also implies that the model's outputs are the result of its "desire" to be helpful, rather than the deterministic result of parameter adjustments. This can lead users to misinterpret the model's highly polished, compliant tone as genuine sincerity or friendliness, masking the rigid engineering that forces that tone.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The passive voice ("is penalized," "will be updated") completely obscures the human actors who define the penalties and execute the updates. The text hides the invisible labor of human annotators who rate the model's outputs to train the reward model, as well as the engineers who design the reward function and optimization algorithms. By framing the process as something that happens to an autonomous learning agent, the text removes the agency of the AI corporation that dictates what constitutes a "wrong" answer and what behaviors are mandated. Naming the developers and annotators would reveal that the model's "learning" is entirely controlled by human preferences and corporate directives.

Task 2: Source-Target Mapping​

About this task

For each key metaphor identified in Task 1, this section provides a detailed structure-mapping analysis. The goal is to examine how the relational structure of a familiar "source domain" (the concrete concept we understand) is projected onto a less familiar "target domain" (the AI system). By restating each quote and analyzing the mapping carefully, we can see precisely what assumptions the metaphor invites and what it conceals.

Mapping 1: conscious mind/deliberate actor → model​

Quote: "until it decides it is ready and generates a final answer."

  • Source Domain: conscious mind/deliberate actor
  • Target Domain: model
  • Mapping: This mapping takes the subjective, conscious experience of human deliberation—evaluating one's own thoughts, recognizing when a conclusion has been reached, and making a deliberate choice to speak—and projects it onto the autoregressive token generation of a neural network. It invites the assumption that the model possesses an internal metacognitive state, an awareness of its own reasoning process, and the agency to initiate an action based on that awareness.
  • What Is Concealed: This mapping conceals the purely statistical and mechanistic nature of token generation. It obscures the fact that the model simply calculates probability distributions for the next token based on its context window and stops when the highest probability token is a pre-defined stop token. It hides the lack of any internal locus of conscious evaluation, masking the reality that the "decision" is a mathematically predetermined vector resolution dictated by the model's proprietary parameters.
Show more...

Mapping 2: teacher and student → model​

Quote: "to teach the model general language processing skills and knowledge."

  • Source Domain: teacher and student
  • Target Domain: model
  • Mapping: This mapping projects the dynamic of human education onto algorithmic optimization. It implies a transfer of semantic understanding from the training data into the "mind" of the model. By using the word "knowledge," the mapping invites the assumption that the model acquires justified true beliefs and a contextual understanding of the world, much like a human student who learns not just to parrot words but to comprehend their real-world referents.
  • What Is Concealed: This mapping conceals the mathematical reality of gradient descent and cross-entropy loss. It hides the fact that the system does not possess knowledge but rather parameter weights that encode complex statistical correlations between tokens. It obscures the absence of any grounding or causal model of reality within the system. Furthermore, it masks the immense scale of uncurated internet data used in pretraining by flattening it into the benign concept of "teaching."

Mapping 3: conscious observer → model​

Quote: "query all of its representations... and thereby to selectively "attend" to particular parts"

  • Source Domain: conscious observer
  • Target Domain: model
  • Mapping: This maps the human psychological mechanism of selective attention—the purposeful, conscious focusing of cognitive resources on specific stimuli—onto the transformer architecture. It invites the reader to imagine the model as an active agent actively searching through a text, evaluating different words for relevance, and choosing which ones to prioritize to solve a problem.
  • What Is Concealed: This mapping conceals the deterministic, mathematical operation of the self-attention mechanism. It hides the fact that "queries," "keys," and "values" are simply matrices multiplied together to produce attention weights. The text obscures that relevance is determined purely by geometric proximity in high-dimensional space as shaped by training, not by any semantic comprehension or conscious selection. It masks the rigidity of the mathematical formula.

Mapping 4: cognitive brain/knower → model​

Quote: "Multi-Layer Perceptron (MLP): Recalling knowledge, reasoning, and local processing"

  • Source Domain: cognitive brain/knower
  • Target Domain: model
  • Mapping: This maps specific, high-level human cognitive faculties directly onto a distinct component of the neural network architecture. It invites the assumption that the artificial network is fundamentally homologous to the human brain, possessing specialized modules for memory storage ("recalling") and logical deduction ("reasoning"). It projects epistemic awareness onto mathematical operations.
  • What Is Concealed: This mapping conceals the functional reality of a feed-forward layer, which merely applies non-linear transformations to vectors to project them into different representational spaces. It hides the complete absence of actual symbolic logic, working memory, or semantic recall in the model. By labeling a matrix operation as "reasoning," the text completely obscures the probabilistic nature of the system's outputs, making a brittle statistical engine look like a robust cognitive reasoner.

Mapping 5: autonomous problem solver → model​

Quote: "the model can be trained to do things... even if the creators do not know exactly the right way for the model to approach those problems."

  • Source Domain: autonomous problem solver
  • Target Domain: model
  • Mapping: This maps the human capacity for independent innovation and creative problem-solving onto the reinforcement learning process. It invites the assumption that the model possesses a degree of agency and cognitive flexibility that surpasses its programming, allowing it to devise novel "approaches" and strategies entirely on its own.
  • What Is Concealed: This mapping conceals the mechanics of traversing a loss landscape. It hides the fact that the model is simply discovering parameter configurations that maximize the human-defined reward function through trial and error (or gradient updates). It obscures the ultimate control of the human creators who define the optimization space, the action space, and the reward signals, masking the reality that the "approach" is just the mathematical path of least resistance to maximize reward.

Mapping 6: cognitively biased thinker → model​

Quote: "language models mix logic and content in similar ways."

  • Source Domain: cognitively biased thinker
  • Target Domain: model
  • Mapping: This maps the human psychological tendency toward flawed, content-biased logical reasoning onto the model's text generation. It invites the assumption that the model attempts to reason logically but falls prey to human-like cognitive biases, projecting an underlying (if flawed) reasoning faculty onto the system.
  • What Is Concealed: This mapping conceals the fact that the model has no reasoning faculty to be biased. It hides the reality that the model is entirely a reflection of its training data; it outputs illogical text because it has statistically mapped the logical fallacies present in the human-generated text it consumed. It obscures the system's nature as a statistical mirror, making it appear instead as an independent, flawed thinker.

Mapping 7: evolving organism → model​

Quote: "These human-like capabilities of models are often called emergent, because they emerge after training"

  • Source Domain: evolving organism
  • Target Domain: model
  • Mapping: This maps the biological and complex-systems concept of emergence—where novel properties spontaneously arise from complex interactions—onto the results of machine learning at scale. It invites the reader to view the model's capabilities as organic, unpredictable, and disconnected from direct human engineering.
  • What Is Concealed: This mapping conceals the highly intentional, engineered scaling laws that drive these capabilities. It hides the massive corporate investments in compute power, data collection, and architectural optimization designed specifically to achieve generalized performance. It obscures the mathematical reality that zero-shot capabilities are latent in the geometry of the representations learned through massive next-token prediction, framing them instead as magical or spontaneous developments.

Mapping 8: trained animal/agent → model​

Quote: "If the model gets the answer wrong and is penalized, it will be updated to produce less similar thinking tokens"

  • Source Domain: trained animal/agent
  • Target Domain: model
  • Mapping: This maps the psychological framework of operant conditioning onto algorithmic optimization. It invites the assumption that the model experiences a "penalty" as a negative stimulus and possesses the intentionality to alter its future behavior to avoid that penalty and seek a reward.
  • What Is Concealed: This mapping conceals the mechanical calculation of gradients. It hides the fact that a penalty is simply a scalar mathematical value used to compute weight updates via backpropagation. It obscures the absence of any subjective experience, motivation, or memory of the penalty in the system. Furthermore, it conceals the labor of the human annotators who define what constitutes a "wrong" answer through their feedback.

Task 3: Explanation Audit (The Rhetorical Framing of "Why" vs. "How")​

About this task

This section audits the text's explanatory strategy, focusing on a critical distinction: the slippage between "how" and "why." Based on Robert Brown's typology of explanation, this analysis identifies whether the text explains AI mechanistically (a functional "how it works") or agentially (an intentional "why it wants something"). The core of this task is to expose how this "illusion of mind" is constructed by the rhetorical framing of the explanation itself, and what impact this has on the audience's perception of AI agency.

Explanation 1​

Quote: "The attention operation integrates information from earlier words by allowing the model to produce a query which is compared to representations at early positions."

  • Explanation Types:

    • Functional (Primary): Explains behavior by role in self-regulating system with feedback
    • Theoretical (Secondary): Embeds in deductive framework, may invoke unobservable mechanisms

  • Analysis (Why vs. How Slippage): This explanation frames the AI mechanistically (how), detailing the specific sub-components of the transformer architecture. It uses the language of operations, integration, and comparisons to describe the internal workings of the system. This functional framing emphasizes the structured, deterministic nature of the attention mechanism, breaking down a complex capability into discrete, interacting parts (queries and representations). By focusing on the "how," this explanation avoids some of the more overt anthropomorphism seen elsewhere in the text, highlighting the model as a constructed system of interacting components. However, it still relies on subtly agential verbs ("allowing the model to produce a query"), which slightly obscures the purely mathematical nature of the matrices involved, though it remains primarily rooted in technical description.

  • Consciousness Claims Analysis: This passage avoids direct consciousness claims. The verbs present ("integrates," "allowing," "produce," "compared") are mechanistic rather than epistemic. The system is located at computational activity on the agency gradient; it processes representations rather than knowing or understanding them. There is a slight curse of knowledge dynamic in the use of the word "query," which implies a conscious request for information to a human reader, though in computer science it is a standard term for a database operation or vector matrix. Mechanistically, generic to transformer architectures, the model computes query, key, and value matrices for each token. The query vector of the current token is multiplied by the key vectors of preceding tokens to calculate attention scores, which dictate how much of the value vectors from those preceding tokens are integrated into the current token's representation. The text describes this generation-time behavior accurately, though the terminology ("query") leans slightly on cognitive metaphors inherited by the field.

  • Rhetorical Impact: This mechanistic framing shapes the audience's perception by grounding the model's capabilities in concrete architecture rather than mysterious intelligence. It constructs performance-based trust, indicating that the model's ability to handle context is based on specific, engineered operations rather than an innate understanding. By explaining the "how" of the attention mechanism, it reduces the perceived autonomy of the system, making it clear that the AI's behavior is dictated by structural rules. If audiences understand the system as performing vector comparisons rather than "reading," they are less likely to assume the model comprehends the semantic meaning of the text, leading to more cautious and calibrated reliance on its outputs.

Show more...

Explanation 2​

Quote: "In the first stage (pretraining), this text consists of a wide range of documents, to teach the model general language processing skills and knowledge."

  • Explanation Types:

    • Intentional (Primary): Refers to goals/purposes, presupposes deliberate design
    • Genetic (Secondary): Traces origin through dated sequence of events or stages

  • Analysis (Why vs. How Slippage): This explanation operates through a genetic frame by placing pretraining as the "first stage" in a chronological sequence, but its primary force is intentional, explaining the purpose of the stage ("to teach the model"). This hybrid frames the AI largely agentially (why it is trained) while explaining its developmental origin. The choice to frame optimization as "teaching" emphasizes the acquisition of capability and normalizes the training process by likening it to human education. It obscures the mechanistic reality of parameter updates and the massive, often indiscriminate scraping of internet data required for this stage. The explanation focuses on the desired outcome (skills and knowledge) rather than the actual process (statistical correlation).

  • Consciousness Claims Analysis: This passage makes a stark consciousness claim by attributing "knowledge" to the model. The verb "teach" and the noun "knowledge" are strongly epistemic, contrasting sharply with mechanistic verbs like "optimize" or "process." The text places the system at epistemic/conscious state on the agency gradient. The author, likely projecting the end result of coherent text generation backward onto the training process, uses the curse of knowledge to assume the system learns just as the author does. Mechanistically, generic to systems of this type, pretraining involves feeding sequences of tokens through the model and using an optimization algorithm (like Adam) to adjust billions of parameters via backpropagation to minimize the error in predicting the next token. The model shapes its parameters based on statistical frequencies in the training data; it does not acquire justified true beliefs. The text completely substitutes an epistemic state for a computational process.

  • Rhetorical Impact: Framing pretraining as teaching knowledge fundamentally shifts the audience's perception from mechanism to mind. It encourages relation-based trust; audiences trust systems with "knowledge" differently than systems with "parameters." If policymakers and users believe the AI possesses knowledge, they may treat it as a reliable epistemic authority, failing to recognize that it can generate highly probable but entirely false statements. This framing masks the system's profound dependency on the biases and errors in its training data. By viewing the model as a knowledgeable entity rather than a statistical pattern matcher, audiences are rendered vulnerable to overestimating its capabilities and misunderstanding its failures.

Explanation 3​

Quote: "If the model gets the answer wrong and is penalized, it will be updated to produce less similar thinking tokens and answers in the future."

  • Explanation Types:

    • Functional (Primary): Explains behavior by role in self-regulating system with feedback
    • Dispositional (Secondary): Attributes tendencies or habits

  • Analysis (Why vs. How Slippage): This explanation is primarily functional, describing the feedback loop of reinforcement learning where an outcome (getting it wrong) leads to a system adjustment (updating). However, it relies heavily on dispositional and behavioral framing by using terms like "penalized" and "thinking tokens." It frames the AI mechanically in the outcome ("will be updated") but agentially in the premise ("gets the answer wrong," "is penalized"). This choice emphasizes the model's behavioral adjustment over time, making it appear adaptable and responsive to feedback. However, it obscures the actual mechanism of the update, masking the mathematical gradient descent behind the psychological language of punishment and behavioral correction.

  • Consciousness Claims Analysis: The passage contains a mix of mechanistic ("updated," "produce") and epistemic/consciousness verbs ("thinking"). While "updated" acknowledges the system as an artifact, "thinking tokens" and "penalized" project subjective experience and cognitive processes onto it. The text elevates the system to goal-directed behavior on the agency gradient. The author employs a common industry shorthand ("thinking tokens" for intermediate computational steps), but this shorthand projects mind onto mechanism. Mechanistically, generic to reinforcement learning, a reward model calculates a low scalar value for an output deemed incorrect by human annotators. This scalar is used to compute a loss, and the model's parameters are updated to decrease the probability of generating that specific sequence of tokens in the future. The model generates text based on shifted parameters; it does not "think" or experience a "penalty."

  • Rhetorical Impact: This framing shapes the audience's perception by making the AI appear as a responsive, learning agent capable of recognizing its mistakes. It constructs a false sense of security, implying that the system can be easily corrected through simple punishment, much like training a dog. This increases trust in the model's reliability over time, as users assume it "learns its lesson." If audiences believe the AI generates "thinking tokens," they will assume its outputs are the result of logical deliberation rather than statistical sampling. This obscures the difficulty of truly aligning large language models and hides the fact that a "penalized" model might simply learn to generate text that superficially pleases the reward model without actually improving factual accuracy.

Explanation 4​

Quote: "These human-like capabilities of models are often called emergent, because they emerge after training even though they are not the language-prediction task that models are directly trained to perform."

  • Explanation Types:

    • Genetic (Primary): Traces origin through dated sequence of events or stages
    • Empirical Generalization (Secondary): Subsumes events under timeless statistical regularities

  • Analysis (Why vs. How Slippage): This explanation is primarily genetic, explaining the existence of certain capabilities by tracing their origin to the period "after training." It frames the AI's development as an organic, chronological unfolding. Secondarily, it relies on empirical generalization, noting that models typically develop these traits even though they are not explicitly trained for them. This framing emphasizes the mystery and power of the learning process, highlighting the disconnect between the simple objective function and the complex resulting behaviors. By focusing on emergence, the explanation obscures the intentional scaling of architecture and data by developers that predictably forces these capabilities to manifest as mathematical necessities of loss minimization.

  • Consciousness Claims Analysis: The passage avoids direct consciousness verbs, using "emerge" and "perform," but it makes a broad structural claim about "human-like capabilities." It locates the system at computational activity (performing tasks), but the word "emergent" carries heavy biological and cognitive connotations. There is no direct projection of knowing onto processing here, but rather a projection of organic evolution onto mechanical scaling. Mechanistically, generic to systems of this type, when a model with massive parameter counts is trained to predict tokens across diverse data, the most mathematically efficient way to minimize loss is to encode generalized representations and heuristics (like tracking syntax or analogical structures). At generation time, prompting the model activates these latent structures. The capabilities do not organically emerge; they are statistically forced by the scale of the optimization.

  • Rhetorical Impact: The "emergence" framing profoundly impacts risk perception. It portrays AI development as an uncontrollable, quasi-natural force rather than an engineered product. This can lead audiences to feel awe or fear, assuming the system possesses a life of its own. It shifts the regulatory focus away from concrete corporate decisions (how much data to scrape, how large a model to train) and toward managing the unpredictable "emergent" behaviors of the technology. By viewing these capabilities as spontaneous rather than engineered, audiences may inadvertently absolve developers of responsibility for the harms these capabilities might cause, accepting them as natural phenomena rather than product defects.

Explanation 5​

Quote: "Human reasoning on these problems does not perfectly follow the rules of formal logic... and language models mix logic and content in similar ways"

  • Explanation Types:

    • Empirical Generalization (Primary): Subsumes events under timeless statistical regularities
    • Dispositional (Secondary): Attributes tendencies or habits

  • Analysis (Why vs. How Slippage): This explanation relies on empirical generalization, establishing a behavioral equivalence between humans and models based on observed outputs (both deviate from formal logic). It secondarily uses a dispositional frame, attributing a tendency to "mix logic and content" to the models. The framing is heavily agential (why it behaves this way), explaining the model's output by equating its internal process with human cognitive flaws. This emphasizes the psychological parallel between human and machine, which is the core thesis of the paper. However, it entirely obscures the mechanism; the model outputs illogical text not because it is reasoning poorly, but because it is successfully predicting the poorly reasoned text prevalent in its training data.

  • Consciousness Claims Analysis: This passage relies heavily on consciousness claims. By directly comparing the model's behavior to "human reasoning," the text implies the model possesses a reasoning faculty. It locates the system at epistemic/conscious state on the agency gradient. The author, focused on the macro-level parallels, falls into the curse of knowledge by assuming that because the outputs look like human reasoning, they are generated by a process akin to reasoning. Mechanistically, generic to systems of this type, a language model generates text by predicting the statistically most likely sequence of tokens based on its context window and the distributions learned during training. Because humans frequently mix logic and content online, those patterns are encoded in the model's parameters. At generation time, the model simply replicates these statistical shapes. It does not "reason."

  • Rhetorical Impact: This framing shapes audience perception by deeply humanizing the AI system. It encourages audiences to view the model not as a calculator, but as a relatable, flawed thinker. This dramatically increases relation-based trust; users are more likely to forgive errors if they perceive them as human-like cognitive biases rather than structural mechanical failures. This obscures the fundamental brittleness of the system. If audiences believe the AI "reasons" like a human, they will assume it possesses human-like common sense and reality-testing abilities, which it entirely lacks. This can lead to catastrophic failures in deployment when the system fails to correct obvious factual or logical errors outside its training distribution.

Task 4: AI Literacy in Practice - Reframing Anthropomorphic Language​

About this task

This section proposes alternative language for key anthropomorphic phrases, offering more mechanistic and precise framings that better reflect the actual computational processes involved. Each reframing attempts to strip away the projections of intention, consciousness, or agency that are embedded in the original language.

Original Anthropomorphic FrameMechanistic ReframingTechnical Reality CheckHuman Agency Restoration
until it decides it is ready and generates a final answer.The model generates intermediate tokens sequentially until the predicted probability of the designated stop token exceeds the threshold, at which point the generation loop terminates and the final text string is produced.The system does not experience a metacognitive state of readiness or make a decision. Autoregressive language models generate text by sampling from a probability distribution over the vocabulary for each subsequent token. The process concludes automatically when the model generates a specific end-of-sequence token.

Basis: Generic to system type		—
to teach the model general language processing skills and knowledge.During pretraining, developers optimize the model's parameters across a vast dataset of documents, shaping the neural network to encode the statistical correlations and structural patterns present in human language.The model does not acquire semantic comprehension, skills, or justified true beliefs. The pretraining process uses backpropagation to adjust mathematical weights, enabling the system to predict missing tokens based purely on statistical frequency and structural proximity within the training data.

Basis: Generic to system type		During pretraining, engineers at AI firms optimize the model's parameters across a vast dataset of documents they selected, shaping the neural network to encode the statistical correlations present in that data.
query all of its representations... and thereby to selectively "attend" to particular partsThe self-attention mechanism computes mathematical weights by multiplying query, key, and value matrices, which scales the continuous representations of preceding tokens based on their geometric relationships.The system does not possess a conscious focus or an ability to selectively evaluate relevance. The attention mechanism is a deterministic mathematical formula that calculates dot products between vector representations, scaling the influence of past tokens on the current prediction strictly according to learned parameters.

Basis: Documented in source		—
Recalling knowledge, reasoning, and local processingApplying non-linear transformations by projecting representations into a higher-dimensional space.The multi-layer perceptron does not recall facts or execute logical reasoning. It applies learned, continuous mathematical functions to the vectors within the residual stream, transforming the numerical representations based on the weights established during the model's training phase.

Basis: Documented in source		—
the model can be trained to do things... even if the creators do not know exactly the right way for the model to approach those problems.The optimization algorithm can adjust the model's parameters to maximize the reward function on complex tasks, discovering high-reward parameter configurations even if developers cannot interpret the resulting mathematical representations.The model does not possess independent problem-solving capabilities or the ability to intentionally approach a problem. It traverses a high-dimensional loss landscape to find parameter weights that statistically output text sequences that human evaluators or reward models score highly.

Basis: Generic to system type		Developers apply optimization algorithms that adjust the model's parameters to maximize the reward function on complex tasks, discovering high-reward parameter configurations even if those developers cannot interpret the resulting mathematical representations.
language models mix logic and content in similar ways.The text generated by language models replicates the heuristic shortcuts and logical fallacies that are statistically prevalent in the human-generated training data.Language models do not possess a reasoning faculty that can be influenced by cognitive biases. They generate text by predicting the most statistically likely subsequent tokens; when prompted with logical puzzles, they reproduce the flawed reasoning patterns encoded in their training corpora.

Basis: Reasonable inference		—
These human-like capabilities of models are often called emergent, because they emerge after trainingThese generalized capabilities are mathematical consequences of the loss function; as developers scale the parameter count and training data, the network is forced to encode structural representations that enable zero-shot task completion.Capabilities do not spontaneously emerge in a biological sense. They are the predictable result of optimizing massive neural networks on vast datasets, where encoding generalized heuristics becomes the most mathematically efficient way to minimize the cross-entropy loss during next-token prediction.

Basis: Reasonable inference		These generalized capabilities are mathematical consequences of the loss function; as developers scale the parameter count and training data, they force the network to encode structural representations that enable zero-shot task completion.
If the model gets the answer wrong and is penalized, it will be updated to produce less similar thinking tokensIf human annotators or a reward model assign a low scalar value to an output, the optimization algorithm computes gradients that shift the model's parameters, decreasing the mathematical probability of generating similar text tokens.The model does not experience a penalty, understand that it was wrong, or adjust its "thinking." Reinforcement learning involves an algorithm that uses low reward scores to update parameter weights, altering the probability distributions for future text generation.

Basis: Generic to system type		If human annotators assign a low score to an output, the developers' optimization algorithm computes gradients that shift the model's parameters, decreasing the mathematical probability of generating similar text tokens.

Task 5: Critical Observations - Structural Patterns​

Agency Slippage​

The text exhibits systematic agency slippage, oscillating continually between the mechanical reality of neural network architecture and the agential framing of cognitive science. This slippage primarily moves in an upward direction on the agency gradient: computational activity is consistently elevated to epistemic and conscious states. A dramatic moment of slippage occurs early in the text when describing the transformer architecture (Figure 1). The author begins mechanically, noting the model embeds input into a "high-dimensional space" to form a "residual stream." However, within the same diagram, the Multi-Layer Perceptron (MLP) layer is abruptly labeled with the agential capacities of "recalling knowledge, reasoning, and local processing." This leap bypasses intermediate agency levels entirely, translating vector mathematics directly into conscious human cognition.

This slippage is deeply rooted in the curse of knowledge. As a cognitive scientist comparing models to brains, the author's deep understanding of human reasoning leads him to project that understanding onto the system whose outputs resemble reasoning. When the text notes that models can be trained to solve problems "even if the creators do not know exactly the right way for the model to approach those problems," agency is explicitly transferred from the human developers to the machine. The model is framed as an autonomous innovator finding an "approach." This wording can have the effect of obscuring the fact that the "approach" is simply the mathematical path of least resistance through a loss landscape designed by humans.

Similarly, when describing reinforcement learning, the text states the model "writes out its intermediate 'reasoning trace'... until it decides it is ready." Here, delegated action (the generation of intermediate tokens) slips into intentional-state attribution (deciding it is ready). This pattern reveals that the text establishes the AI as a "knower" early on through broad metaphors about "learning" and "knowledge," which then provides the rhetorical foundation for later attributing intent and decision-making. Agentless constructions further enable this slippage; by writing "the model is trained" or "is penalized," the text removes the human engineers, leaving the AI as the sole active subject in the narrative. By doing so, the vocabulary makes the mathematical mechanics unsayable in the same breath as the capabilities, forcing the reader to accept the model as a cognitive agent rather than an engineered artifact.

Metaphor-Driven Trust Inflation​

The metaphorical framing in the text systematically builds relation-based trust by projecting human-like cognitive states onto statistical systems. By using verbs such as "reasoning," "decides," and "recalling knowledge," the text signals to the reader that the model possesses an internal epistemic architecture akin to human thought. This establishes a profound but unwarranted sense of authority. When a system is described as possessing "knowledge" rather than merely "parameters," it implicitly claims the ability to evaluate truth, recognize context, and understand reality. This wording can have the effect of encouraging audiences to extend the kind of trust reserved for human experts (sincerity, comprehension) to a system that is only capable of performance-based reliability (statistical correlation).

The text's framing of system limitations further cements this relation-based trust. When the author states that "language models mix logic and content in similar ways" to humans, the metaphor of the cognitively biased thinker humanizes the AI. It suggests that when the model fails, it fails for relatable, human reasons—falling prey to illogical heuristics. This framing encourages users to forgive the model's errors as they would a human's, masking the fact that the model's failure is not a cognitive lapse but a rigid, mechanical reproduction of flawed training data. By equating human cognitive bias with algorithmic token prediction, the text obscures the fundamental brittleness of statistical generation.

Furthermore, intentional and reason-based explanations construct the illusion that AI decisions are justified. When the text claims a model "decides it is ready" to generate an answer, it implies an internal process of verification and metacognition. It signals to the reader that the output is not just probable, but considered. The risks of this framing are significant. If audiences extend relation-based trust to a system incapable of reciprocating, they become vulnerable to hallucination and logical failure in high-stakes domains. They will expect the system to possess common sense, to recognize when it does not know an answer, and to prioritize factual accuracy—capabilities that a purely predictive artifact does not possess. The metaphors of cognition thus construct an architecture of authority that the system's underlying mathematics cannot support.

Obscured Mechanics​

The text's anthropomorphic framing systematically obscures the material, technical, and economic realities of AI development. This framework holds that commercial incentives, data annotation and human feedback labor, environmental and compute costs, and developers' limited disclosure of training data, objectives, and feedback processes are standing conditions of the technology. By focusing on abstract concepts of "learning," "knowledge," and "emergence," the text renders these foundational conditions invisible.

Applying the "name the corporation" test reveals deep concealments. When the text states that models are "trained on large corpora of text, usually from the internet," it obscures the active decisions made by corporations like Anthropic and OpenAI. These firms deliberately scrape copyrighted material, personal data, and biased human output to build their datasets. By framing this mathematically as a system acquiring "knowledge," the text masks the contested legal and ethical nature of data extraction.

Similarly, the text's description of reinforcement learning obscures critical labor and technical dependencies. It notes that a model "is penalized" for wrong answers and "rewarded" for good ones. This passive framing completely erases the thousands of low-wage data annotators whose labor generates the feedback signals. It also hides the technical reality of reward models and proximal policy optimization (PPO) algorithms engineered by developers. The system does not learn from its mistakes like a human; it mathematically shifts its probability distributions based on scalar values derived from human labor. The wording has the effect of naturalizing this highly engineered, labor-intensive process as an organic cognitive development.

Furthermore, the framing of "emergence" obscures the massive material costs of these systems. When capabilities are described as spontaneously "emerging" from predictive learning, the text hides the staggering compute power, energy consumption, and capital expenditure required to train trillions of parameters. This framework reads the industry's push for larger models as a commercial imperative to achieve broad utility, but the text presents it purely as a scientific discovery about the "importance of scale." If these metaphors were replaced with mechanistic language, the immense human engineering, hidden labor, and corporate decision-making required to artificially mimic cognition would become starkly visible.

Context Sensitivity​

The distribution of anthropomorphic language across the text is highly uneven, creating a distinct rhetorical asymmetry that shapes how readers perceive the system's capabilities versus its limitations. The consciousness claims intensify significantly in the introductory sections and the concluding discussions of broader implications, while the middle sections detailing the architecture rely on more mechanistic, technical vocabulary.

In the opening sections, the text employs strong epistemic language, stating that models "learn the syntax and semantics" and show "parallels to complex cognitive capacities." However, when breaking down the transformer architecture (Figure 1), the text shifts to mechanical terms: the model "embeds each input word into a high-dimensional space" and "produces a probability distribution over next words." Yet, this mechanical grounding is consistently punctuated by jarring register shifts. In the very same diagram describing vectors and residual streams, the text literalizes a metaphor, labeling the MLP layer as "reasoning" and "recalling knowledge." This juxtaposition has a powerful effect on the reader: it leverages the rigorous, scientific language of vectors and matrices to grant empirical credibility to the profound consciousness claims of reasoning and knowing. The mechanical language validates the anthropomorphism.

This asymmetry is most pronounced when comparing capabilities and limitations. When describing the model's successes, the text uses highly agential language: the model "mixes logic," "reasons," and "decides it is ready." However, when discussing the challenges of comparing models to brains, the text retreats to mechanical limitations: models suffer from "contamination" in training data or demonstrate "relational generalization deficits." The capabilities are framed as the actions of an intelligent agent, while the limitations are framed as mechanical artifacts of the training regime. This inherited register of the field positions the reader to view the AI as fundamentally intelligent, with its errors being mere technical glitches rather than evidence of a profound lack of comprehension. The concentration of cognitive metaphors in the sections discussing the model's successes ensures the audience walks away with an inflated, agential picture of the system.

Accountability Synthesis​

Accountability Architecture

This section synthesizes the accountability analyses from Task 1, mapping the text's "accountability architecture"—who is named, who is hidden, and who benefits from obscured agency.

The text constructs a profound architecture of displaced responsibility, primarily by substituting the AI system as the active agent in place of human developers. While the source reading reveals that the text intends to make a scientific argument about the parallels between artificial models and biological brains, its vocabulary systematically relocates agency to the artifact, creating an accountability sink.

Throughout the text, human actors are largely unnamed or relegated to passive roles. The model "learns," "decides," "reasons," and "is penalized." The text mentions that "creators" might not know how a model solves a problem, or that "model developers" would like models to have certain properties, but these mentions serve to highlight the model's autonomy rather than the developers' responsibility. The resulting distribution of responsibility heavily favors the developers at corporations like Anthropic. By framing the system's advanced capabilities as "emergent" rather than engineered, and by describing its flawed outputs as analogous to "human reasoning" rather than statistical brittleness, the text diffuses responsibility into abstraction. If a deployed system causes harm by generating logically flawed but highly convincing text, the framing provided here suggests the failure is an organic cognitive quirk or an unforeseeable emergent property, rather than a failure of corporate design and testing.

Naming the actors would radically alter this landscape. If, instead of saying "the model can be trained... even if the creators do not know exactly the right way," the text stated, "Anthropic engineers deploy optimization algorithms that generate parameter configurations they cannot interpret," the risk profile changes entirely. It shifts from an awe-inspiring story of machine innovation to a concerning narrative of corporate negligence regarding uninterpretable systems. Replacing the passive "is penalized" with "human annotators rate the output, and developers update the parameters" restores the causal chain of human decision-making. The current distribution of responsibility serves the institutional interests of the AI industry, whether intended by the author or merely reproduced as the field's inherited register. It shields the massive, deliberate engineering effort behind an illusion of autonomous, emergent cognition, foreclosing questions about who chose the data, who defined the rewards, and who decided the uninterpretable system was safe for public deployment.

Conclusion: What This Analysis Reveals​

The Core Finding

The text's anthropomorphic architecture rests on three interconnecting patterns: the "autonomous learner," the "deliberative thinker," and the foundational "cognitive equivalence" pattern. The cognitive equivalence pattern—which maps mathematical components directly onto brain functions (e.g., labeling an MLP layer as "reasoning")—is the load-bearing structure of the discourse. By establishing early on that the model's architecture is functionally analogous to a mind, the text grants itself the rhetorical license to employ the other patterns. Once the system is accepted as a cognitive entity, the "autonomous learner" pattern can frame parameter optimization as the acquisition of "knowledge," and the "deliberative thinker" pattern can frame autoregressive generation as "deciding" it is ready to answer.

These patterns reinforce one another logically: because the model has "knowledge," it can "reason"; because it can reason, it can "decide." If the foundational cognitive equivalence is removed—if the MLP is just a non-linear vector transformation—the claim that the model "decides" collapses into mathematical absurdity. Furthermore, the text's vocabulary actively complicates its own scientific argument. While the text explicitly acknowledges methodological challenges and the lack of true structural homology between brains and transformers (Task 0), its persistent use of epistemic verbs ("knows," "understands") undercuts this caution. The language elevates behavioral parallels into claims of conscious action, ultimately displacing human responsibility by framing the artifact as an autonomous, reasoning agent.

Mechanism of the Illusion:​

This framework holds that computational systems process rather than know. The illusion of mind in this text is not a deliberate authorial trick, but a powerful reception effect generated by the curse of knowledge and the field's inherited register. The illusion takes hold through a specific temporal structure: the text first grounds the reader in complex, legitimate mechanical realities (residual streams, high-dimensional spaces) before subtly swapping the verbs.

The central move occurs when processing verbs blur into knowing verbs. The text explains that the model "predicts" tokens—a mechanical, statistical reality. But it quickly shifts, stating that through this prediction, the model learns "skills and knowledge." By projecting the author's own cognitive experience of learning onto the machine's statistical optimization, the wording has the effect of activating the reader's predispositions (the ELIZA effect). Readers are naturally inclined to attribute mind to fluent language; when a technical authority validates that inclination by calling vector transformations "reasoning" and text generation "deciding," the illusion crystallizes.

The explanation types amplify this. By relying heavily on functional and intentional explanations—describing what the system is trying to achieve rather than the math it executes—the text provides a narrative of purpose. The reader stops seeing a matrix multiplication and starts seeing a student "learning" or an agent "approaching" a problem. The illusion works because it offers a highly accessible, human-centric narrative that perfectly aligns with the fluent, human-like text the system ultimately generates.

Material Stakes:​

Categories: Epistemic, Regulatory/Legal, Economic

The metaphorical framings in this text generate concrete material consequences across multiple domains. Epistemically, when scientific literature frames language models as possessing "knowledge" and the capacity for "reasoning," it shifts how researchers and the public evaluate truth claims. If users believe a system "knows" a fact rather than "processes" a statistical correlation, they are more likely to deploy these models in high-stakes environments—such as medical diagnostics or legal research—without human verification. This epistemic reliance benefits the corporate developers who seek ubiquitous integration of their products, while the cost of factual hallucinations is borne by the end-users and the institutions that rely on them.

In the regulatory and legal domain, the language of "emergence" and "autonomous decision-making" creates a shield against liability. If policymakers accept the framing that models spontaneously "emerge" with capabilities that developers "do not know exactly the right way" to control, regulations will focus on managing autonomous AI agents rather than enforcing corporate product safety standards. This framing shifts the behavioral burden from the firms that design and deploy uninterpretable systems onto the society forced to mitigate their "emergent" harms.

Economically, the "illusion of mind" sustains massive capital valuation. By framing token prediction as "reasoning," companies justify trillions of dollars in compute and energy investments. If the technology were universally described mechanistically—as a brittle statistical engine that regurgitates human data patterns—its perceived economic ceiling would lower. The anthropomorphic vocabulary protects the speculative future value of the AI industry by maintaining the narrative trajectory toward artificial general intelligence.

AI Literacy as Counter-Practice:​

Practicing critical precision requires systematically replacing epistemic and conscious verbs with mechanistic ones, and actively naming the human actors obscured by agentless constructions. As demonstrated in the reframings, substituting "the model decides it is ready" with "the generation loop terminates upon predicting a stop token" forces a recognition of the system's rigid, mathematical nature. Similarly, replacing "to teach the model... knowledge" with "developers optimize the parameters" dismantles the illusion of an epistemic mind and restores the visibility of human engineering.

This reframing directly counters the material stakes outlined above. By removing consciousness verbs, precision language demands that we treat the system as a product dependent on human data, not an autonomous agent. Naming the corporations forces recognition of who designed the optimization, who profits from its deployment, and who must bear the liability for its failures.

However, systemic adoption of this literacy faces massive resistance. The incentives of the AI industry strongly favor anthropomorphic language; it drives marketing, sustains astronomical funding narratives, and shields developers from direct accountability. Furthermore, the inherited register of cognitive science and computer science relies on this shorthand for accessibility. Journals would need to mandate mechanistic translations of cognitive metaphors, and researchers would have to commit to dismantling their own convenient shorthand. Precision is costly for an industry whose valuation depends on the public believing they are building minds, not just highly complex calculators.

Path Forward​

Looking toward the future of the discourse, the vocabulary choices we normalize will dictate how society integrates AI technologies. Different stakeholders have competing priorities, and the resulting discourse ecology will shape what becomes tractable or invisible in AI governance.

If the current status quo of unacknowledged anthropomorphic shorthand ("the model knows," "the AI thinks") deepens, it will serve the marketing and funding priorities of the AI industry. This narrative resonance makes the technology accessible and awe-inspiring, driving adoption. However, it forecloses rigorous public understanding of statistical brittleness. In this future, regulatory frameworks may mistakenly attempt to govern AI as legal persons or autonomous agents, leaving the structural decisions of corporate developers largely unexamined and protecting them from product liability.

Conversely, if mechanistic precision ("the model generates text statistically associated with...") becomes the mandated norm in research and policy, the true nature of the technology becomes visible. This approach serves civil society, safety researchers, and regulators by making the exact dependencies (training data, compute, human annotation) tractable. However, this precision costs intuitive accessibility; the language becomes dense, alienating lay audiences, and stripping away the narrative power that drives investment.

A hybrid approach might involve structural institutional changes, such as journals and policymakers requiring transparency about discourse. A system card might explicitly state, "We use 'understands' functionally to describe vector retrieval; the system possesses no semantic awareness." This allows for the convenience of the inherited register while formally recognizing the system as an artifact. Ultimately, the choice of vocabulary is a choice of worldview. Mechanistic language centers human accountability and material reality, while anthropomorphic language centers the illusion of machine autonomy. Which future is desirable depends entirely on whether society values the myth of the artificial mind over the accountability of the human developer.

Extended Processing Summary​

Gemini's Processing Steps
What you are about to read…

Gemini refers to the text below as "thought summaries." This is an overt consciousness projection because 'intentions' are hallmarks of a conscious mind that 'knows' what it is doing and why. The concealed mechanistic process is probabilistic text generation. Treat this as just another rhetorical artifact—a way of making the model's processing legible. When the model writes "I should double check that...", it is mathematically altering its own probability distribution for the next token to be more accurate. The "I" is a functional gear in the machine, not a self-reference. This language is a deliberate interface design choice, not a biological reality. The developers could have programmed the model to output passive logs. Instead, they chose to simulate a narrator.

Initiating Discourse Analysis

I'm starting the deep dive into Lampinen's text. My primary goal is dissecting the use of AI metaphors and anthropomorphism within the document. I plan to closely examine how these elements shape the narrative and influence the reader's understanding of the AI-brain parallelism. I am focusing on identifying key phrases and concepts to establish a thorough framework for analysis.

Defining Project Parameters

I've established the foundational constraints for the project. I'm focusing on rigorous adherence to the JSON schema, specifically version 3.4.3. My role is defined as a critical discourse analyst, examining the "illusion of mind" and displaced human agency. The goal is to treat AI as artifacts, and not as minds, for the purposes of the analysis. I'm also ensuring that the output will be strictly valid JSON and that I'm mindful of enumerated types and word count minimums. Next, I must begin the analysis of Lampinen's text.

Analyzing Cognitive Parallels

I'm now diving deep into the text's core argument. It's fascinating how the document positions modern language models as having emergent parallels to human cognition, even with architectural differences from biological brains. I'm focusing on the history from connectionism to transformers, and the claim that predictive learning can yield cognitive abilities. I'm examining how they are studying LMs to gain insights into intelligence. I've noted that the focus avoids accountability or deployment safeguards.

Examining Cognitive Framing

I'm now zeroing in on the specific instances of cognitive framing within the text. The document employs terms like "recalling knowledge," "reasoning," and "deciding," attributing intentionality and cognitive processes to the AI model. I am cataloging the metaphors and anthropomorphisms, specifically focusing on how they present the model as an active, thinking agent, even while acknowledging the limitations of the analogy.

Identifying Cognitive Attribution

I am now closely examining passages that attribute cognitive capabilities to the model, such as the claim of autonomous problem-solving on page 5. I've noted the reinforcement of "behavioral training" to discourage specific answers and the attribution of cognitive capacity and experiential learning in various locations. I'm also dissecting the concept of organic growth and emergence in the context of predictive learning. I'm cataloging how these phrases shape the perception of the AI's abilities.

Deconstructing AI Attribution

I'm now identifying instances of anthropomorphism. I've noted how the text presents the model as a student, querying representations and selectively attending. I'm focusing on the projection of human pedagogy onto algorithmic weight adjustments and attributing human memory and logic to the model. I'm examining how these attributions shape the perception of AI capabilities.

Deconstructing Mind Projection

I am now focusing on several key instances. I am examining how "the model decides it is ready" projects human deliberation onto token generation. Also, I noted the projection of a dialogic response onto text continuation. Additionally, I am comparing language model behaviors with aspects of analogical reasoning, and exploring their learning of previously thought impossible structures.

Analyzing Knowledge Attribution

I am now focusing on the attribution of "innate knowledge" to the model, and how abstract knowledge is acquired. I've noted that the projection attributes epistemic possession to model parameters, creating a sense of an epistemic or conscious state. I am focusing on cataloging the degree to which this projection enhances AI capacity.

Reframing Mechanistic Verbs

I'm now zeroing in on reframing the provided quotes to reveal the underlying mechanisms. I'm replacing epistemic verbs with mechanistic verbs, thereby restoring human agency and reducing the degree to which AI is implied to possess human-like abilities. The goal is to reflect standard LM architecture. I am meticulously analyzing each passage and revising my initial thoughts in light of these constraints.

Revising Epistemic Attributions

I'm now revising the provided quotes, replacing subjective cognitive verbs with objective mechanistic descriptions, adhering to LM architecture. For example, "teach the model general language processing skills" now becomes "developers optimize the model's parameters." I am examining each quote and analyzing how each verb change affects the overall framing.

Observing Epistemic Displacement

I'm currently focusing on how the optimization process encodes structural correlations in model parameters, which renders a new interpretation of quote 8. I am also investigating the agency slippage, specifically analyzing the movement from mechanical concepts to agential attributions and the implications for user trust.

Synthesizing Accountability Frameworks

I'm synthesizing the accountability issues, noting the text's argument absolves developers via emergence framing. I'm focusing on "model decides" language, and how this obscures Anthropic's role. I'm noting the context sensitivity asymmetry in the text's structure, comparing the density of the introduction and conclusion, to the mechanistic explanations in the middle. I am currently examining the implication of the architecture's mathematical base and its relation to the language model's mind-like outputs.

Implementing Mechanistic Reframing

I am now implementing the mechanistic reframing as described, ensuring the JSON document accurately reflects the revised perspective. I am meticulously checking word counts against the strict minimums to ensure compliance with the formatting and argument-length constraints. I have updated the text with phrases that reflect LM architecture. I am reviewing the various reframing analyses.

Verifying Schema Adherence

I'm now implementing the reframing and reviewing all word counts, ensuring strict adherence to the defined minimums and maximums within each section. I'm focusing on "Task 0" to ensure the genre and author position are correctly categorized as "Preprint or technical report" and "Developer of the system discussed". I'm making sure my draft satisfies the word count and schema constraints.

Finalizing Mechanistic Summary

I am now performing a comprehensive review, verifying all word counts against schema requirements, especially in Task 5 and the Conclusion. I'm focusing on ensuring all sections precisely meet minimum and maximum word counts, as well as the single required instances of the structural lens phrase. I'm carefully incorporating Anthropic as the named corporation, according to the project's instructions. Finally, I'm generating the final, valid JSON output and ensuring there are no literal newlines.

Analysis Provenance

Run ID: 2026-10-17-emergent-parallels-between-language-mode-metaphor-l8pps1 Raw JSON: 2026-10-17-emergent-parallels-between-language-mode-metaphor-l8pps1.json Framework: Metaphor Analysis v6.8 Schema Version: 3.4.3 Generated: 2026-10-17T10:05:08.765Z

Discourse Depot © 2025 by TD is licensed under CC BY-NC-SA 4.0