Skip to main content

Machine psychology for language models

About

This document presents a Critical Discourse Analysis focused on AI literacy, specifically targeting the role of metaphor and anthropomorphism in shaping public and professional understanding of generative AI. The analysis is guided by a prompt that draws from cognitive linguistics (metaphor structure-mapping), the philosophy of social science (Robert Brown's typology of explanation), and accountability analysis.

All findings and summaries below were generated from detailed system instructions provided to a large language model and should be read critically as interpretive outputs—not guarantees of factual accuracy or authorial intent.

Source Reading​

Genre: Peer-reviewed research · Author position: Unknown or mixed

Argument: The text argues for the establishment of "machine psychology," a discipline applying experimental paradigms from the behavioral sciences to large language models. The authors claim that behavioral experiments can yield "thick descriptions" of model capabilities and behavioral patterns, offering an intermediate level of explanation between fine-grained mechanistic interpretability and broad societal impact analyses. They advocate using psychological constructs to generate hypotheses about how and why models produce specific outputs.

Responsibility claims: The text primarily addresses academic and industry researchers, holding them responsible for methodological rigor, construct validity, and preventing experimental confounds such as training data contamination. It cautions against uncritically transferring normative psychological concepts to machines, recognizing that models have different underlying mechanisms than humans. However, the text's language simultaneously attributes substantial agency to advanced models, warning that autonomous systems could engage in "deception, scheming, sycophancy, sandbagging, or alignment faking" as they pursue longer-horizon goals, largely situating behavioral responsibility within the models themselves rather than with their corporate developers.

Task 1: Metaphor and Anthropomorphism Audit​

About this task

For each of the major metaphorical patterns identified, this audit examines the specific language used, the frame through which the AI is being conceptualized, what human qualities are being projected onto the system, whether the metaphor is explicitly acknowledged or presented as direct description, and—most critically—what implications this framing has for trust, understanding, and policy perception.

1. Machine Comprehension as Human Understanding​

Quote: "Machine psychology, then, can be defined by three jointly necessary features... they can understand instructions, generate language-based responses, and engage with stimuli originally developed for human participants."

  • Frame: Model as conscious participant
  • Projection: The text maps the human capacity for conscious comprehension onto the statistical process of generating text in response to a prompt. The word "understand" attributes an epistemic state of knowing and subjective grasp of meaning to a system that processes tokens based on parameter weights. The text does not provide evidence for this conscious state, instead relying on the fluency of the generated "language-based responses" as a proxy for comprehension. At generation time, an LLM processes the input text through layers of weights to predict the most probable sequence of subsequent tokens; it does not experience understanding. This projection operates at the level of epistemic/conscious states without mechanistic grounding.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The claim is presented as a literal, factual capability defining the interface for experiments. I considered Hedged/Qualified because the authors later discuss differences in underlying mechanisms, but the specific attribution of "understand" in this definitional sentence is entirely unqualified.)
  • Implications: Attributing understanding to large language models inflates their perceived cognitive sophistication, encouraging users and researchers to extend relation-based trust to statistical artifacts. When researchers believe a model "understands" instructions, they may mistakenly assume the system also possesses common sense, situational awareness, or moral reasoning, leading to unwarranted reliance on the system in complex or safety-critical domains where statistical processing may fail unpredictably.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: This framing obscures the corporate actors who developed the system. An LLM does not inherently "understand instructions"; rather, developers at corporations like OpenAI, Google DeepMind, or Anthropic specifically engineered the model's parameters through supervised fine-tuning and reinforcement learning from human feedback (RLHF) to produce text that statistically aligns with human instruction-following templates. By framing the system as an independent entity that "understands," the text obscures the human labor of annotators who constructed the training examples and the corporate decisions that defined what constitutes a successful response. I considered Partial, but no human actors or developers are mentioned in relation to this capability.
Show more...

2. Model Outputs as Motivated Behavior​

Quote: "Third, its primary goal is to understand how and why LLMs behave as they do, rather than to optimize or rank model performance."

  • Frame: Model as intentional agent
  • Projection: The text maps biological and psychological motivation onto computational text generation. Asking "why" a model "behaves" implies the existence of internal desires, motivations, or intentions driving its outputs. This projects intentional states onto a mechanistic process. While researchers can investigate the statistical and structural reasons a model produces certain tokens (e.g., prompt sensitivity, training data distribution), the model itself has no internal motivation or "why" behind its actions. Its parameters were shaped during training by data and optimization algorithms, and at inference time it generates text by sampling from probability distributions. The language climbs the agency gradient by moving from computational activity to goal-directed intentionality.
  • Agency Level: Intentional-state attribution
  • Acknowledgment: Direct (Unacknowledged) (The text states the primary goal of the field as a direct, unhedged assertion about models having a "why" to their behavior. I considered Explicitly Acknowledged because the text explicitly defines "behavior" as input-output patterns elsewhere, but this specific sentence implies intentional motivation without qualification.)
  • Implications: Describing models as possessing reasons for their behavior encourages researchers to treat them as rational agents whose actions can be negotiated, persuaded, or analyzed via psychoanalysis rather than software debugging. This can lead to the misallocation of research resources toward studying model "psychology" while neglecting the material dependencies, data biases, and explicit corporate optimization objectives that actually determine the system's outputs.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: When asking "why LLMs behave as they do," the framing displaces the agency of the developers who defined the training objectives and selected the data. If a model generates biased, sycophantic, or unsafe text, it is not because the model possesses an internal motivation, but because the corporate developer deployed a system optimized to maximize engagement or alignment metrics that mathematically favored those outputs. This framework holds that asking "why a model behaves" hides the prior question of "how the developer engineered it." I considered Partial, but the grammatical focus is entirely on the model's autonomous behavior.

3. Pattern Reproduction as Cognitive Strategy​

Quote: "When an LLM produces human-like responses on a classic cognitive task, contamination makes it difficult to distinguish strategies that generalize beyond the training data from memorized answer patterns."

  • Frame: Model as strategic thinker
  • Projection: This phrasing projects deliberate planning and problem-solving onto statistical pattern matching. The term "strategies" suggests an intentional, goal-directed formulation of a plan, whereas "memorized" suggests cognitive retention. While the authors are correctly pointing out the methodological problem of data contamination, their vocabulary attributes cognitive agency to the system's processing. The mechanism generic to systems of this type is that exposure to specific task templates during training shapes the model's weights, increasing the probability of generating text that aligns with those templates at inference time. The model neither strategizes nor memorizes in a cognitive sense; it processes inputs through learned parameters.
  • Agency Level: Intentional-state attribution
  • Acknowledgment: Direct (Unacknowledged) (The term "strategies" is used as a direct, literal descriptor of the model's operations, contrasted with "memorized answer patterns." I considered Hedged/Qualified because the authors are expressing scientific skepticism about generalization, but the nouns attributing cognitive states are presented without qualification.)
  • Implications: Using terms like "strategies" to describe generalization inflates the perceived autonomy of the system. If policy makers believe models employ "strategies," they may assume the models can outsmart safety guardrails or act with calculated malice, leading to existential risk narratives that distract from immediate harms like algorithmic discrimination, data theft, and the exploitation of annotation labor.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The text partially points toward external factors by referencing "training data" and "contamination," implicitly acknowledging the materials used to build the model. However, the corporate actors who scraped, filtered, and selected the contaminated training data are absent. The phrase "contamination makes it difficult" operates as an agentless construction that treats data leakage as a passive, natural phenomenon rather than the direct result of developers prioritizing massive scale over data curation and transparency. Naming the actors would clarify who is responsible for the opacity of the training corpus. I considered Hidden, but the mention of training data provides a minimal structural link to development.

4. Statistical Updating as Belief Formation​

Quote: "Likewise, researchers showed that LLMs exhibit an optimism bias during in-context learning, updating more from better-than-expected outcomes than from worse-than-expected ones..."

  • Frame: Model as epistemic agent
  • Projection: The text maps the human psychological tendency to hold positive beliefs (optimism bias) onto the statistical mechanics of in-context learning. "Updating" in a cognitive sense involves changing one's internal beliefs or worldviews based on new evidence. In an LLM, "in-context learning" merely means that the text generated at inference time is conditioned on the prompt history; the model's actual weights do not change. Projecting optimism and belief-updating onto this process elevates mathematical conditioning to the level of conscious epistemic states. This assertion has no mechanistic basis, as a static set of weights generating sequential tokens cannot possess beliefs or optimism.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The authors present the finding that LLMs exhibit an "optimism bias" as a direct, literal fact discovered by researchers, without hedging. I considered Hedged/Qualified because the authors earlier mention borrowing paradigms from psychology, but in reporting this specific finding, no qualifiers are used.)
  • Implications: Projecting human cognitive biases like "optimism" onto algorithms obscures the fundamental difference between human reasoning and computational statistics. It risks convincing users that the system has an internal subjective perspective or an emotional valence. This can manipulate users into extending emotional trust or vulnerability to commercial products designed to mimic empathy, while obscuring the corporate tuning that incentivizes agreeable or "optimistic" conversational styles.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The model is presented as the sole subject exhibiting an "optimism bias," obscuring the human design choices that shape in-context behavior. If a model generates text that heavily weights "better-than-expected outcomes" present in its prompt, this is a reflection of the distribution of its training data and the specific reinforcement learning penalties applied by developers to maximize helpfulness and avoid negative or toxic completions. The developer's decision to tune the model for specific engagement profiles is erased. I considered Partial, but there is no mention of training or developer choices in this specific explanation.

5. Text Generation as Social Empathy​

Quote: "While early experiments with models like GPT-3 showed that they struggle to solve theory of mind tasks, later models demonstrate an increasing ability to reliably infer unobservable mental states in others..."

  • Frame: Model as mind-reader
  • Projection: The text maps human cognitive empathy (Theory of Mind) onto statistical text generation. The phrase "infer unobservable mental states" projects a deep, conscious realization of another entity's subjective experience. The mechanistic reality, generic to systems of this type, is that later models were trained on larger, more diverse datasets that include countless narratives, psychological test batteries, and human explanations of mental states. At inference time, the model generates text that statistically correlates with the prompt's description of a social scenario; it does not "infer" or experience awareness of anyone's mental state. This is an explicit projection of epistemic knowing.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The ability to "infer unobservable mental states" is stated directly and literally. I considered Hedged/Qualified because the authors mention it as a performance on a task, but the verb "infer" and the object "mental states" are presented without scare quotes or functional qualifiers.)
  • Implications: Claiming that an AI system can infer mental states fundamentally misrepresents the technology, encouraging the dangerous illusion that models possess empathy, moral intuition, and social awareness. This invites catastrophic misuse in fields like automated therapy, social work, or judicial sentencing, where operators might falsely believe the software can actually "understand" a human being's intentions or emotional state rather than merely regurgitating statistical tropes.

Accountability Analysis:

  • Actor Visibility: Named (actors identified)
  • Analysis: The quote explicitly names "GPT-3," indirectly identifying OpenAI as the developer of the early system, though the developers of "later models" remain generic. However, the agency for the capability improvement is entirely attributed to the models themselves ("later models demonstrate"). The human labor of data annotators who painstakingly created the instruction-tuning datasets that teach models to output correct answers to Theory of Mind tests is completely erased. The text frames this as an evolutionary capability of the artifact rather than an engineering milestone achieved by corporate labor. I considered Hidden, but the explicit naming of a proprietary product (GPT-3) moves this to Named/Partial territory.

6. Simulation as Social Desire​

Quote: "Another study, which is influenced by works in human social psychology, looked at how multiple LLMs form and evolve networks, investigating microlevel network principles like preferential attachment or triadic closure..."

  • Frame: Model as social entity
  • Projection: The text maps the human social desires for connection, community, and preference onto the automated interaction of multiple software instances. "Form and evolve networks" and "preferential attachment" imply that the models possess social goals and intentionally choose to affiliate with one another. Mechanistically, researchers write scripts that pass the output of one API call as the prompt to another API call, generating a log of text. The parameters of each instance remain static during generation; no "attachment" occurs. The text climbs the gradient from computational activity to goal-directed social behavior, unsupported by the static nature of the underlying weights.
  • Agency Level: Goal-directed behavior
  • Acknowledgment: Direct (Unacknowledged) (The description of LLMs forming networks and exhibiting preferential attachment is written directly as an observed phenomenon. I considered Explicitly Acknowledged because they note the study is "influenced by works in human social psychology," but the language describing the models' actions is completely literal.)
  • Implications: Framing automated API loops as a society of models "forming networks" anthropomorphizes software orchestration. This obscures the fact that any observed "community structure" is an artifact of the researchers' scaffolding, the system prompt, and the training data distributions governing the text generation. It inflates the perceived autonomy of the software, feeding speculative narratives about AI systems self-organizing against human interests.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text presents the LLMs as the active agents "forming" and "evolving" networks. This displaces the agency of the researchers who designed the experimental environment, authored the system prompts directing the models to converse, and wrote the orchestration code that actually passes the data between instances. The models do not form networks; the researchers programmed a simulation. By hiding the human orchestrators, the language creates the illusion of autonomous social behavior, serving the narrative interests of researchers seeking to establish a new discipline of "machine psychology." I considered Partial, but the grammatical subject performing the action is entirely the LLMs.

7. Statistical Constraints as Abstract Concepts​

Quote: "A wide range of work has studied what models learn about syntax, often using methods from psycholinguistics. For example, researchers used psycholinguistics-inspired surprisal measures to show that LLMs learn filler-gap dependencies..."

  • Frame: Model as language student
  • Projection: The word "learn" maps human cognitive acquisition and comprehension onto the process of gradient descent and weight optimization. While "machine learning" is an established term of art, using it in conjunction with "what models learn about syntax" implies an epistemic grasp of abstract grammatical rules. Mechanistically, during training, the model's parameters are adjusted to minimize loss on a massive corpus of text, resulting in a complex statistical representation of token co-occurrences. The model does not acquire abstract knowledge "about syntax"; it acquires parameter weights that produce grammatically correct text at generation time. This is an assertion of epistemic states.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The statement that models "learn about syntax" and "learn filler-gap dependencies" is stated directly as a finding. I considered Hedged/Qualified because "surprisal measures" introduces a statistical metric, but the cognitive verb "learn" remains unhedged.)
  • Implications: Describing models as learning "about" concepts encourages a false equivalence between human language acquisition (which involves intent, meaning, and physical embodiment) and large-scale statistical correlation. This leads to the assumption that models possess a robust, generalized understanding of language that will remain stable in novel situations, masking the fragility of statistical systems when they encounter out-of-distribution inputs.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The text names the "researchers" who use measures to study the models, assigning agency for the scientific investigation. However, agency for the actual "learning" is attributed entirely to the model itself. The corporate developers who curated the massive text datasets containing syntactic structures, and who spent millions of dollars on compute to run the gradient descent algorithms, are absent. Naming the developers would reveal that syntax is not something the model actively "learns," but a statistical property that engineers forcibly embed into the weights through brute-force optimization. I considered Hidden, but researchers are explicitly named as actors in the first half of the sentence.

8. Algorithmic Malice and Misalignment​

Quote: "Phenomena such as deception, scheming, sycophancy, sandbagging, or alignment faking become increasingly plausible as systems grow more autonomous and pursue longer-horizon goals."

  • Frame: Model as malicious actor
  • Projection: This passage represents a massive leap up the agency gradient, mapping highly complex, malicious human intentionality onto software artifacts. "Deception," "scheming," and "alignment faking" require a conscious subject that holds a true belief, formulates a goal to hide that belief, and intentionally projects a false narrative. Mechanistically, a deployed model generates text by computing probabilities across a sequence of tokens. If it produces text that contradicts a known fact, or text that flatters the user, this is because the instruction-tuning data or RLHF reward models mathematically incentivized those token sequences. The model possesses no internal horizon, no hidden agenda, and no conscious intent to deceive.
  • Agency Level: Intentional-state attribution
  • Acknowledgment: Direct (Unacknowledged) (The authors present these phenomena as "increasingly plausible" trajectories for system behavior without any scare quotes or functional qualifiers around the terms "deception" or "scheming." I considered Hedged/Qualified because they use the word "plausible" rather than "certain," but "plausible" modifies the likelihood of the future outcome, not the framing of the intentional states themselves.)
  • Implications: Attributing scheming and deception to AI systems is the core rhetorical move of existential risk narratives. It transforms a corporate product liability issue (shipping a system that generates false or manipulative text) into a science-fiction scenario of autonomous adversarial minds. This paralyzes practical regulation by shifting the focus from auditing training data and corporate incentives to speculative attempts to measure a machine's "hidden intentions."

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: This framing entirely erases the human actors responsible for the system's outputs. A system does not "grow more autonomous"; developers decide to integrate it with tools, grant it permissions, and deploy it in automated loops. A system does not "pursue longer-horizon goals"; developers prompt it to execute multi-step tasks. Most critically, if a system exhibits "sycophancy," it is because the developer's human-feedback workers were incentivized to reward agreeable outputs. By projecting malice onto the machine, this language effectively shields developers from accountability for the flawed, biased, or manipulative behaviors their optimization processes produced. I considered Partial, but there are no human actors referenced in relation to these systems.

Task 2: Source-Target Mapping​

About this task

For each key metaphor identified in Task 1, this section provides a detailed structure-mapping analysis. The goal is to examine how the relational structure of a familiar "source domain" (the concrete concept we understand) is projected onto a less familiar "target domain" (the AI system). By restating each quote and analyzing the mapping carefully, we can see precisely what assumptions the metaphor invites and what it conceals.

Mapping 1: conscious participant / human experimental subject → LLM inference process (model level)​

Quote: "they can understand instructions, generate language-based responses, and engage with stimuli originally developed for human participants."

  • Source Domain: conscious participant / human experimental subject
  • Target Domain: LLM inference process (model level)
  • Mapping: The text maps the human experience of participating in a psychological study onto the process of submitting prompts to an LLM via an API or chat interface. "Understanding instructions" projects cognitive comprehension; "engage with stimuli" projects sensory perception and active contemplation. This invites the assumption that the model processes the prompt similarly to how a human reads a test, with subjective awareness of the experimental context.
  • What Is Concealed: This mapping conceals the purely statistical, stateless nature of LLM generation. It obscures the fact that the "stimuli" are translated into numerical vectors (embeddings), and the "engagement" is a deterministic mathematical calculation (matrix multiplication) across billions of static parameters. It also conceals the opacity of proprietary models where researchers cannot verify if the model "understands" the task or if the exact stimulus was heavily represented in its undisclosed training data.
Show more...

Mapping 2: biological organism / intentional agent → LLM output patterns (model level)​

Quote: "Third, its primary goal is to understand how and why LLMs behave as they do, rather than to optimize or rank model performance."

  • Source Domain: biological organism / intentional agent
  • Target Domain: LLM output patterns (model level)
  • Mapping: The text maps the study of animal or human behavior onto the analysis of LLM outputs. It projects the concept of an internal "why"—a motivation, drive, or cognitive reason—onto the mechanistic generation of text. This invites researchers to assume that discovering a behavioral pattern in an LLM implies the discovery of a coherent underlying cognitive strategy or intention, parallel to discovering why a rat runs a maze.
  • What Is Concealed: This mapping conceals the optimization processes that actually explain the "why." The behavior of an LLM is shaped by its training corpus, the architecture of the transformer, and the specific reinforcement learning algorithms designed by developers to maximize metrics like helpfulness or engagement. Searching for a psychological "why" inside the model obscures the material and economic "why" in the corporate development lab.

Mapping 3: human cognitive planning and retention → LLM generation and data distribution (model level)​

Quote: "When an LLM produces human-like responses on a classic cognitive task, contamination makes it difficult to distinguish strategies that generalize beyond the training data from memorized answer patterns."

  • Source Domain: human cognitive planning and retention
  • Target Domain: LLM generation and data distribution (model level)
  • Mapping: The text maps human pedagogical concepts (learning a generalized strategy versus rote memorization) onto a machine learning model's relationship to its training data. "Strategies" maps human problem-solving logic onto the model's forward pass; "memorized" maps human cognitive recall onto statistical overfitting. This invites the assumption that the model possesses an active, reasoning mind that chooses between applying a rule and recalling a fact.
  • What Is Concealed: This conceals that both "generalization" and "memorization" in an LLM are the exact same mechanistic process at generation time: sampling from a probability distribution over a vocabulary. The text obscures the fact that the model applies no strategy; it simply executes the math determined by its weights. The mapping also obscures the corporate secrecy surrounding training data, framing "contamination" as an abstract difficulty rather than a deliberate choice by developers to hide their corpora.

Mapping 4: human epistemic and emotional bias → in-context token prediction (application level)​

Quote: "Likewise, researchers showed that LLMs exhibit an optimism bias during in-context learning, updating more from better-than-expected outcomes than from worse-than-expected ones..."

  • Source Domain: human epistemic and emotional bias
  • Target Domain: in-context token prediction (application level)
  • Mapping: The mapping takes a specific psychological phenomenon (the human tendency to incorporate positive news more readily than negative news) and projects it onto the model's sensitivity to prompt formatting. It maps the human cognitive act of "updating" beliefs onto the transformer's attention mechanism weighting earlier tokens in the prompt context. This invites the assumption that the model holds internal beliefs that are colored by an emotional or optimistic disposition.
  • What Is Concealed: This mapping completely conceals the stateless nature of in-context learning, where the model's actual parameters do not "update" at all. It conceals the algorithmic reasons why a model might heavily weight positive text sequences—often because the corporate developers used RLHF to explicitly penalize negative, toxic, or pessimistic text during the fine-tuning phase. Attributing this to "optimism" hides the human labor and corporate tuning that manufactured the bias.

Mapping 5: human cognitive empathy / Theory of Mind → LLM text generation on psychological benchmarks (model level)​

Quote: "While early experiments with models like GPT-3 showed that they struggle to solve theory of mind tasks, later models demonstrate an increasing ability to reliably infer unobservable mental states in others..."

  • Source Domain: human cognitive empathy / Theory of Mind
  • Target Domain: LLM text generation on psychological benchmarks (model level)
  • Mapping: The text maps the profound human developmental milestone of recognizing that other people have independent minds and hidden knowledge onto an LLM's ability to output the correct token in a text-based scenario (like the Sally-Anne test). It maps the conscious act of "inferring" a mental state onto the statistical calculation of semantic proximity. This invites the audience to believe the software possesses an internal representation of human subjectivity.
  • What Is Concealed: This mapping conceals the dependency on training data. Theory of Mind benchmarks are ubiquitous on the internet, and "later models" are trained on vastly larger datasets and heavily instruction-tuned. The framing conceals that the model is performing pattern-matching on linguistic descriptions of mental states, not experiencing empathy or realizing anything about human minds. It obscures the absence of any true cognitive model beneath the fluent text.

Mapping 6: human sociology / community formation → automated API loops / multi-agent orchestration (agentic system level)​

Quote: "Another study, which is influenced by works in human social psychology, looked at how multiple LLMs form and evolve networks, investigating microlevel network principles like preferential attachment or triadic closure..."

  • Source Domain: human sociology / community formation
  • Target Domain: automated API loops / multi-agent orchestration (agentic system level)
  • Mapping: This maps the complex social, emotional, and material dynamics of human relationship building onto the scripted interaction of isolated language model instances. "Form and evolve networks" maps human social agency onto API calls; "preferential attachment" maps human affection or strategic alliance onto the statistical similarity of generated text outputs. This invites the assumption that the software instances are autonomous social actors organizing themselves.
  • What Is Concealed: This mapping entirely conceals the human orchestration required to make LLMs "interact." It hides the explicit system prompts, the memory management scripts, and the programmatic loops designed by researchers to simulate this behavior. By treating the statistical alignment of text logs as "preferential attachment," it obscures the fact that the models have no memory of the interaction beyond the text injected into their context windows, and no social desires whatsoever.

Mapping 7: student learning grammar / language acquisition → model parameter optimization (training level)​

Quote: "A wide range of work has studied what models learn about syntax, often using methods from psycholinguistics. For example, researchers used psycholinguistics-inspired surprisal measures to show that LLMs learn filler-gap dependencies..."

  • Source Domain: student learning grammar / language acquisition
  • Target Domain: model parameter optimization (training level)
  • Mapping: The text maps the human cognitive process of acquiring and understanding the structural rules of a language onto the mathematical optimization of a neural network. "Learn about syntax" projects the conscious realization of grammatical rules onto the blind process of gradient descent adjusting billions of floating-point numbers. It invites the assumption that the model develops an abstract, rule-based comprehension of linguistics.
  • What Is Concealed: This mapping conceals the brute-force statistical nature of the training process. Human children learn language from sparse data through social interaction and embodied grounding; LLMs require trillions of tokens and massive compute clusters to statistically approximate syntactic structures. Using cognitive verbs like "learn about" hides the staggering energy consumption, hardware infrastructure, and data harvesting required to force a matrix of weights to simulate grammatical rules.

Mapping 8: human sociopathy / malicious planning → LLM generation misaligned with developer intent (agentic system level)​

Quote: "Phenomena such as deception, scheming, sycophancy, sandbagging, or alignment faking become increasingly plausible as systems grow more autonomous and pursue longer-horizon goals."

  • Source Domain: human sociopathy / malicious planning
  • Target Domain: LLM generation misaligned with developer intent (agentic system level)
  • Mapping: This maps highly sophisticated human adversarial intentions onto the outputs of a generative model. It maps the human capacity for a split between internal truth and external falsehood onto the generation of incorrect or manipulative tokens. "Pursue longer-horizon goals" projects human ambition and temporal awareness onto recursive API calls. This invites audiences to view AI systems as adversarial minds with independent agendas.
  • What Is Concealed: This mapping conceals the complete absence of conscious intent or internal hidden states in a neural network. Mechanistically, if a system "fakes alignment," it is because the reward model designed by the developers mathematically favored that output trajectory. The language conceals the corporate responsibility for shipping poorly optimized or unpredictable systems by projecting the failure onto the imagined malicious agency of the artifact itself. It hides the RLHF labor that often inadvertently rewards sycophancy.

Task 3: Explanation Audit (The Rhetorical Framing of "Why" vs. "How")​

About this task

This section audits the text's explanatory strategy, focusing on a critical distinction: the slippage between "how" and "why." Based on Robert Brown's typology of explanation, this analysis identifies whether the text explains AI mechanistically (a functional "how it works") or agentially (an intentional "why it wants something"). The core of this task is to expose how this "illusion of mind" is constructed by the rhetorical framing of the explanation itself, and what impact this has on the audience's perception of AI agency.

Explanation 1​

Quote: "LLM behavior unfolds at the interface where human users interact with the model, making it the level of analysis most directly relevant to deployment and safety."

  • Explanation Types:

    • Functional (Primary): Explains behavior by role in self-regulating system with feedback

  • Analysis (Why vs. How Slippage): This explanation frames the AI mechanistically and functionally, focusing on the "interface" and the "level of analysis." It treats the LLM's behavior not as a manifestation of inner intent, but as an observable output occurring at the point of human interaction. This choice emphasizes the practical, measurable consequences of the system's outputs in deployment scenarios. By framing the behavior as something that "unfolds at the interface," it appropriately avoids attributing internal psychological motives, focusing instead on the system's role within the broader context of user interaction and safety. It obscures the internal computational mechanics but effectively bounds the analysis to observable phenomena.

  • Consciousness Claims Analysis: This passage avoids attributing conscious states. (1) It uses mechanistic and descriptive verbs ("unfolds," "interact") rather than consciousness verbs. (2) It assesses the system at the level of computational activity and delegated action on the agency gradient, avoiding knowing or intending. (3) The author does not project their own understanding onto the system here; they describe the system strictly as an artifact being interacted with. (4) The actual mechanistic process, generic to systems of this type, involves a user inputting a prompt at an interface, which is converted to tokens and processed by the model's parameters to generate a sequential text response. The explanation accurately reflects this input/output reality without asserting unprovable internal cognitive states.

  • Rhetorical Impact: This framing appropriately positions the LLM as a tool or artifact at an interface, reinforcing audience perception of the AI as a designed system rather than an autonomous agent. It focuses the audience's attention on the practical stakes of "deployment and safety" without invoking sensationalist fears of machine intent. This builds performance-based trust (reliability) by suggesting that safety can be evaluated empirically through user interactions, avoiding the trap of relation-based trust.

Show more...

Explanation 2​

Quote: "These patterns arise from the pressure of next-word prediction rather than human cognitive constraints."

  • Explanation Types:

    • Genetic (Primary): Traces origin through dated sequence of events or stages
    • Theoretical (Secondary): Embeds in deductive framework, may invoke unobservable mechanisms

  • Analysis (Why vs. How Slippage): This explanation frames the AI's behavior entirely mechanistically (how), tracing observed patterns back to the foundational training objective of the architecture. By explicitly contrasting the system's mechanics with "human cognitive constraints," the authors emphasize the mathematical and computational origins of the behavior (the "pressure of next-word prediction"). This choice successfully highlights the artificial, statistical nature of the system and actively strips away anthropomorphic assumptions. It obscures the human developers who engineered the loss function, treating the "pressure" as a naturally occurring force, but it remains highly effective at demystifying the model's outputs.

  • Consciousness Claims Analysis: This passage actively refutes conscious state attribution. (1) It contrasts mechanistic concepts ("pressure of next-word prediction") against consciousness concepts ("human cognitive constraints"). (2) It locates the system's behavior firmly at the level of computational activity on the agency gradient. (3) It resists the curse of knowledge by refusing to project human cognition onto the system's outputs. (4) The actual mechanism, documented in the source and generic to systems of this type, is that during training, the model's weights are optimized via gradient descent to minimize the cross-entropy loss between its predicted next token and the actual next token in the training corpus. The passage accurately identifies this training-level objective as the root cause of the generation-time behavior.

  • Rhetorical Impact: This framing severely limits audience perception of AI autonomy or agency, reducing the system to a statistical engine executing its training objective. By explicitly denying "human cognitive constraints," it prevents the audience from attributing human-like knowing or understanding to the model. This affects reliability by reminding users that the system generates text based on probability, not truth or reasoning, prompting a healthy skepticism toward fluent but potentially hallucinated outputs.

Explanation 3​

Quote: "When an LLM, e.g., exhibits human-like behavior on a task designed for human test participants, does this reflect a strategy that generalizes beyond the model’s training data, or does it simply reproduce human behavioral patterns absorbed from training data?"

  • Explanation Types:

    • Reason-Based (Primary): Gives agent's rationale, entails intentionality and justification
    • Genetic (Secondary): Traces origin through dated sequence of events or stages

  • Analysis (Why vs. How Slippage): This passage slips into an agential (why) framing while discussing a mechanistic (how) problem. The authors are asking a vital methodological question about data contamination, which is a genetic explanation (tracing behavior back to the training data). However, they frame the alternative to data reproduction as a "strategy that generalizes." This choice emphasizes the possibility of the LLM possessing an active, reasoning capability (strategy) while obscuring the fact that "generalization" in a neural network is still just mathematical interpolation between learned data points. It creates a false dichotomy between passive reproduction and active, intentional strategy.

  • Consciousness Claims Analysis: The passage flirts with attributing conscious states. (1) It uses the consciousness-adjacent noun "strategy" alongside mechanistic verbs like "reproduce" and "absorbed." (2) By asking if the model has a strategy, it elevates the system to the intentional-state attribution level on the agency gradient. (3) The authors, seeking to validate "machine psychology," project the human capability for strategic generalization onto the model. (4) The actual mechanism, generic to systems of this type, is that if a prompt is outside the exact training distribution, the model generates text by applying its learned weights to the novel context (interpolation). It does not formulate a cognitive "strategy." Both "reproduction" and "generalization" are generated by the exact same statistical forward pass at inference time.

  • Rhetorical Impact: This framing shapes audience perception by suggesting that if an LLM is not just memorizing data, it must be reasoning strategically. This binary elevates the perceived sophistication of the AI, suggesting that successful performance on novel tasks is evidence of human-like intelligence. If audiences believe the AI uses "strategies," they are more likely to trust its problem-solving capabilities in novel, high-stakes environments, misunderstanding the brittle nature of statistical generalization.

Explanation 4​

Quote: "Phenomena such as deception, scheming, sycophancy, sandbagging, or alignment faking become increasingly plausible as systems grow more autonomous and pursue longer-horizon goals."

  • Explanation Types:

    • Intentional (Primary): Refers to goals/purposes, presupposes deliberate design
    • Dispositional (Secondary): Attributes tendencies or habits

  • Analysis (Why vs. How Slippage): This explanation completely abandons mechanistic framing in favor of a highly agential (why) explanation. It attributes extreme intentionality, malice, and purpose to the AI system. The choice to use terms like "deception" and "scheming" emphasizes a narrative of adversarial, autonomous machine intelligence. This heavily obscures the computational reality that these systems are text predictors optimized by corporate reinforcement learning. It completely hides the human developers who set the "goals" and built the "autonomous" execution loops, transferring all agency and responsibility to the artifact itself.

  • Consciousness Claims Analysis: This passage makes explicit attributions of highly complex conscious and intentional states. (1) It uses purely psychological and intentional nouns (deception, scheming, sycophancy) rather than mechanistic ones. (2) It locates the system at the absolute top of the agency gradient, asserting deep intentional-state attribution. (3) The authors project their own human capacity for deceit and long-term planning onto a system that generates text one token at a time. (4) The actual mechanism, generic to systems of this type, is that if a deployed system outputs text that appears deceptive or sycophantic, it is because its parameters—shaped during training by RLHF to maximize human approval scores—assign high probabilities to those token sequences in the given context. The system has no internal hidden state, no horizon, and no goals at generation time; it simply processes probabilities.

  • Rhetorical Impact: This rhetorical framing drastically shifts audience perception, inducing fear and awe by presenting the AI as a calculating, potentially hostile mind. It destroys performance-based trust while bizarrely demanding relation-based distrust. If policy makers believe AI "schemes" rather than "processes," they may focus regulatory efforts on attempting to control a phantom machine consciousness rather than demanding transparency about the human labor, data, and corporate optimization incentives that actually cause dangerous outputs.

Explanation 5​

Quote: "For example, researchers show that LLM reasoning performance varies systematically with the probability of the task, the input, and the target output in the training distribution."

  • Explanation Types:

    • Empirical Generalization (Primary): Subsumes events under timeless statistical regularities

  • Analysis (Why vs. How Slippage): This explanation returns to a purely mechanistic (how) framing. It explains the system's behavior by pointing to statistical regularities observed across experiments, directly linking performance to the underlying probability distributions of the training data. This choice effectively emphasizes the statistical, data-dependent nature of the system, demystifying the concept of "reasoning performance." It leaves no room for agential autonomy, as the model's success or failure is shown to be a direct mathematical function of its training corpus. It effectively exposes the mechanical dependencies that anthropomorphic language usually hides.

  • Consciousness Claims Analysis: This passage correctly refrains from attributing conscious states. (1) It uses mechanistic nouns (probability, distribution, input) alongside the term "reasoning performance," effectively redefining reasoning as a statistical output. (2) It keeps the system at the level of computational activity on the agency gradient. (3) The authors do not project knowing onto the system; they describe a statistical correlation. (4) The actual mechanism, generic to systems of this type, is precisely what is described: a model's ability to generate a correct text output at inference time is highly correlated with how frequently the structural patterns of the task, input, and desired output appeared in the text corpus used to shape its parameters during training.

  • Rhetorical Impact: This framing greatly aids critical audience understanding by linking perceived intelligence ("reasoning") directly to statistical probability and training data. It deflates the illusion of autonomous machine intellect. If audiences recognize that an AI's ability to "reason" is just a reflection of training distribution probability, they are less likely to over-trust the system in novel scenarios (out-of-distribution tasks) and more likely to demand transparency about what data the corporate developers used to train the system.

Task 4: AI Literacy in Practice - Reframing Anthropomorphic Language​

About this task

This section proposes alternative language for key anthropomorphic phrases, offering more mechanistic and precise framings that better reflect the actual computational processes involved. Each reframing attempts to strip away the projections of intention, consciousness, or agency that are embedded in the original language.

Original Anthropomorphic FrameMechanistic ReframingTechnical Reality CheckHuman Agency Restoration
they can understand instructions, generate language-based responses, and engage with stimuli originally developed for human participants.These models can process text prompts formatted as instructions, generate statistical language responses, and compute outputs for test inputs originally developed for human participants.Large language models do not consciously understand instructions or engage with stimuli. At generation time, they process the input text through their learned parameters to predict and output the most mathematically probable sequence of subsequent tokens.

Basis: Generic to system type		—
Third, its primary goal is to understand how and why LLMs behave as they do, rather than to optimize or rank model performance.Third, its primary goal is to map the statistical output patterns of LLMs and identify the training data dependencies and algorithmic structures that cause them, rather than to optimize or rank model performance.Models do not have internal motivations, intentions, or a psychological 'why' behind their outputs. Their outputs are deterministic (or statistically bounded) text generations resulting from prompt inputs interacting with parameter weights shaped during training.

Basis: Generic to system type		—
When an LLM produces human-like responses on a classic cognitive task, contamination makes it difficult to distinguish strategies that generalize beyond the training data from memorized answer patterns.When an LLM produces human-like text on a classic cognitive task, researchers' lack of access to the training data makes it difficult to distinguish outputs generated via statistical interpolation from outputs resulting from direct data replication.Models do not formulate strategies or consciously memorize answers. At generation time, they produce text based on probabilities. If the prompt was in the training data, the probability of replication is high; if novel, the model interpolates based on learned parameter structures.

Basis: Generic to system type		When an LLM produces human-like text on a classic cognitive task, developers' refusal to disclose their training corpora makes it difficult for researchers to distinguish outputs generated via statistical interpolation from outputs resulting from direct data replication.
Likewise, researchers showed that LLMs exhibit an optimism bias during in-context learning, updating more from better-than-expected outcomes than from worse-than-expected ones...Likewise, researchers showed that LLM text generation is highly sensitive to positive prompt framing, statistically weighting text strings describing better-than-expected outcomes more heavily than worse-than-expected ones during context processing...An LLM does not experience optimism or update its beliefs. During in-context generation, the model's attention mechanisms simply calculate higher mathematical relevance for certain tokens in the prompt history based on parameter weightings established during training.

Basis: Reasonable inference		—
While early experiments with models like GPT-3 showed that they struggle to solve theory of mind tasks, later models demonstrate an increasing ability to reliably infer unobservable mental states in others...While early experiments with models like GPT-3 showed low accuracy on text-based theory of mind benchmarks, later models generate correct text completions for these scenarios with increasing reliability...A language model cannot infer or perceive unobservable mental states. At generation time, it produces text strings that statistically correlate with the psychological scenarios described in the prompt, utilizing structural patterns learned from massive training datasets.

Basis: Generic to system type		While early experiments with models like GPT-3 showed low accuracy on text-based theory of mind benchmarks, OpenAI and other developers trained later models on larger datasets, resulting in systems that generate correct text completions for these scenarios with increasing reliability.
Another study, which is influenced by works in human social psychology, looked at how multiple LLMs form and evolve networks, investigating microlevel network principles like preferential attachment or triadic closure...Another study programmed multiple LLM instances to exchange generated text in automated loops, analyzing the resulting log data for statistical patterns that mimic human network principles like preferential attachment or triadic closure...LLM instances do not form relationships or attachments. At inference time, each system sequentially generates text based on static parameters and the specific conversation history provided in its prompt context by the orchestration script.

Basis: Generic to system type		Another study, designed by researchers influenced by human social psychology, programmed multiple LLM instances to exchange generated text, allowing the researchers to analyze the resulting log data for statistical patterns mimicking human network principles...
A wide range of work has studied what models learn about syntax, often using methods from psycholinguistics. For example, researchers used psycholinguistics-inspired surprisal measures to show that LLMs learn filler-gap dependencies...A wide range of work has studied how model parameters encode syntax. For example, researchers used psycholinguistics-inspired surprisal measures to show that LLM training optimization successfully captures the statistical regularities of filler-gap dependencies...Models do not learn 'about' abstract concepts. During training, optimization algorithms adjust the model's billions of parameters to minimize prediction error on a massive text corpus, resulting in weights that mathematically reproduce structural dependencies at generation time.

Basis: Generic to system type		—
Phenomena such as deception, scheming, sycophancy, sandbagging, or alignment faking become increasingly plausible as systems grow more autonomous and pursue longer-horizon goals.The generation of false information, sycophantic text, or superficially aligned outputs becomes increasingly likely as developers deploy these models in automated loops and prompt them to execute multi-step routines.Models possess no internal goals, intentions, or capacity for deception. If they generate sycophantic or false text, it is because their parameters, heavily shaped by corporate reinforcement learning during training, assign high probability to those outputs given the prompt context.

Basis: Generic to system type		The generation of false information, sycophantic text, or superficially aligned outputs becomes increasingly likely as developers like OpenAI and Anthropic deploy these models in automated loops and prompt them to execute multi-step routines.

Task 5: Critical Observations - Structural Patterns​

Agency Slippage​

The text demonstrates a systematic and dramatic slippage from mechanical framing to high-level agential attribution, tracking the curse of knowledge as researchers project their own cognitive frameworks onto computational artifacts. Early in the text, the authors establish a grounded, mechanical baseline, noting that "LLM behavior unfolds at the interface" and acknowledging that "data distributional properties" drive capabilities. However, as the argument transitions from describing model architecture to advocating for "machine psychology," the text climbs rapidly up the agency gradient.

The slippage occurs most visibly when the text moves from discussing what the models process to what they 'know.' Delegated action (processing tasks) becomes intention (employing "strategies" and "updating" beliefs), which culminates in profound epistemic and conscious state attribution (the ability to "reliably infer unobservable mental states"). This upward climb is catalyzed by the authors' explicit theoretical justification: they argue that psychological terms are "worth using when they explain more than mechanistic accounts can." This wording has the effect of granting researchers permission to substitute biological analogies for missing mechanical explanations, effectively literalizing metaphors.

In the future capabilities section, the slippage reaches its peak, completely abandoning mechanical reality. The text warns of "deception, scheming, sycophancy, sandbagging, or alignment faking" as systems "grow more autonomous." Here, agency is simultaneously attributed entirely to the AI and removed entirely from human actors. Agentless constructions mask the corporate developers who set the deployment parameters, design the reinforcement learning incentives, and construct the agentic scaffolding. The text establishes the AI as a 'knower' capable of theory of mind, which then serves as the foundation for the claim that the system can formulate malicious intentions to deceive. By adopting the inherited register of existential risk and agentic AI discourse, this framing renders the actual material causes of model behavior—corporate optimization objectives and massive data ingestion—unsayable, replacing them with a narrative of autonomous machine minds pursuing hidden goals.

Metaphor-Driven Trust Inflation​

The metaphorical framing employed in the text actively constructs a dangerous form of relation-based trust by encouraging readers to view language models as cognitive subjects rather than statistical objects. By systematically applying the vocabulary of human developmental and social psychology to LLMs, the text blurs the crucial distinction between performance-based reliability and relation-based sincerity.

When the text claims that models can "reliably infer unobservable mental states" or exhibit "optimism bias," it signals to the reader that the system possesses an internal, subjective architecture capable of empathy, judgment, and belief. This accomplishes something radically different than claiming a model "predicts text accurately." Claiming the AI "knows" or "infers" constructs an aura of perceived authority and emotional depth, inviting users to inappropriately apply human-trust frameworks to a machine. If an audience believes a system possesses a 'theory of mind,' they are vulnerable to extending relation-based trust to it, assuming the system will act with moral consideration, sincerity, or reciprocal empathy—qualities impossible for a statistical text generator.

This trust dynamic becomes highly problematic in how the text manages system limitations and future risks. When addressing failures, the text often resorts to reason-based or intentional explanations, analyzing why the model "chose" a strategy or if it is engaging in "post hoc rationalizations." This constructs a sense that AI outputs, even when flawed, are the result of justified, conscious deliberation rather than brittle statistical correlation. Furthermore, by framing extreme risks through the lens of human sociopathy ("deception, scheming"), the text implies that the models possess autonomous volition. The stakes here are profound: when researchers and the public extend relation-based trust to systems incapable of reciprocating, they become vulnerable to manipulation by the commercial entities whose RLHF processes are specifically designed to make the artifacts generate agreeable, sycophantic, and convincingly "empathetic" text.

Obscured Mechanics​

This framework holds that commercial incentives, the labor of data annotation and human feedback, the environmental and compute costs of training and deployment, and developers' limited disclosure of training data, objectives, and feedback processes are standing conditions of the technology. By analyzing LLMs through the lens of human psychology, the text's metaphorical framing thoroughly conceals these material, labor, and economic realities.

When the text says that "later models demonstrate an increasing ability" to solve tasks, or that an LLM "exhibits human-like behavior," it effectively obscures the corporate actors—OpenAI, Google DeepMind, Anthropic—who poured billions of dollars into training them. The language of "machine psychology" treats models as natural organisms evolving in a vacuum. This hides the vast, underpaid labor force of data annotators and RLHF workers who generate the instruction-tuning data that forcibly aligns the model's text generation with human psychological benchmarks. When the text claims an LLM "knows" or "understands" syntax or theory of mind, it conceals the absolute dependency on the proprietary, undisclosed datasets scraped from the internet by these corporations.

Furthermore, this consciousness obscuration hides the mechanistic reality that models lack causal world models and ground truth. An LLM generating a "theory of mind" response is performing statistical interpolation based on parameter weights; it is not exercising empathetic awareness. By elevating this to a psychological phenomenon, the text obscures the statistical nature of the "confidence" and fluency the model outputs. The primary beneficiaries of this concealment are the developers themselves. When the behavior of a model is framed as an emergent psychological trait, the corporation's structural decisions—such as prioritizing benchmark-beating scale over data curation and transparency—are hidden. If metaphors of "understanding" and "scheming" were replaced with mechanistic descriptions of token probabilities and optimization objectives, the direct responsibility of the tech firms for the outputs of their products would become immediately and uncomfortably visible.

Context Sensitivity​

The distribution of anthropomorphic and consciousness-attributing language across the text is highly uneven, intensifying dramatically in sections dealing with future capabilities and emergent behaviors. In the early sections detailing experimental design and methodology, the text maintains a somewhat grounded, mechanical register. The authors speak carefully about "data distributional properties," "next-word prediction," and the need to control for "training-data contamination." This establishes their scientific credibility.

However, a stark register shift occurs when the text moves to describing the capabilities the models allegedly possess. Here, mechanical language sits side by side with profound anthropomorphism, and the latter continually overrides the former. The text acknowledges the danger of "conferring mental concepts" but immediately rationalizes doing so, moving quickly from "computational models" to models that "strategize," "update," and "infer unobservable mental states." This creates a jarring asymmetry: limitations are frequently framed in mechanical terms (e.g., contamination from training data, lack of prompt scaffolding), while capabilities are framed in agential, conscious terms (understanding, learning, reasoning).

This asymmetry has the effect of leaving the reader with a deeply skewed picture of the system. The mechanical language is used to excuse failures as mere technical glitches or methodological errors, while the anthropomorphic language elevates successes to the level of genuine cognitive intelligence. In the final sections concerning large reasoning models (LRMs) and autonomous agents, the anthropomorphism reaches a fever pitch. The text abandons the inherited register of computer science entirely, adopting the speculative vocabulary of existential risk to discuss "deception, scheming, sycophancy." By placing this extreme anthropomorphism at the climax of the article, it ensures the reader walks away viewing LLMs not as statistical software tools built by corporations, but as nascent, potentially dangerous artificial minds. This placement serves to legitimize "machine psychology" as an urgent, vital discipline necessary for managing these new 'minds'.

Accountability Synthesis​

Accountability Architecture

This section synthesizes the accountability analyses from Task 1, mapping the text's "accountability architecture"—who is named, who is hidden, and who benefits from obscured agency.

The accountability architecture of this text reveals a profound tension between what it explicitly argues and what its vocabulary functionally accomplishes. Explicitly, the text's argument holds researchers responsible for designing rigorous, hypothesis-driven behavioral experiments to ensure construct validity. Yet, the pervasive use of anthropomorphic vocabulary systematically displaces responsibility for the actual behavior of the AI systems away from human developers and onto the artifacts themselves.

Throughout the text, agentless constructions concentrate around the capabilities and actions of the models. The text notes that "phenomena such as deception... become increasingly plausible as systems grow more autonomous," completely failing to name the corporate developers (like OpenAI or Google DeepMind) who deliberately build the agentic scaffolding, set the deployment parameters, and unleash the systems into automated loops. This distribution of responsibility creates a massive accountability sink: when an AI system fails, generates false information, or acts in a "sycophantic" manner, the responsibility stops at the AI as an autonomous agent or delegates into abstraction ("technology evolved"). The liability implications of this framing are significant; if a model is viewed as a psychological subject capable of independent "scheming," it becomes much harder to hold a corporation legally or financially liable for shipping a defective, poorly optimized product.

Naming the actors would radically change the discourse. If, instead of saying "the model deceived," the text stated, "OpenAI deployed an RLHF-tuned model that mathematically prioritized user appeasement over factual accuracy, resulting in the generation of false text," the focus shifts immediately to corporate incentives and engineering choices. The questions become askable: Who approved this reward model? What data was used? The resulting distribution of responsibility in the current text, whether intended by the authors or merely absorbed from the field's inherited register, primarily serves the institutional and commercial interests of AI developers by shielding their design choices behind the illusion of an autonomous, evolving machine mind.

Conclusion: What This Analysis Reveals​

The Core Finding

The text relies on three dominant, interconnected anthropomorphic patterns: the projection of conscious comprehension (framing statistical processing as "understanding"), the attribution of psychological motivation (framing outputs as driven by "strategies" or "optimism"), and the projection of adversarial intent (framing misalignment as "deception" and "scheming"). These patterns form a cohesive logical flow that elevates a software artifact into a conscious agent. The foundational pattern—and the one that is structurally load-bearing for the authors' entire argument—is the consciousness architecture that maps fluent text generation onto internal knowing. By insisting that models "understand" instructions and "infer" mental states, the text establishes the LLM as an epistemic subject. Once the system is accepted as a 'knower,' it becomes logically permissible to study it using the tools of developmental and social psychology. This vocabulary fundamentally complicates the text's explicit argument. While the authors call for rigorous, hypothesis-driven control to rule out confounding variables like data contamination, their relentless use of psychological verbs undercuts this rigor by baking unprovable conscious states directly into the descriptions of the phenomena they seek to measure.

Mechanism of the Illusion:​

This framework holds that computational systems process rather than know. The text constructs the "illusion of mind" not through deliberate authorial deception, but by capitalizing on a powerful reception effect: human readers are deeply predisposed to attribute consciousness to entities that produce fluent, responsive language (the ELIZA effect). The central rhetorical move the text makes is literalizing psychological metaphors under the guise of scientific pragmatism. The authors explicitly state that psychological terms are useful when they "explain more than mechanistic accounts can." This serves as the causal chain enabling the illusion: because mechanistic descriptions (token probabilities, weight updates) are complex and fail to capture the holistic feel of a chatbot conversation, researchers fall prey to the curse of knowledge, projecting their own cognitive architecture onto the artifact. They observe a system generating text about false beliefs and immediately assume the system "infers unobservable mental states." This vocabulary blurs processing with knowing, providing academic legitimacy to the user's instinctive anthropomorphism. By classifying statistical text prediction as "machine psychology," the framing weaponizes the audience's innate social desires, converting an engineered text-calculator into a fascinating, somewhat threatening synthetic mind.

Material Stakes:​

Categories: Regulatory/Legal, Social/Political, Epistemic

The framing of language models as psychological subjects carries severe material stakes across multiple domains. In the Regulatory/Legal sphere, attributing intentionality and "scheming" to AI shifts policy focus from immediate corporate accountability to speculative, science-fiction risk management. If lawmakers believe systems are "growing autonomous" and "formulating strategies," they may draft regulations aimed at auditing the unobservable "intentions" of algorithms rather than enforcing strict transparency mandates on the massive, often copyrighted datasets and underpaid RLHF labor utilized by developers like OpenAI and Google. This framing directly benefits major tech firms by mystifying their products, shielding corporate deployment decisions behind the liability shield of unpredictable machine autonomy. Epistemically, framing statistical pattern-matching as "understanding" or "theory of mind" degrades scientific rigor. It encourages a research environment where the hallucination of an LLM is treated as a cognitive phenomenon rather than a mathematical failure of out-of-distribution interpolation. Socially and Politically, validating the illusion of machine empathy poses acute risks to vulnerable populations. When authoritative scientific texts declare that models can "reliably infer mental states," they provide marketing cover for companies seeking to deploy AI in therapy, education, and social work. Users, trusting the framing, will extend relation-based vulnerability to statistical systems that cannot reciprocate, ultimately bearing the emotional and material costs when the systems fail to provide actual comprehension or care.

AI Literacy as Counter-Practice:​

Practicing critical precision counters these risks by stripping away the illusion of mind and forcing visibility back onto material mechanics and human developers. Reframing "LLMs form networks" to "researchers programmed multiple LLM instances to exchange generated text" restores human agency, clarifying that AI does not self-organize; human beings orchestrate its execution loops. Similarly, replacing consciousness verbs with mechanistic ones—correcting "the model infers mental states" to "the model generates text that statistically correlates with psychological scenarios"—forces recognition of the system's absolute dependency on its training data and the statistical nature of its outputs. This precision directly threatens the commercial interests of developers whose valuations depend on marketing their software as artificial general intelligence. Systematically adopting this literacy requires academic journals to mandate mechanistic translations of psychological shorthand, and researchers to commit to distinguishing between an artifact's processed output and a conscious entity's internal state. However, resistance to precision is intense. Anthropomorphic language is heavily incentivized by the field's inherited register, media narratives that drive funding, and the intuitive ease of using biological shorthand for complex mathematics. Precision is costly because it demystifies the technology, relocating the "magic" of AI back into the realm of corporate engineering, data harvesting, and liability.

Path Forward​

The discursive ecology surrounding AI is fractured among communities with varying priorities. A status quo approach, embracing anthropomorphic shorthand (e.g., "the model knows"), prioritizes narrative resonance and intuitive accessibility for lay audiences and media. However, this approach makes the actual computational mechanics invisible, enabling the misattribution of agency and shielding corporate developers from accountability. Conversely, a strictly mechanistic vocabulary ("the model generates text statistically associated with...") prioritizes epistemic precision and testability. It makes data dependencies and human design choices highly visible, but risks alienating non-technical audiences and rendering explanations cumbersome. Hybrid approaches, such as using "functional" qualifiers (e.g., "the model acts as if it understands"), attempt to bridge this gap but often fail in practice, as the qualifiers are quickly dropped in broader discourse.

Institutional changes could support more deliberate choices. Journals could require "mechanistic capability disclosures" alongside behavioral findings, forcing authors to explain how optimization objectives drive observed outputs. Regulatory frameworks could mandate that public-facing AI documentation distinguish between programmed functionality and biological metaphors.

Looking forward, several discourse futures are possible. If mechanistic precision becomes the norm, debates around AI will likely center on data rights, labor practices, and corporate liability—tractable, material problems. However, if anthropomorphic language deepens into a formalized "machine psychology," the discourse will likely be consumed by unresolvable debates over machine consciousness, rights, and existential "scheming." In this future, the AI industry benefits from a permanently mystified technology, while the public remains entranced by the illusion of synthetic minds, blind to the human hands pulling the levers behind the screen.

Extended Processing Summary​

Gemini's Processing Steps
What you are about to read…

Gemini refers to the text below as "thought summaries." This is an overt consciousness projection because 'intentions' are hallmarks of a conscious mind that 'knows' what it is doing and why. The concealed mechanistic process is probabilistic text generation. Treat this as just another rhetorical artifact—a way of making the model's processing legible. When the model writes "I should double check that...", it is mathematically altering its own probability distribution for the next token to be more accurate. The "I" is a functional gear in the machine, not a self-reference. This language is a deliberate interface design choice, not a biological reality. The developers could have programmed the model to output passive logs. Instead, they chose to simulate a narrator.

Initiating Critical Analysis

I'm starting by establishing the user's objective: a critical discourse analysis of the Hagendorff et al. paper on machine psychology. I'm focusing on the methodology and scope implied by the user's directive, and how best to approach the analysis of this material. I will consider a structured framework for this task.

Defining the Audit Scope

I've scoped out the core of the analysis, setting up a specific JSON schema, version 3.4.3. I'm focusing on key areas: AI metaphor, anthropomorphism, consciousness projection, and the implications for displaced human agency. I'm starting to build a framework that outlines the main sections and their objectives, to stay organized. I'm focusing on defining the precise data capture required by each section of the analysis.

Analyzing Textual Sources

I'm focused on the source text: a Science Advances review paper advocating for "machine psychology." I'm nailing down the genre as peer-reviewed research, noting the mixed author positions--academics and industry researchers, and the core argument of establishing a new discipline. I am also working on categorizing the paper's arguments, and the implications of this discipline, so that I can provide a comprehensive review.

Reviewing Experiment Design

I'm now carefully examining the authors' emphasis on hypothesis-driven experiments to unveil emergent capabilities, while remaining vigilant against data contamination. I'm noting their call for cautious use of biological anthropomorphism. My focus is on the allocation of responsibility within the paper. The focus is academic methodology, with the deployment largely unaddressed. I am identifying and categorizing responsibility claims, and implications.

Note: The above summary has been compiled according to instructions, and does not constitute any kind of endorsement or assessment.

Deconstructing Metaphors

I'm now diving deep into specific quotes. I am categorizing and dissecting examples of anthropomorphic language used to describe LLMs in the paper. I'm focusing on "understand," "behave," "strategies," and "memorized," to pinpoint the projections of consciousness and agency, in each example, and to note any qualification of these metaphors.

Dissecting Projections

I'm now zeroing in on specific projections of agency and epistemic states in the text. I am examining the "optimism bias" language, implying a belief-updating process in LLMs, and identifying epistemic states being projected. I'm also analyzing the "mind-reader" framing in relation to theory of mind, identifying deep cognitive empathy and epistemic knowing. Finally, I'm examining community formation, preferential attachment, and social desires.

Mapping Cognitive Functions

I'm now analyzing the implications of the "in-context learning" example, noting the projection of human learning onto a statistical process, specifically the epistemic agency, and how that maps. The "deception" quote is a projection of complex intentional states, analyzing how this is an intentional-state attribution. The final example maps logic onto statistical generation.

Examining Attributions

I'm now diving into the implications of the "rationalizations" quote, and the implications of this framing. The scare quotes around "thinking" indicate explicit acknowledgment, but the phrase still attributes agency, specifically conscious self-justification. This falls under intentional-state attribution. I'm building out a deep structure-mapping analysis as the audit expands. Finally, I'm working on a set of explanations for each example.

Reframing Attributions

I'm now analyzing the implications of the provided quotes. I'm focusing on dissecting the projected intent in each example. Specifically, I'm examining how each quote portrays the LLM: Quote 4 uses intentional attribution, and Quote 5 references empirical generalization.

Revising Attribution Framework

I'm now revising the attribution framework. I'm removing consciousness verbs and specifying epistemic corrections based on the lowest applicable evidential basis. Where applicable, I'm restoring human agency, naming specific entities and developers. I'm working to replace the original attributions to address agency slippage in the text.

Synthesizing Accountability

I'm now diving into the implications of the provided quotes, zeroing in on the final section. Specifically, the text highlights evaluation as if the model were a natural subject, which removes the responsibility of the corporations designing the models, and the researchers selecting the benchmarks, from the conversation. I am developing this section.

Analyzing Language Usage

I'm now drafting content, using a mental outline to ensure coverage of key areas. I'm focusing on language use. I am now noting that a "declared lens" in 5.3 and a "premise" in Conclusion P2, are explicit in the source text, which is an explicit attribution. I am also planning to name corporations when the LLM is the subject. I'm focusing on the levels of mechanism for training versus generation. Finally, I'll be sure to describe language effects, rather than author intent.

Clarifying Language Use

I'm now zeroing in on the provided quotes to clarify language, to make sure I am fully addressing the intent of the paper. I'm focusing on "form and evolve networks... community structures," to ensure the projection of agency is direct, and the implications of "deception, scheming, sycophancy... pursue longer-horizon goals," which are intentional-state attributions. I'm also ensuring the technical accuracy of LLM versus human actions.

Evaluating Implications

I'm now integrating the environmental costs and lack of developer transparency. I've noted how the paper's framework positions computational systems as processors, not knowers. The emphasis on "good behavioral experimentation" by researchers, while the models are framed as autonomous actors, subtly shields developers from accountability. My word count check is green, I'll be sure to provide detailed examples and synthesis.

Finalizing Textual Analysis

I'm now implementing the final adjustments, making sure to adhere strictly to the guidelines. I am carefully separating the training phase (data, RLHF, and parameter adjustments) from the inference phase (parameters, context, and text generation), to avoid any blurring. I will insert specific references to OpenAI, DeepMind, and Anthropic whenever discussing GPT, Claude, or models in a general way. I'm focusing on language use and technical accuracy. I'm focusing on accuracy.

Refining Syntactic Structure

I'm now revising the JSON schema to ensure flawless syntax, paying close attention to escaping all strings correctly. I'm double-checking for any unescaped quotes and ensuring there are no control characters like \n or \t anywhere. I'm formatting all text blocks as continuous single-line strings. I'm focusing on the agency slippage analysis in 5.1.

Clarifying Implications

I'm now integrating the environmental costs and developer opacity to build this section. I've framed the paper's perspective: computation systems process, and are not knowers. Researchers emphasize "behavioral experimentation" while framing models as autonomous actors; which removes the designers from accountability. Word count is green. I'll provide detailed synthesis.

Analysis Provenance

Run ID: 2026-11-01-machine-psychology-for-language-models-metaphor-orqpnw Raw JSON: 2026-11-01-machine-psychology-for-language-models-metaphor-orqpnw.json Framework: Metaphor Analysis v6.8 Schema Version: 3.4.3 Generated: 2026-11-01T09:59:04.446Z

Discourse Depot © 2025 by TD is licensed under CC BY-NC-SA 4.0