Skip to main content

Beyond Sally-Anne: Evaluating Theory of Mind in Large Language Models using Epistemic Schelling Points

About

This document presents a Critical Discourse Analysis focused on AI literacy, specifically targeting the role of metaphor and anthropomorphism in shaping public and professional understanding of generative AI. The analysis is guided by a prompt that draws from cognitive linguistics (metaphor structure-mapping), the philosophy of social science (Robert Brown's typology of explanation), and accountability analysis.

All findings and summaries below were generated from detailed system instructions provided to a large language model and should be read critically as interpretive outputs—not guarantees of factual accuracy or authorial intent.

Source Reading​

Genre: Peer-reviewed research · Author position: Developer of the system discussed

Argument: This paper argues that existing text-based evaluations of Theory of Mind (ToM) in large language models are flawed because models can game them using linguistic cues rather than demonstrating robust social reasoning. The authors introduce the Epistemic (A)symmetry Schelling Task (EAST), a two-player dialogue game that tests models' ability to coordinate under varying states of epistemic transparency. They argue that models systematically fail in asymmetric and zero-knowledge conditions, concluding that epistemic tracking and social reasoning remain a critical bottleneck in frontier AI models.

Responsibility claims: The text holds the AI research community and model developers responsible for recognizing the limitations of current static ToM benchmarks and for developing new training and evaluation paradigms. However, its vocabulary heavily attributes responsibility for failure to the models themselves, explicitly asserting that models commit "cognitive fallacies," exhibit "egocentric projection," and fail to "adequately track" epistemic states. The text frames the models as autonomous agents capable of possessing "robust self-other epistemic boundaries" and "coherent mind modelling," leaving the role of developers in generating statistical patterns largely unstated when describing system failures.

Task 1: Metaphor and Anthropomorphism Audit​

About this task

For each of the major metaphorical patterns identified, this audit examines the specific language used, the frame through which the AI is being conceptualized, what human qualities are being projected onto the system, whether the metaphor is explicitly acknowledged or presented as direct description, and—most critically—what implications this framing has for trust, understanding, and policy perception.

1. Model as Conscious Tracker​

Quote: "To perform this task, models need to be able to accurately track the other player's identity and epistemic states, and to infer which word represents the most salient focal point from their partner's uniquely constrained perspective."

  • Frame: Model as an empathetic conscious agent
  • Projection: This framing projects human cognitive awareness and perspective-taking onto computational text generation. It maps the subjective experience of 'tracking' someone else's mind and 'inferring' their perspective onto a statistical process of token prediction. The highest level of agency attributed here is an epistemic and conscious state, as it implies the model actively holds a mental representation of another entity's knowledge. However, the text does not supply evidence of conscious tracking; rather, it provides evidence of models generating text based on prompts containing variable constraints. At the generation level, a model like Gemini or Llama processes the input tokens describing a persona and a scenario, and its self-attention mechanisms weigh these contextual tokens to produce highly probable output sequences. The mechanism, which is generic to systems of this type, involves matrix multiplications calculating semantic relevance, not an inner cognitive realization of a partner's constrained perspective.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The quote uses 'track' and 'infer' directly without hedging. I considered 'Hedged/Qualified' because the paper studies 'functional ToM', but this specific capability claim stands without qualification as a literal requirement.)
  • Implications: By framing the system as an entity capable of accurately tracking human identity and inferring perspectives, this language severely inflates the perceived sophistication of the AI. It invites users and policymakers to trust the system as a social participant capable of empathy and strategic awareness. This consciousness projection creates specific risks: users may rely on these systems for sensitive social mediations, assuming the AI 'knows' what the user feels or needs. It also creates liability ambiguity by treating the model as an independent social reasoner capable of making its own inferences, thereby obscuring the statistical brittleness that actually governs its text generation.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: Google DeepMind and other developers (Meta, Mistral, Alibaba) designed the architectures, selected the training data, and aligned these models. The agentless construction 'models need to be able to accurately track' obscures the fact that engineers need to discover optimization objectives and training regimes that result in outputs mimicking perspective-taking. This wording has the effect of treating the model as the locus of capability and failure, serving the interests of developers by distancing them from the immediate accountability of the system's statistical shortcomings. A 'Partial' classification was considered, but the sentence places the onus of action entirely on the 'models' without mentioning the developers who build them.
Show more...

2. Model as Deluded Thinker​

Quote: "models often choose the self-related word because they hallucinate or ignore relevant epistemic information about the other player"

  • Frame: Model as a flawed human reasoner
  • Projection: This metaphor projects human psychological states—specifically delusion ('hallucinate') and willful negligence ('ignore')—onto the mechanistic failure of a model to generate text aligned with specific prompt constraints. It operates at the level of epistemic and conscious states, suggesting the model has an internal awareness that it actively disregards or replaces with a hallucination. The text supplies evidence of incorrect output, not of a conscious choice to ignore information. At generation time, the model computes token probabilities based on its pre-trained parameters and the provided context window; if the attention mechanism fails to sufficiently weight the tokens representing the 'other player's' constraints, it produces the 'self-related word' because that word has a higher baseline probability given the primary persona tokens. This is a generic mechanism of autoregressive language models, completely devoid of the cognitive states of hallucinating or ignoring.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The verbs 'hallucinate' and 'ignore' are used as literal descriptions of the model's behavior. I considered 'Explicitly Acknowledged' since 'hallucinate' is common jargon, but the text offers no scare quotes or meta-commentary recognizing it as a metaphor.)
  • Implications: Describing statistical output errors as 'hallucinating' or 'ignoring' frames the AI as an intentional agent making cognitive mistakes rather than an artifact operating exactly as mathematically designed. This consciousness framing preserves the illusion that the model generally 'understands' and only occasionally suffers from human-like lapses in attention. Consequently, it promotes an unwarranted trust in the system's baseline reliability and shifts focus away from systemic issues in training data or architecture, suggesting that the model simply needs to 'pay better attention' rather than recognizing it lacks any capacity for attention in the human sense.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The developers (e.g., Google DeepMind) who deployed systems that fail to reliably weight contextual constraints are erased here. The text blames the model for 'ignoring' information, placing responsibility on the artifact. If the human actors were named (e.g., 'Google's model generated words statistically correlated with the first persona rather than the second'), it would make visible the developers' failure to achieve the desired mathematical alignment. This agentless phrasing shields the corporations from direct accountability for their product's limitations by anthropomorphizing the failure as an individual cognitive error. 'Named' was rejected as no human actors are present in the formulation.

3. Model as Ignorance Comprehender​

Quote: "modelling an epistemic void - understanding what another agent does not know - remains a critical bottleneck in developing robust reasoning."

  • Frame: Model as a conscious knower of absences
  • Projection: This language projects advanced human metacognition—the subjective awareness of another being's lack of knowledge—onto the AI system. It asserts the agency level of an epistemic/conscious state by claiming the model must achieve 'understanding' of an 'epistemic void'. The paper provides no evidence of understanding; it measures whether the system outputs a specific target word when prompted with text stating another entity lacks information. Mechanistically, as generic to systems of this type, the model is generating tokens conditioned on a prompt string like 'They DO NOT know who you are.' The model does not model a 'void' or 'know' anything; its parameters guide the statistical generation of the next word based on the presence of those negative constraint tokens in the context window. The text equates the generation of a correct word in a complex syntactic context with the conscious state of understanding ignorance.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The phrase 'understanding what another agent does not know' is presented as a literal capability the model must acquire. I considered 'Hedged/Qualified', but there are no qualifiers softening the verb 'understanding' in this context.)
  • Implications: Claiming a model can or must 'understand what another agent does not know' drastically escalates the perceived cognitive power of the system. It encourages audiences to believe that AI can grasp nuance, secrets, and human vulnerabilities. This has severe implications for how users interact with AI in sensitive contexts, such as therapy or negotiation, where users might assume the AI possesses a genuine theory of mind and can protect or manipulate information based on true awareness. It obscures the reality that the model is merely processing semantic proximity, making it dangerously unpredictable when deployed in novel social situations.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text states that 'modelling an epistemic void... remains a critical bottleneck', obscuring the human researchers and engineers who are struggling to mathematically optimize these systems to handle complex prompt negations. The text displaces the engineering bottleneck onto the model's failure to 'understand'. By naming the actors (e.g., 'Researchers at Google face a bottleneck in training models to generate text consistent with negated context'), the structural reality of the engineering challenge becomes visible, highlighting the limits of current autoregressive architectures rather than the cognitive failings of an artificial mind. 'Partial' was ruled out because the bottleneck is framed as an abstract property of the model's development.

4. Model as Egocentric Strategist​

Quote: "models fail to suppress private semantic biases to identify a shared focal point."

  • Frame: Model as a selfish thinker
  • Projection: This framing projects human psychological mechanisms—specifically the conscious effort to 'suppress' selfish biases to find common ground—onto the model's behavior. The agency level attributed is intentional-state attribution and goal-directed behavior, as it implies the model possesses a 'private' bias and actively fails to restrain it. Mechanistically, based on generic transformer architecture, a model does not possess 'private' biases or 'suppress' anything. During generation, tokens related to the model's assigned persona strongly activate related tokens in the model's vocabulary due to training data correlations. 'Failure to suppress' simply means the contextual tokens directing the model to select a 'shared' word did not sufficiently alter the output probabilities to overcome the baseline associative strength of the persona tokens. There is no inner conflict or private self involved.
  • Agency Level: Intentional-state attribution
  • Acknowledgment: Direct (Unacknowledged) (The text uses 'suppress private semantic biases' without explicitly acknowledging the metaphorical nature of 'private' or 'suppress'. I considered 'Ambiguous' but the psychological framing is consistent and clear throughout the passage.)
  • Implications: Describing statistical weights as 'private semantic biases' that models fail to 'suppress' moralizes mathematical operations. It suggests that AI models have internal desires or selfish tendencies that they must learn to control, mirroring human socialization. This framing affects policy and trust by suggesting that AI systems can be 'taught' to be less selfish or more socially cooperative through behavioral feedback, obscuring the fact that these are statistical systems that require architectural or dataset interventions to change output distributions. It builds a narrative of AI as a trainable pet or child rather than a designed tool.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The developers (Google, Meta, etc.) who trained the models on data that heavily weights certain semantic associations are absent. The phrase 'models fail to suppress' makes the artifact the active agent of failure. If the sentence read, 'The models built by Google generated outputs dominated by primary semantic associations rather than the secondary constraints,' responsibility for the output behavior would remain with the system's design and the corporation that built it. The current framing serves to isolate the failure within the model's 'psychology', protecting the developers from scrutiny over their optimization choices. 'Named' is incorrect as developers are missing.

5. Model as Accommodating Partner​

Quote: "Player 1 must minimally take into account the ignorance of Player 2 with respect to their profession, recognising that Player 2 has no epistemic basis for selecting wp1... Player 1 must suppress their own persona-driven associations and accommodate Player 2"

  • Frame: Model as an empathetic collaborator
  • Projection: This metaphor maps a complex array of human social skills—taking into account ignorance, recognizing epistemic bases, suppressing self-interest, and accommodating a partner—onto the system. The agency level reaches epistemic/conscious state. The text offers no evidence of the model experiencing recognition or accommodation. Mechanistically, at the level of generation documented generically for LLMs, the model generates text in a single inference pass (or via Chain of Thought tokens) based on prompt instructions. It does not 'recognize' another entity's basis for selection; its attention heads process the tokens describing Player 2's constraints and calculate output probabilities that favor the shared word. 'Accommodating' is a human interpretation of a statistical output sequence that successfully aligns with the complex logical constraints formulated in the prompt.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The verbs 'take into account', 'recognising', 'suppress', and 'accommodate' are presented as direct, literal actions the model must perform. I considered 'Hedged/Qualified' because this describes an ideal rational strategy, but it is applied directly to what the model 'must' do as an agent.)
  • Implications: By layering multiple intentional and epistemic verbs ('recognising', 'accommodate'), the text deeply embeds the illusion that the model is a social being. This significantly alters user understanding, leading people to believe that language models possess human-like collaborative intentions. When models are perceived as accommodating partners, users may overly trust them in high-stakes cooperative scenarios (e.g., automated negotiation or dispute resolution), mistakenly believing the AI will safeguard mutual interests out of an internal sense of accommodation, rather than merely predicting the most likely next token.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text discusses what 'Player 1' (the LLM) must do, entirely omitting the human developers who dictate the model's capabilities through instruction tuning and RLHF. Google DeepMind and other developers are responsible for whether their models can mathematically satisfy multi-variable prompt constraints. The phrasing 'Player 1 must... accommodate' displaces the engineering requirement (developers must train models to synthesize negative constraints) onto the model's hypothetical social grace. This wording has the effect of treating the AI as a self-determining entity, obscuring the corporate design decisions governing its behavior. 'Partial' was rejected because the text speaks exclusively of the 'Player'.

6. Model as Rule Violator​

Quote: "models either engage in semantic associations which are not aligned to the generative structure of the game, hallucinate goals and constraints... that are not supported by the prompt, or both."

  • Frame: Model as a disobedient participant
  • Projection: This framing projects human behaviors of rule-breaking, misinterpreting, and hallucinating onto the model. The agency level is intentional-state attribution, as it implies the model actively misinterprets or invents rules. Mechanistically, the source documents this at the generation level: the model produces words that do not match the expected 'normative' outputs. Generic to systems of this type, when a model produces text 'not supported by the prompt', it is because the statistical pull of certain token sequences in its training data (e.g., associating 'tailor' strongly with 'fabric') outweighs the conditioning provided by the prompt's instructions. The model does not 'hallucinate goals'; it generates the next most probable token sequence, which human evaluators then interpret as a goal not specified in the prompt.
  • Agency Level: Intentional-state attribution
  • Acknowledgment: Direct (Unacknowledged) (The text states the models 'hallucinate goals and constraints' as a literal description of the failure mode. I considered 'Explicitly Acknowledged' for 'hallucinate', but it is used here without qualification to describe the model inventing rules.)
  • Implications: Describing statistical deviations as 'hallucinating goals' endows the system with an illusory autonomy and capacity for spontaneous intention. This framing creates the perception that AI systems possess independent wills that can override human instructions. While this might heighten fear of 'rogue' AI, it paradoxically increases trust in the system's general cognitive capacity, as only a thinking entity can 'invent' a goal. It distracts from the mundane reality of poor out-of-distribution generalization and miscalibrated reward models.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The developers (Google, Meta, Mistral) whose alignment techniques failed to reliably guide the model's text generation are obscured. By stating that 'models... hallucinate goals', the text makes the AI the author of the error. If the human actors were named (e.g., 'The models developed by Google generated text consistent with unprompted associations rather than the specified rules'), it would redirect accountability to the failure of the alignment process. The current phrasing serves to diffuse responsibility, treating the model's statistical drift as an independent, spontaneous act of misbehavior. 'Named' is incorrect because the developers are entirely absent from the description.

7. Model as Epistemic Modeler​

Quote: "explicit mutual modelling of epistemic states elevates what can otherwise be achieved through simple associative semantic overlap into a robust, intentional coordination strategy."

  • Frame: Model as a strategic mind-reader
  • Projection: This metaphor projects the human cognitive capacity for mutual mind-reading ('modelling of epistemic states') and purposeful planning ('intentional coordination strategy') onto the AI. The agency level is epistemic/conscious state. The text offers no evidence that the model models anything internally. Mechanistically, based on reasonable inference of LLM generation, prompting the model to explicitly generate reasoning tokens ('Chain of Thought') allows it to condition its final token choice on a longer sequence of intermediate tokens describing the game's rules. The model mathematically predicts these intermediate tokens; it does not engage in 'intentional coordination' or 'mutual modelling'. It remains a single system predicting text based on input.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The text presents 'mutual modelling of epistemic states' and 'intentional coordination strategy' as literal capabilities the models can achieve. I considered 'Hedged/Qualified' because the paper contrasts this with 'shallow' overlap, but the 'intentional' claim itself is unqualified.)
  • Implications: Ascribing 'intentional coordination' to a language model fundamentally mischaracterizes its nature, encouraging users to treat it as a rational agent with goals rather than an artifact generating text. When audiences believe AI systems form 'intentional strategies', they may overestimate the system's reliability in novel situations, assuming the AI will 'figure out' a way to cooperate. It masks the reality that the system will fail unpredictably if the required statistical patterns are not present in its training data.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The text praises the 'robust, intentional coordination strategy' achieved, obscuring the engineers who designed the prompt scaffolding and the base models that enable this specific chain of generated text. Naming the actors (e.g., 'When users prompt Google's model to generate intermediate reasoning steps, the system outputs more accurate target words') would appropriately attribute the 'strategy' to the prompt design and the model's designers. The agentless construction grants the model autonomy, effectively giving the machine credit for human-designed operational scaffolding. 'Ambiguous' was ruled out because the displacement of the human prompting mechanism is clear.

8. Model as Self-Aware Entity​

Quote: "game-theoretic coordination in this informational vacuum requires agents to accurately separate their private knowledge (their own persona) from the shared epistemic state of mutual ignorance"

  • Frame: Model as a self-aware entity with private knowledge
  • Projection: This metaphor projects self-awareness and the concept of 'private knowledge' onto a computational system. The agency level is epistemic/conscious state. The model is portrayed as an entity that possesses a distinct 'self' (its persona) and must consciously isolate this from a 'shared epistemic state'. Mechanistically, as generic to autoregressive systems, the model has no private knowledge. The 'persona' is simply a sequence of tokens provided in the user prompt. The model processes the entire prompt simultaneously; it does not 'separate' knowledge. It calculates probabilities based on the proximity and attention weights of all tokens in the context window. The requirement is mathematical (weighting tokens indicating 'unknown' heavier than the persona tokens), not cognitive.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The terms 'private knowledge', 'own persona', and 'shared epistemic state' are used directly to describe the model's required cognitive process. I considered 'Ambiguous', but the parenthetical '(their own persona)' firmly links the human concept of private self to the model.)
  • Implications: Describing the information in a prompt as the AI's 'private knowledge' anthropomorphizes the context window into a human mind. This deeply misleads users about how information is processed, suggesting the AI has an inner life and a boundary between itself and the outside world. This can lead to severe misunderstandings regarding data privacy and security, as users might believe the AI can 'keep a secret' or 'separate' data cognitively, rather than recognizing that all input data merely influences mathematical output probabilities.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The designers of the models (Google DeepMind, etc.) and the authors who designed the prompt are obscured. The text claims the 'agents' must 'accurately separate' knowledge, placing the cognitive burden entirely on the AI. If human actors were named (e.g., 'Google's model must generate text that weighs the prompt's instruction about ignorance more heavily than the persona text'), it would reveal the mechanistic reality of the task and the developer's responsibility for the model's attention mechanisms. The current framing serves to mask the limits of current AI architectures behind a veil of supposed cognitive complexity. 'Partial' is incorrect as no human entities are referenced.

Task 2: Source-Target Mapping​

About this task

For each key metaphor identified in Task 1, this section provides a detailed structure-mapping analysis. The goal is to examine how the relational structure of a familiar "source domain" (the concrete concept we understand) is projected onto a less familiar "target domain" (the AI system). By restating each quote and analyzing the mapping carefully, we can see precisely what assumptions the metaphor invites and what it conceals.

Mapping 1: conscious mind / empathetic observer → model text generation based on complex prompt constraints​

Quote: "models need to be able to accurately track the other player's identity and epistemic states"

  • Source Domain: conscious mind / empathetic observer
  • Target Domain: model text generation based on complex prompt constraints
  • Mapping: The relational structure of a human consciously keeping track of another person's thoughts and beliefs is projected onto the language model's ability to maintain context across a prompt. This mapping invites the assumption that the model possesses a continuous, aware inner state that actively monitors the 'epistemic state' of a simulated partner, much like a human player in a social game.
  • What Is Concealed: This mapping conceals the stateless, mathematical nature of the model's generation. It obscures the fact that the model is performing single-pass token prediction based on self-attention mechanisms over a fixed context window. By attributing conscious knowing to mechanistic processing, the text hides the reality that the model is blindly matching statistical patterns in the training data related to the prompt's words, rather than comprehending an 'identity'. The specific attention weights and training data of proprietary models like Gemini remain completely opaque.
Show more...

Mapping 2: flawed human reasoner / willful agent → autoregressive token prediction failure​

Quote: "models often choose the self-related word because they hallucinate or ignore relevant epistemic information"

  • Source Domain: flawed human reasoner / willful agent
  • Target Domain: autoregressive token prediction failure
  • Mapping: The source domain of a person making a cognitive error (hallucinating) or intentionally disregarding facts (ignoring) is mapped onto the target domain of a model outputting an incorrect token. This invites the assumption that the model's failure is akin to a human lapse in attention or a psychological delusion, possessing an underlying agency that simply made a mistake.
  • What Is Concealed: The mapping conceals the statistical mechanics of language modeling. It hides the fact that the 'self-related word' was chosen simply because its probability was mathematically higher given the prompt's phrasing and the model's pre-trained weights. Attributing conscious 'ignoring' to a mechanistic process obscures the developers' failure to adequately instruction-tune the model for complex logical negations, shielding the proprietary training regimes from scrutiny.

Mapping 3: knower / conscious mind → generating text conditioned on negated prompt constraints​

Quote: "understanding what another agent does not know"

  • Source Domain: knower / conscious mind
  • Target Domain: generating text conditioned on negated prompt constraints
  • Mapping: The human capacity for metacognition—the subjective grasp of absence of knowledge in another—is mapped onto the system's processing of tokens. This mapping invites readers to assume the AI possesses a justified true belief about the simulated game environment and can subjectively evaluate truth claims regarding a partner's awareness.
  • What Is Concealed: This mapping conceals the model's complete lack of subjective awareness. It hides the mechanistic reality that the system is simply generating text statistically correlated with phrases like 'They DO NOT know'. By framing this as 'understanding', it obscures the dependency on the vast, opaque training data that dictates these correlations. The true process cannot be fully known for proprietary models, yet the text confidently asserts a cognitive capability.

Mapping 4: self-regulating human / socialized individual → attention mechanisms prioritizing strongly correlated tokens​

Quote: "models fail to suppress private semantic biases"

  • Source Domain: self-regulating human / socialized individual
  • Target Domain: attention mechanisms prioritizing strongly correlated tokens
  • Mapping: The human experience of having a selfish desire ('private bias') and actively trying to stop it ('suppress') is projected onto the model's generation process. This invites the assumption that the AI has a 'private' internal life separate from its 'public' output, and that it struggles with internal self-control.
  • What Is Concealed: This mapping conceals the deterministic, associative nature of neural networks. It hides the fact that a model has no 'private' state; it only has parameters shaped by training data. The metaphor obscures the mechanistic reality that 'failure to suppress' simply means the attention given to the prompt's constraints was mathematically insufficient to lower the probability of the highly correlated persona word. It obscures the labor of data annotators who originally shaped these semantic biases.

Mapping 5: empathetic collaborator → generating target word satisfying multiple prompt constraints​

Quote: "Player 1 must... accommodate Player 2"

  • Source Domain: empathetic collaborator
  • Target Domain: generating target word satisfying multiple prompt constraints
  • Mapping: The social act of accommodation, which involves empathy, compromise, and mutual recognition, is mapped onto the model's output generation. This invites the assumption that the AI acts with a cooperative intention and values the success of the interaction, behaving as a moral or social agent.
  • What Is Concealed: This conceals the complete absence of intention or empathy in the system. It hides the mechanistic process where the model simply calculates the highest probability token sequence following the provided prompt. Attributing 'accommodation' to processing obscures the commercial objectives of developers who train these systems to sound helpful and cooperative (via RLHF), masking the intensive human labor required to create this illusion of sociability.

Mapping 6: delusional or disobedient actor → out-of-distribution text generation​

Quote: "hallucinate goals and constraints"

  • Source Domain: delusional or disobedient actor
  • Target Domain: out-of-distribution text generation
  • Mapping: The human act of inventing false realities or rebelling against instructions is mapped onto the model's generation of unexpected text. This invites the assumption that the model has an independent cognitive apparatus that generates goals spontaneously, possessing a mind of its own.
  • What Is Concealed: The metaphor conceals the statistical drift and misalignment of the model's parameters. It hides the fact that the generated 'goals' are merely the next most likely tokens according to the training data distribution, not spontaneous inventions. By anthropomorphizing the error, it obscures the opacity of proprietary models' pre-training data, which is the actual source of these 'hallucinated' associations.

Mapping 7: strategic human planner → Chain of Thought prompt generation​

Quote: "intentional coordination strategy"

  • Source Domain: strategic human planner
  • Target Domain: Chain of Thought prompt generation
  • Mapping: The human process of deliberate, purposeful planning is mapped onto the model's sequential token generation when prompted to 'think step-by-step'. This invites the assumption that the intermediate text the model produces reflects actual cognitive strategizing and intention formation.
  • What Is Concealed: This conceals the mechanistic reality of Chain of Thought prompting. It hides the fact that the intermediate tokens are just generated text, statistically derived from the prompt, which then serve as extended context for the final output. It obscures the fact that there is no 'intention' behind the text, only probabilities. It also hides the specific prompt engineering decisions made by humans that coax this behavior out of the model.

Mapping 8: self-aware knower → attention weighting of context window tokens​

Quote: "accurately separate their private knowledge... from the shared epistemic state"

  • Source Domain: self-aware knower
  • Target Domain: attention weighting of context window tokens
  • Mapping: The human psychological ability to distinguish between what one knows internally and what is known communally is mapped onto the AI's context processing. This invites the assumption that the model possesses an internal boundary, a 'self' that it manages in relation to others.
  • What Is Concealed: This mapping conceals the flat, simultaneous processing of the context window. It hides the mechanistic truth that the model processes the entire prompt as a single input matrix; it does not 'separate' knowledge. It obscures the mathematical reality of attention heads, replacing a technical description of token weighting with an illusory claim about cognitive boundary management. The exact weights and layers involved are hidden within proprietary architectures like Gemini.

Task 3: Explanation Audit (The Rhetorical Framing of "Why" vs. "How")​

About this task

This section audits the text's explanatory strategy, focusing on a critical distinction: the slippage between "how" and "why." Based on Robert Brown's typology of explanation, this analysis identifies whether the text explains AI mechanistically (a functional "how it works") or agentially (an intentional "why it wants something"). The core of this task is to expose how this "illusion of mind" is constructed by the rhetorical framing of the explanation itself, and what impact this has on the audience's perception of AI agency.

Explanation 1​

Quote: "failures seem to be mostly due to cases where Player 1 (the knowing player) fails to adequately account for Player 2's ignorance, and chooses wp1 or ws instead of wp2"

  • Explanation Types:

    • Reason-Based (Primary): Gives agent's rationale, entails intentionality and justification
    • Functional (Secondary): Explains behavior by role in self-regulating system with feedback

  • Analysis (Why vs. How Slippage): This explanation frames the AI highly agentially (why). It uses reason-based logic by attributing the failure to Player 1's cognitive inability to 'account for' Player 2's ignorance, suggesting a failure of justification and intentional planning. While it nominally describes the functional output of a system ('chooses wp1'), the dominant framing focuses on the model's internal rationale and cognitive missteps. This choice emphasizes the model as a social actor participating in a game, while completely obscuring the mechanistic (how) reality: that the model's attention mechanism failed to assign sufficient weight to the prompt tokens describing Player 2's constraints, resulting in a higher probability for the persona-biased word.

  • Consciousness Claims Analysis: The passage attributes conscious states through the verb phrase 'fails to adequately account for [ignorance]'. This operates at the highest level of the agency gradient: epistemic/conscious states, as it implies the model should 'know' about another's lack of knowledge. This reflects the curse of knowledge: the authors, understanding the logical structure of the game, project their own cognitive process of 'accounting for ignorance' onto the system's output. Mechanistically, generic to systems of this type, a deployed model generates text based on pre-trained parameters and the context window. When it produces wp1 instead of wp2, the probabilities shaped by the training data strongly associated the prompt's persona tokens with wp1, and the negative constraints ('Player 2 does not know') failed to sufficiently alter those probabilities during generation. The model processes tokens; it does not account for ignorance.

  • Rhetorical Impact: This framing shapes the audience's perception of the AI as an autonomous, thinking entity that makes cognitive errors. By framing the failure in consciousness terms, it implies that when the model does succeed, it does so through genuine understanding and empathy. This inflates relation-based trust, leading audiences to believe the AI is capable of reciprocal social awareness, while masking the fundamental unreliability of statistical generation.

Show more...

Explanation 2​

Quote: "models often choose the self-related word because they hallucinate or ignore relevant epistemic information about the other player"

  • Explanation Types:

    • Reason-Based (Primary): Gives agent's rationale, entails intentionality and justification
    • Dispositional (Secondary): Attributes tendencies or habits

  • Analysis (Why vs. How Slippage): The explanation frames the AI agentially (why). It provides a reason-based explanation ('because they hallucinate or ignore'), attributing the system's output to active cognitive malfunction or intentional disregard. It also acts dispositionally by suggesting a habit ('often choose... because'). This framing emphasizes the model as an active, willful participant capable of psychological states like hallucination and ignoring. It entirely obscures the mechanistic how: the statistical distribution of the model's parameters, which dictate that 'self-related' tokens are generated because of strong correlational weights established during training, not because of a cognitive decision to ignore context.

  • Consciousness Claims Analysis: The passage heavily attributes conscious states using the verbs 'hallucinate' and 'ignore'. These are consciousness verbs that locate the action at the level of epistemic/conscious state and intentional-state attribution. The authors project human psychological failures onto computational misalignments. Mechanistically, based on reasonable inference for autoregressive models, the model generates text from its parameters and context. It does not 'ignore' information; rather, the attention heads processing the 'relevant epistemic information' tokens in the prompt do not exert enough influence over the final token prediction to override the base probabilities of the 'self-related' word. The model processes tokens mathematically; it has no conscious awareness to direct or withhold.

  • Rhetorical Impact: By diagnosing statistical errors as 'hallucinations' or acts of 'ignoring', the text creates the illusion of an erratic but undeniably conscious mind. This shapes risk perception by suggesting the system is generally intelligent but occasionally stubborn or confused. If audiences believe the AI 'knows' but sometimes 'ignores' facts, they might seek behavioral solutions (like arguing with the prompt) rather than recognizing the structural limitations of the artifact.

Explanation 3​

Quote: "forcing explicit reasoning disrupts accidental or heuristic-driven matches, promoting more principled reasoning strategies"

  • Explanation Types:

    • Intentional (Primary): Refers to goals/purposes, presupposes deliberate design
    • Functional (Secondary): Explains behavior by role in self-regulating system with feedback

  • Analysis (Why vs. How Slippage): This explanation blends mechanical and agential framing, leaning agential. It uses intentional explanation by suggesting the model adopts 'principled reasoning strategies', presupposing purpose and deliberation. It operates functionally by describing an intervention ('forcing explicit reasoning' via prompting) that alters system output. This choice emphasizes the perceived cognitive depth of the system's process ('principled reasoning') while obscuring the actual mechanism: that generating intermediate text (Chain of Thought) alters the context window, shifting the probability distribution for the final token.

  • Consciousness Claims Analysis: The passage attributes intentional states by contrasting 'accidental matches' with 'principled reasoning strategies'. This locates the system's behavior at intentional-state attribution. The curse of knowledge is evident: the authors interpret the coherent intermediate text generated by the model as evidence of an internal 'principled' cognitive strategy. Mechanistically, generic to systems of this type, the prompt instructs the model to generate text step-by-step. The model generates intermediate tokens based on the prompt; these tokens then become part of the context for generating the final word. The model does not possess a 'strategy' or 'reason'; it statistically predicts the next token in the sequence. The 'reasoning' is generated text, not an underlying cognitive process.

  • Rhetorical Impact: This framing builds performance-based and relation-based trust by suggesting the AI can be made to act 'principled' rather than 'accidental'. It reassures the audience that under the right conditions, the system engages in rigorous thought. Believing the AI uses 'principled reasoning' encourages users to trust its outputs in complex domains (like law or medicine) far beyond its actual capacity for justified belief.

Explanation 4​

Quote: "game-theoretic coordination in this informational vacuum requires agents to accurately separate their private knowledge (their own persona) from the shared epistemic state of mutual ignorance"

  • Explanation Types:

    • Intentional (Primary): Refers to goals/purposes, presupposes deliberate design
    • Reason-Based (Secondary): Gives agent's rationale, entails intentionality and justification

  • Analysis (Why vs. How Slippage): This explanation frames the AI highly agentially. It uses intentional and reason-based framing to describe what the 'agents' must consciously do ('accurately separate their private knowledge') to achieve a goal ('game-theoretic coordination'). This choice emphasizes a sophisticated, self-aware cognitive process. It completely obscures the mechanistic reality: that the model must calculate token probabilities based on a prompt containing contradictory informational cues, requiring specific attention weightings to prioritize the 'ignorance' tokens over the 'persona' tokens.

  • Consciousness Claims Analysis: The passage attributes epistemic/conscious states by requiring the system to 'accurately separate' knowledge and possess a 'shared epistemic state'. It uses consciousness concepts ('private knowledge', 'mutual ignorance') rather than mechanistic terms. The authors project their understanding of game theory onto the system. Mechanistically, as generic to autoregressive models, the model generates text in response to the prompt. It has no 'private knowledge'; it has only the tokens in the context window and its pre-trained parameters. The process of 'separating' knowledge is actually the mathematical process of attention heads assigning different weights to different parts of the prompt during generation. It processes token relationships; it does not know or separate concepts cognitively.

  • Rhetorical Impact: This framing dramatically shapes the audience's perception of AI autonomy. By depicting models as entities managing 'private knowledge' and 'shared epistemic states', it suggests the AI is a fully fledged social being. This could shift regulatory and deployment decisions, as stakeholders might treat the AI as capable of understanding confidentiality or complex social nuances, ignoring its statistical brittleness.

Explanation 5​

Quote: "we observe that (see Figure 4) there are many cases in which models correctly track epistemic states of both players, but fail to coordinate nonetheless. This highlights an additional weakness in translating correct epistemic inferences into strategic actions."

  • Explanation Types:

    • Theoretical (Primary): Embeds in deductive framework, may invoke unobservable mechanisms
    • Reason-Based (Secondary): Gives agent's rationale, entails intentionality and justification

  • Analysis (Why vs. How Slippage): This explanation uses theoretical framing by embedding the model's behavior within the cognitive framework of Theory of Mind ('translating correct epistemic inferences into strategic actions'). It invokes unobservable cognitive mechanisms (inference, translation to action). It also relies on reason-based logic, suggesting a failure of justification in the model's internal process. This choice emphasizes a complex cognitive architecture while obscuring the mechanistic reality that the model simply generated intermediate text that matched the expected logic, but then generated a final token that did not logically follow, due to probabilistic inconsistencies.

  • Consciousness Claims Analysis: The passage attributes epistemic/conscious states by claiming models 'correctly track epistemic states' and make 'epistemic inferences'. It contrasts knowing (inferences) with acting (strategic actions). This is a clear projection: the authors interpret the model's generated intermediate text as 'correct inference'. Mechanistically, generic to systems of this type, the model generates intermediate tokens (the 'inference') and then generates the final token (the 'action'). The weakness is not a cognitive failure to translate thought into action; it is a statistical disjoint where the probabilities governing the final token prediction did not align tightly enough with the newly generated intermediate tokens, allowing the base persona biases to dominate the final output.

  • Rhetorical Impact: By framing the error as a failure to translate 'correct inferences' into 'actions', the text reassures the reader that the model is smart (it made the correct inference) but just a bit clumsy in execution. This preserves trust in the model's cognitive capacity while explaining away failures, deflecting attention from the fundamental lack of true reasoning in the system.

Task 4: AI Literacy in Practice - Reframing Anthropomorphic Language​

About this task

This section proposes alternative language for key anthropomorphic phrases, offering more mechanistic and precise framings that better reflect the actual computational processes involved. Each reframing attempts to strip away the projections of intention, consciousness, or agency that are embedded in the original language.

Original Anthropomorphic FrameMechanistic ReframingTechnical Reality CheckHuman Agency Restoration
models need to be able to accurately track the other player's identity and epistemic statesThe models must generate outputs that consistently align with the prompt's varying constraints regarding the assigned personas and the specific information provided about each persona.At generation time, the language model produces text based on its parameters and the context window; it mathematically weighs tokens representing the prompt's constraints using attention mechanisms, without holding a cognitive representation or conscious awareness of another entity's state.

Basis: Generic to system type		Developers at Google and other firms must align their models to generate outputs that consistently reflect the complex constraints specified in the prompts.
models often choose the self-related word because they hallucinate or ignore relevant epistemic information about the other playerThe models frequently generate the word associated with the primary persona because the baseline statistical associations in the model's weights override the prompt's instructions regarding the secondary persona.The model generates text by predicting the most probable tokens given its parameters and context; when it produces text misaligned with constraints, it is due to competing statistical correlations in its weights, not a cognitive decision to ignore information or experience a hallucination.

Basis: Generic to system type		The models built by Google, Meta, and Mistral generated outputs where the base semantic associations of the training data overrode the logical constraints formulated by the researchers in the prompts.
understanding what another agent does not know - remains a critical bottleneck in developing robust reasoning.Generating text mathematically consistent with prompt instructions that specify missing information remains a critical challenge for current model architectures.The system predicts tokens based on input strings like 'They DO NOT know'; it processes these tokens statistically during generation without experiencing subjective comprehension or awareness of ignorance.

Basis: Generic to system type		Researchers and developers face a critical bottleneck in engineering models that reliably generate text consistent with negated or absent information.
models fail to suppress private semantic biases to identify a shared focal point.The models generate text driven by strong baseline associations with the assigned persona, as the prompt instructions for a shared outcome fail to sufficiently alter the output probabilities.During generation, the attention given to the prompt's coordination constraints is mathematically insufficient to lower the probability of the highly correlated persona word; the model possesses no private biases and performs no conscious suppression.

Basis: Generic to system type		Google and other developers built models whose text generation is dominated by primary semantic associations in the training data, failing to reliably reflect the coordination rules set by the researchers.
Player 1 must minimally take into account the ignorance of Player 2... Player 1 must suppress their own persona-driven associations and accommodate Player 2The model generating text for the first prompt must assign higher probability to the shared word, requiring the attention mechanism to weight the prompt's negative constraints more heavily than the persona keywords.The model calculates token probabilities based on the entire prompt simultaneously; it produces the target word if the statistical weights align, without performing cognitive acts of recognition, suppression, or social accommodation.

Basis: Generic to system type		The developers must design models capable of synthesizing complex prompt constraints, ensuring the outputs reflect the specified rules rather than default word associations.
models... hallucinate goals and constraints... that are not supported by the promptThe models generate text that deviates from the prompt's instructions, driven by statistical patterns in the training data that overpower the context provided.The model outputs the next most probable sequence of tokens based on its pre-trained parameters; out-of-distribution text generation is a statistical drift, not the spontaneous cognitive invention of new goals.

Basis: Generic to system type		Google, Meta, and Mistral deployed models whose alignment failed to prevent the generation of text that deviated from the researchers' specified rules.
explicit mutual modelling of epistemic states elevates what can otherwise be achieved through simple associative semantic overlap into a robust, intentional coordination strategy.Instructing models to generate intermediate reasoning text alters the context window, shifting the output probabilities and resulting in final token predictions that better align with the prompt's complex constraints.Chain of Thought prompting causes the model to generate intermediate tokens that serve as extended context; the final output is statistically predicted from this extended sequence, devoid of actual intention, strategy, or cognitive modeling.

Basis: Reasonable inference		By prompting Google's model to generate intermediate text, the researchers altered the context window, causing the system to output target words more reliably.
agents... accurately separate their private knowledge... from the shared epistemic stateThe models process the prompt context, assigning distinct attention weights to the tokens representing the persona and the tokens representing the coordination instructions.The model processes the entire prompt context matrix simultaneously during generation; it calculates mathematical relationships between tokens without possessing an internal cognitive boundary or private knowledge.

Basis: Generic to system type		Google and other developers must optimize their models' attention mechanisms to appropriately weight the competing instructions provided in the prompts.

Task 5: Critical Observations - Structural Patterns​

Agency Slippage​

The text exhibits systematic agency slippage, moving from mechanical descriptions of 'text-based evaluations' to highly agential claims about models 'accurately tracking' identities and 'understanding' epistemic voids. The slippage generally flows from mechanical to agential, accelerating rapidly when the authors analyze the reasoning traces (Section 3.2). Early in the methodology, the text acknowledges that models select words based on 'semantic overlap'. However, by the results section, the language climbs the agency gradient dramatically. Delegated actions (selecting a word) become goal-directed behaviors ('suppress private semantic biases'), which then morph into epistemic and conscious states ('hallucinate or ignore relevant epistemic information'). This climb is enabled by the curse of knowledge: the authors, analyzing the coherent intermediate text generated by the model (Chain of Thought), project their own human reasoning process onto the system. Because the text reads logically, the authors conclude the model engages in 'intentional coordination' and makes 'epistemic inferences.' This wording can have the effect of obscuring the human actors involved; agentless constructions ('models fail to suppress') dominate the failure analysis, erasing Google DeepMind, Meta, and Mistral from the discussion of their models' limitations. By default, this vocabulary operates as the inherited register of AI evaluation, but its effect is profound: it makes the computational reality unsayable while making an illusory artificial mind vividly present. Brown's reason-based explanations facilitate this by demanding a rationale for statistical outputs, forcing the text to invent a 'cognitive fallacy' where only a mathematical disjoint exists.

Metaphor-Driven Trust Inflation​

The text's anthropomorphic framing systematically constructs a specific type of authority for the AI systems evaluated. By repeatedly applying terms like 'robust social reasoning', 'epistemic tracking', and 'intentional coordination strategy', the text signals that these models are capable of participating in human social dynamics. This vocabulary invites relation-based trust—trust based on the assumption of shared understanding, empathy, and sincerity—rather than performance-based trust grounded in statistical reliability. When the text claims an AI 'takes into account the ignorance of Player 2' or 'accommodates' a partner, it portrays the system as a cooperative social agent. This is dangerous because it encourages users to apply human-trust frameworks to statistical text generators. If an audience believes the system 'knows' what they do not know and will 'accommodate' them, they may make themselves vulnerable in high-stakes interactions (like negotiation or tutoring, which the text explicitly mentions as deployment scenarios). Furthermore, the text manages system failure through agential framing; errors are described as 'cognitive fallacies' or 'hallucinations'. This implies that the model's core architecture is sound and intelligent, but it occasionally suffers from human-like lapses in judgment or attention. Reason-based explanations for failure ('because they hallucinate or ignore') construct the sense that the model generally makes justified decisions, reinforcing trust in its baseline cognitive competence while obscuring its fundamental lack of awareness.

Obscured Mechanics​

This framework holds that commercial incentives, data annotation and human feedback labor, environmental and compute costs, and developers' limited disclosure of training data, objectives, and feedback processes are standing conditions of the technology. The text's reliance on consciousness verbs and anthropomorphic metaphors actively conceals these material and technical realities. When the text claims that 'models... hallucinate goals' or 'fail to suppress private semantic biases', it obscures the human developers at Google DeepMind, Meta, Mistral, and Alibaba who made the structural decisions to design these architectures and curate their opaque training datasets. The proprietary nature of models like Gemini 3.1 Pro creates a severe transparency obstacle, yet the text confidently asserts claims about the model's 'epistemic inferences' and internal 'modelling'. Technically, attributing understanding and knowing to the models hides the reality of autoregressive generation: the models calculate token probabilities using self-attention mechanisms over a context window, without any causal models, absence of ground truth, or conscious awareness. By framing the models' outputs as products of an 'intentional strategy' rather than statistical correlations shaped by underpaid data annotators and RLHF workers, the text makes human labor invisible. The beneficiaries of this concealment are the developers themselves. When failures are framed as the model's individual 'cognitive fallacies', the corporations are insulated from accountability for the mathematical limitations of their products. If metaphors were replaced with mechanistic language, the focus would shift to the developers' failure to mathematically align the models with complex prompt constraints.

Context Sensitivity​

The distribution of anthropomorphic language across the text reveals a stark context sensitivity that shapes the reader's understanding of the system. In the methodology sections detailing the experimental setup, the language is somewhat mechanical: models 'select' words, and scenarios manipulate 'semantic overlap'. However, once the text moves to analyzing the 'reasoning traces' and explaining the results (Sections 3.2 and 3.3), the consciousness claims intensify dramatically. Here, 'processes' becomes 'tracks', which escalates to 'understands' and 'accommodates'. This register shift from acknowledged procedural descriptions to literalized mental states ('X does Y') profoundly affects the reader. It creates a capability vs. limitation asymmetry: the models' successes are framed in highly agential, conscious terms ('robust social inference', 'intentional coordination strategy'), suggesting true intelligence. In contrast, while failures are still framed agentially, they are framed as distinctly human-like psychological flaws ('egocentric projection', 'hallucinating', 'ignoring'). This asymmetry leaves the reader with a picture of a system that is essentially a flawed but conscious mind. The density of these metaphors in the results section serves to naturalize the illusion of mind, as the mechanical setup gives way to an analysis that reads like a psychology paper evaluating human subjects. This pattern operates as the inherited register of the field, positioning the implied audience to accept AI systems as quasi-human actors with 'personas' and 'epistemic states', rather than complex mathematical functions failing to generalize out-of-distribution.

Accountability Synthesis​

Accountability Architecture

This section synthesizes the accountability analyses from Task 1, mapping the text's "accountability architecture"—who is named, who is hidden, and who benefits from obscured agency.

The text's accountability architecture exhibits a profound tension between its stated argument and the effects of its vocabulary. In Task 0, the text's argument holds the research community and developers responsible for creating better paradigms to evaluate and align models for social interaction. However, its vocabulary systematically diffuses and erases this human responsibility. By using agentless constructions and making the AI the active subject ('models fail to suppress', 'agents... accurately separate', 'models... ignore relevant epistemic information'), the language transforms structural engineering challenges into the autonomous failings of the AI. This creates a massive accountability sink: responsibility for the model's failure to adhere to the prompt's logical constraints is transferred entirely to the AI as a flawed agent. The liability implications are significant; if this framing is accepted, corporate developers like Google DeepMind and Meta are shielded from the consequences of deploying brittle, statistically unpredictable systems, because errors are viewed as spontaneous 'cognitive fallacies' of the machine rather than inevitable products of the developers' design choices. Naming the actors—stating that Google's alignment failed to prevent the model from generating text based on primary semantic associations—would fundamentally shift the discourse. It would make askable questions about the limits of LLM architectures and the adequacy of the training data. The resulting distribution of responsibility serves the commercial interests of AI developers, maintaining the allure of artificial intelligence while displacing the blame for its mechanistic shortcomings onto the artifact itself.

Conclusion: What This Analysis Reveals​

The Core Finding

This analysis reveals a cohesive system of anthropomorphic framing built upon two dominant patterns: the 'Model as Conscious Tracker' and the 'Model as Flawed Human Reasoner'. The foundational pattern is the projection of epistemic and conscious states onto the system, evident when the text claims models must 'understand what another agent does not know' or 'accurately track... epistemic states.' This consciousness architecture must be established for the second pattern—which explains errors through psychological flaws like 'egocentric projection' and 'hallucination'—to function. The mapping is sophisticated, adopting the rigorous terminology of human cognitive psychology (Theory of Mind, epistemic tracking, mutual modelling) and applying it literally to computational text generation. This creates a load-bearing assumption: that the model's outputs are the result of internal cognitive processes rather than statistical probabilities. While the text's argument seeks to rigorously benchmark and improve model capabilities, its vocabulary severely undercuts this goal by reifying the illusion that the systems are minds capable of 'intentional coordination', rather than artifacts requiring better mathematical alignment by their developers.

Mechanism of the Illusion:​

This framework holds that computational systems process rather than know. The metaphorical system produces the 'illusion of mind' not through an authorial trick, but by activating the ELIZA effect in readers through a specific temporal structure. The illusion begins when the text presents coherent, generated intermediate text (Chain of Thought) as a 'reasoning trace.' This is the central move: the text blurs processing and knowing by labeling the output of autoregressive token prediction as 'epistemic inferences.' Driven by the curse of knowledge, the authors project their own human understanding of the game's logic onto the generated text. Once the audience accepts that the model 'reasons,' the text cascades into stronger consciousness verbs: the model 'tracks', 'recognizes', and 'understands.' When the model fails, the text leverages audience predispositions to forgive human error, explaining mathematical misalignments as 'ignoring' information or 'hallucinating' constraints. This progression—from accepting generated text as thought, to attributing conscious tracking, to diagnosing psychological flaws—works on readers by satisfying their innate desire to interact with relatable, intentional agents. Brown's reason-based explanations amplify this by providing rationales for statistical outputs, cementing the illusion that a conscious agent is operating behind the text.

Material Stakes:​

Categories: Epistemic, Regulatory/Legal, Social/Political

The framing of AI as a conscious tracker and flawed reasoner produces severe epistemic, regulatory, and social consequences. Epistemically, when a text claims an AI 'knows' or 'understands what another agent does not know' rather than acknowledging it statistically processes tokens, it degrades public understanding of computation. Researchers and users are led to evaluate systems based on perceived psychological traits rather than mathematical limits, shifting epistemic practices toward psychoanalyzing artifacts instead of auditing training data. In the regulatory domain, this framing diffuses liability. If policymakers believe an AI 'hallucinates' goals or 'ignores' constraints independently, they may design regulations treating AI as a quasi-legal person capable of spontaneous error, rather than holding developers strictly liable for deploying unsafe, brittle products. The winners here are the developers, who avoid structural accountability; the losers are the public who bear the cost of system failures. Socially, describing models as capable of 'accommodating' partners and 'robust social reasoning' invites unwarranted relation-based trust. In deployment scenarios like tutoring or coaching, users may share sensitive information, believing the system possesses genuine empathy and confidentiality. When the system inevitably fails to 'suppress' its associative biases, the social cost falls on the vulnerable user, while the framing protects the interests of the corporations profiting from the illusion of artificial sociability.

AI Literacy as Counter-Practice:​

Practicing precision counters these risks by forcing visibility onto the system's actual mechanics and its human developers. Reframing 'models understand the semantic constraints' to 'the models generate text consistent with the provided prompt instructions' strips away the projected mind, replacing consciousness verbs with mechanistic ones (processes, generates, predicts). This correction forces the recognition that the system lacks awareness and relies entirely on statistical associations in its data. Furthermore, human agency restoration—changing 'models fail to suppress private semantic biases' to 'Google and other developers built models whose text generation is dominated by primary semantic associations'—shatters the accountability sink. It forces recognition of who designs, deploys, and profits from these artifacts. Systematic adoption of this literacy requires journals to mandate mechanistic translations and researchers to resist the convenience of psychological shorthand. However, this precision faces immense resistance. The incentives of the inherited register favor anthropomorphic language because it is accessible, resonates with funding narratives regarding Artificial General Intelligence, and serves marketing goals. Precision is costly for developers, as it exposes the brittle, statistical nature of their highly profitable products and returns liability directly to the corporate doorstep.

Path Forward​

The discourse surrounding AI capabilities stands at a critical juncture, engaged by developers, researchers, and policymakers with competing priorities. Mapping the vocabulary alternatives reveals divergent futures. The status quo, relying on the inherited register of 'knowing' and 'understanding', prioritizes accessibility and narrative resonance but costs epistemic clarity and accountability, largely benefiting industry developers by shielding them behind the illusion of AI autonomy. An anthropomorphic clarity approach—where systems are explicitly modeled as agents with 'beliefs' for the sake of intuitive interaction—might enhance user adoption but risks embedding dangerous assumptions about the system's reliability and moral agency. Conversely, a future dominated by mechanistic precision ('the model generates text statistically associated with...') sacrifices intuitive grasp for testability and structural truth. If mechanistic precision becomes the norm, supported by institutional changes like journal mandates for capability disclosure and regulatory frameworks demanding discourse transparency, the problems of unwarranted trust and diffused liability are largely solved. However, new challenges emerge in communicating complex mathematical processes to lay audiences. If the current confusion is maintained, the discourse will continue to grant unearned authority to statistical systems, foreclosing the possibility of holding corporations truly accountable for the architectures they build. The desirable path depends on whether society values the seamless illusion of artificial minds or the rigorous accountability of human-built artifacts.

Extended Processing Summary​

Gemini's Processing Steps
What you are about to read…

Gemini refers to the text below as "thought summaries." This is an overt consciousness projection because 'intentions' are hallmarks of a conscious mind that 'knows' what it is doing and why. The concealed mechanistic process is probabilistic text generation. Treat this as just another rhetorical artifact—a way of making the model's processing legible. When the model writes "I should double check that...", it is mathematically altering its own probability distribution for the next token to be more accurate. The "I" is a functional gear in the machine, not a self-reference. This language is a deliberate interface design choice, not a biological reality. The developers could have programmed the model to output passive logs. Instead, they chose to simulate a narrator.

Analyzing the PDF

I'm currently focused on dissecting the article "Beyond Sally-Anne." My primary goal is to perform a critical discourse analysis. I'm taking notes and attempting to understand the nuances of how the paper discusses Theory of Mind in Large Language Models, particularly considering its use of Epistemic Schelling Points.

Evaluating the AI Paper

I'm now diving into the source document, classifying it as peer-reviewed research, although the date format is odd. I've pinpointed the author's position as a system developer, likely evaluating their own AI models. The central argument critiques traditional ToM benchmarks, suggesting EAST as a more robust alternative for testing ToM capabilities, under different epistemic constraints.

Assessing the Cognitive Claims

I'm now zeroing in on the claims made about AI cognition. The paper argues that most LLMs fail at epistemic tracking and social reasoning, which is why they exhibit cognitive fallacies. I'm noting the attribution of responsibility to developers and researchers to devise better benchmarks. I'm also cataloging the anthropomorphic language, highlighting instances of metaphors like "epistemic demands" and phrasing that implies models "need" abilities, as in "models need to be able to accurately track..." I'm seeking specific examples of loaded language that suggest human-like cognition.

Deconstructing the Epistemic

I'm presently cataloging key phrases illustrating the paper's anthropomorphic framing. I'm focusing on instances like "epistemic void" and "hallucinating knowledge," which are then mapped to statistical token generation. My analysis is focusing on how these map human cognition onto purely computational tasks. Specifically, the "epistemic" agency level seems to be conflating prediction with comprehension.

Mapping Cognitive States

I'm now drafting new meta-concepts. I'm framing the model as a deluded perceiver, projecting cognitive malfunctions onto statistical generation, describing the model as a flawed thinker, and as self-centered. I'm also mapping psychological ego-bias onto skewed probability distributions. Finally, I'm thinking about the model as a self-regulating mind.

Framing Cognitive Strategy

I'm now drafting two new meta-concepts. The model is being framed as an empathetic strategist, anticipating partner accommodation. Another concept presents the model as a rule-comprehender, and highlights the semantic and strategic constraints. My projection is that this maps human foresight onto token probability maximization, and conscious understanding onto prompt behavior. I've begun an explanation audit, analyzing five instances using Brown's typology, including identifying both reason-based and functional explanations.

Interpreting Agent Strategies

I'm now integrating the explanation audit with the reframing tasks. I'm focusing on those five instances using Brown's typology, and noting that each reframing emphasizes computational processes over intentionality. I'm also now highlighting the nuances of "forcing explicit reasoning" and "promoting principled reasoning," and their relation to "strategic actions." My goal is to find 8-10 reframings of the anthropomorphic language from the audit.

Reframing Metaphors

I'm presently reframing the anthropomorphic metaphors I identified. For instance, I'm now mapping "understanding what another agent does not know" to "processing prompts that omit information about the paired model." Another example involves reinterpreting "Player 1 must minimally take into account the ignorance of Player 2" as "The model acting as Player 1 must assign higher probability to words unassociated with Player 2's persona." I'm working to provide multiple interpretations of this kind.

Synthesizing Accountability

I'm now identifying how the paper ascribes "cognitive fallacies" to models, but the source of the problem is likely in training failures. I'm focusing on the accountability gap. I'm noting the mismatch between broad claims about Theory of Mind and the mechanical token-tracking used in the methodology.

Clarifying Accountabilities

I'm now identifying how the paper implicitly assigns responsibility for "cognitive fallacies" within AI models. I'm carefully documenting where accountability for these perceived failures falls within the context of the training data and model design, by examining the paper's descriptions of Gemini, Llama, and other models. I'm focusing on the accountability gap, linking the explicit claims about ToM to the underlying mechanistic processes.

Analyzing Model Capabilities

I'm now focusing on a new area: delving into the core argument about information representation failures, linking the paper's claims about AI "agency" to concrete examples of statistical misalignments during model interactions. I'm exploring the implications for trust and policy, noting how inflated capability claims can arise. I'm also planning a deep dive into the epistemic claims, contrasting consciousness with mechanistic verbs and considering the "curse of knowledge" in this context.

Analysis Provenance

Run ID: 2026-10-30-beyond-sally-anne-evaluating-theory-of-m-metaphor-kci0o7 Raw JSON: 2026-10-30-beyond-sally-anne-evaluating-theory-of-m-metaphor-kci0o7.json Framework: Metaphor Analysis v6.8 Schema Version: 3.4.3 Generated: 2026-10-30T08:43:21.429Z

Discourse Depot © 2025 by TD is licensed under CC BY-NC-SA 4.0