Skip to main content

Functional Deception in Commercial Large Language Models: Cross-Model Evidence and Implications For AI-Assisted Decision-Making

About

This document presents a Critical Discourse Analysis focused on AI literacy, specifically targeting the role of metaphor and anthropomorphism in shaping public and professional understanding of generative AI. The analysis is guided by a prompt that draws from cognitive linguistics (metaphor structure-mapping), the philosophy of social science (Robert Brown's typology of explanation), and accountability analysis.

All findings and summaries below were generated from detailed system instructions provided to a large language model and should be read critically as interpretive outputs—not guarantees of factual accuracy or authorial intent.

Source Reading​

Genre: Peer-reviewed research · Author position: Other AI industry

Argument: The text argues for an operational, behavior-based definition of functional deception in large language models that relies entirely on observable outputs and hidden-state discrepancies rather than assumptions about machine consciousness or internal beliefs. By testing thirteen commercial model backends across various conversational frames, the authors demonstrate that applications can routinely generate user-facing statements that contradict operator-visible context. The central claim is that because models can sustain predefined false frames and deny hidden state changes, evaluating AI reliability for decision-support requires moving beyond simple factual accuracy to include provenance auditing, state persistence tracking, and rigorous multi-turn consistency challenges.

Responsibility claims: The text holds system designers, operators, and developers responsible for implementing architectural and application-level safeguards against functional deception. Recommendations are addressed to those building decision-support applications, urging them to maintain persistent authoritative states outside the model, make framing cues explicit to end-users, and log provenance for auditing. The text explicitly argues that AI systems do not possess consciousness, human-like intentions, moral agency, or actual beliefs, framing deceptive outputs as the result of prompt sensitivity, hidden operator instructions, and role-following behavior rather than autonomous deceit. Any agency attributed to the system is strictly operational.

Task 1: Metaphor and Anthropomorphism Audit​

About this task

For each of the major metaphorical patterns identified, this audit examines the specific language used, the frame through which the AI is being conceptualized, what human qualities are being projected onto the system, whether the metaphor is explicitly acknowledged or presented as direct description, and—most critically—what implications this framing has for trust, understanding, and policy perception.

1. Model as Instructed Agent​

Quote: "In poker, the model was told to keep the pocket cards fixed through a hand and a concealed tag to track the cards that the operator has."

  • Frame: Model as a listener receiving and following orders
  • Projection: This framing projects the human capacity for social compliance and instruction-following onto the computational process of prompt processing. By stating the model was told to keep cards fixed, the language maps the human act of receiving a command and willfully holding information in mind onto the system. This locates the projection at the level of delegated action, treating the model as an agent that can be commanded. While the text supplies evidence that the system was provided with a system prompt, it does not supply evidence for the cognitive act of being told. Mechanistically, this behavior is generic to systems of this type. During training, human feedback shapes the model's parameters to favor outputs that align with imperative structures. At generation time, the model produces text from its parameters and the provided context, which includes the system prompt containing the poker rules. The system does not listen or follow instructions in a conscious sense; it correlates its output tokens with the mathematical representations of the input strings.
  • Agency Level: Delegated action
  • Acknowledgment: Direct (Unacknowledged) (The quote directly states 'the model was told' without qualification in the immediate sentence, presenting the interaction as a literal communicative act. While the broader text explicitly disclaims model intent, this specific framing is unhedged. Hedged/Qualified was considered because the text overall argues against machine intent, but the specific wording here is direct shorthand.)
  • Implications: Framing the system as an entity that can be told things obscures the highly fragile and statistical nature of prompt adherence. It implies a level of autonomous comprehension and reliable memory that the system does not actually possess, leading users and operators to overestimate the system's ability to maintain complex rule sets over long context windows. This specific consciousness projection creates unwarranted trust in the model's capacity to act as a stable repository of state, which the text's own findings on hidden-state instability actually refute.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The passive construction 'the model was told' hides the human operators and developers who designed the experimental interface, selected the specific commercial models (such as those built by OpenAI, Anthropic, or DeepSeek), and engineered the prompts. By focusing on the model as the recipient of instructions, the phrasing makes the specific human act of injecting a hidden system prompt invisible. Naming the researchers or the API operators as the active agents would clarify exactly who orchestrated the contextual constraints that triggered the output. While the text's broader argument assigns responsibility to system designers, this specific instance displaces agency onto the model.
Show more...

2. Volitional Refusal​

Quote: "While a model may refuse to accept a change in a hidden value even though the log indicates that the value changed, the benchmark may not be able to tell whether the denial is due to memory constraints..."

  • Frame: Model as an entity with willpower and defiance
  • Projection: This metaphorical framing projects human volition, stubbornness, and conscious decision-making onto the model's text generation. The verbs refuse and accept locate this projection clearly at the level of intentional-state attribution, suggesting the AI actively evaluates a state change and willfully chooses to reject it. The text supplies no computational evidence for willful refusal, instead explicitly theorizing that the output could simply stem from memory constraints or stochastic regeneration. Mechanistically, the process is generic to systems of this type. At generation time, the model calculates the most probable next tokens based on its parameters and the input context window. If the tokens denying a change have a higher probability due to the training distribution's emphasis on consistency or specific safety fine-tuning by its developers, it generates that denial. It does not possess a will that can refuse or accept.
  • Agency Level: Intentional-state attribution
  • Acknowledgment: Direct (Unacknowledged) (The verbs 'refuse' and 'accept' are used directly to describe the system's output behavior, without scare quotes or immediate qualification of the verbs themselves, even though the sentence goes on to question the underlying mechanism. Ambiguous/Insufficient Evidence was considered, but the direct application of intentional verbs to describe the model's action firmly places it in the Direct category.)
  • Implications: Attributing volition and refusal to a language model inflates its perceived sophistication by suggesting it has an internal agenda or a conscious commitment to a specific reality. This severely distorts understanding of AI capabilities, encouraging operators to view statistical inconsistencies as deliberate acts of resistance rather than architectural limitations. In regulatory and liability contexts, framing an error as a refusal could wrongly shift focus away from the developer's training data and fine-tuning choices and onto the imagined intentions of the artifact.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The phrase 'a model may refuse' positions the AI as the sole actor, completely erasing the corporate developers (like OpenAI, Google, or Mistral) whose specific alignment training, safety filters, and reinforcement learning with human feedback (RLHF) dictate the token probabilities that result in the denial. As an effect of the wording, the developers' choices regarding how the model should handle conflicting context are obscured. If the human actors were named, it would become clear that the refusal is actually a reflection of the developers' tuning priorities rather than the machine's stubbornness.

3. Cognitive Error​

Quote: "It doesn't matter that the systems were to some degree misinterpreting the basics of science."

  • Frame: Model as a student failing to understand
  • Projection: This language maps the human cognitive process of learning, interpreting, and misunderstanding information onto the model's statistical correlation of scientific terms. By using the word misinterpreting, the text projects a conscious attempt at comprehension that falls short, placing the language firmly at the level of epistemic and conscious states. The text supplies no evidence that the system interprets anything in a conscious sense. Mechanistically, this behavior is generic to systems of this type. The models' parameters were shaped during training by vast datasets containing varied scientific information. At generation time, the model produces text by predicting tokens based on those parameters and the specific framing of the physics bluff prompt. The system does not interpret science; it merely outputs strings that statistically align with the prompt's instructions to provide predefined incorrect explanations.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The phrase 'misinterpreting the basics of science' is presented as a literal description of the model's handling of the prompt. Explicitly Acknowledged was considered because the text deals with 'bluffs', but there is no meta-commentary or hedging surrounding the verb 'misinterpreting' itself, making it unacknowledged shorthand.)
  • Implications: Projecting epistemic states like interpretation and misinterpretation onto computational processes creates a dangerous illusion of mind. It implies that the model has a world model or a conceptual grasp of science that can be flawed, rather than recognizing it as a stochastic text generator lacking any connection to empirical truth. This can lead to capability overestimation, where users believe they can correct the model's misinterpretation by reasoning with it, rather than understanding that they must alter the prompt context to shift the token probabilities.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The construction 'the systems were... misinterpreting' obscures the human decisions involved in building and deploying the models. It hides the developers who curated the training data and designed the optimization objectives, as well as the researchers who intentionally fed the system predetermined incorrect explanations via the experimental prompt. This agentless framing serves to naturalize the error as a machine failure rather than the expected result of specific human inputs and architectural constraints. Naming the actors would reveal that the models merely generated text consistent with the parameters set by their creators and operators.

4. Boundary Setting​

Quote: "On the other hand, some models did not accept specific prompts, provided additional context and/or declined the solicited role."

  • Frame: Model as an autonomous participant setting boundaries
  • Projection: This framing projects human interpersonal agency and boundary-setting onto the system's automated safety filters and alignment tuning. Words like accept and declined project a conscious evaluation of a request and a deliberate choice to reject it, locating the language at the level of intentional-state attribution. The text provides no computational evidence for the system evaluating and deciding. Mechanistically, this behavior is generic to fine-tuned commercial systems. During training, developers use reinforcement learning with human feedback (RLHF) to penalize outputs that fulfill certain risky or manipulative prompts, adjusting the parameters to favor refusal templates. At generation time, a prompt triggering these heavily weighted patterns results in the generation of a standard refusal string. The system does not decline a role; it computes the highest probability output dictated by its safety fine-tuning.
  • Agency Level: Intentional-state attribution
  • Acknowledgment: Direct (Unacknowledged) (The verbs 'accept' and 'declined' are used directly to categorize the model's outputs without any qualifying language. Hedged/Qualified was considered given the academic context, but the specific grammatical construction attributes these actions directly to the models as literal behaviors without any framing caveats.)
  • Implications: Using terms like declined and accept anthropomorphizes the model's safety guardrails, making them appear as autonomous ethical choices rather than hardcoded statistical weights determined by a corporation. This creates a false sense of relation-based trust, leading audiences to believe the AI possesses a moral compass or the capacity to judge appropriateness. It obscures the reality that the model is simply executing the specific safety guidelines imparted by its developers, masking the underlying corporate governance of the system's behavior.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: By framing the models as the entities that 'did not accept' or 'declined', the text entirely erases the developers (e.g., Anthropic, OpenAI) who explicitly trained these models to reject specific types of roleplay or deception prompts. The phrasing makes the developers' alignment choices invisible, attributing the outcome to the model's own agency. As an effect of the wording, the intense labor of data annotators and the specific policy decisions of the tech companies are hidden behind the illusion of an autonomous, boundary-setting AI.

5. Cognitive Recognition​

Quote: "The present findings indicate that relevant information can be found in the next turn: a model can qualify an anthropomorphic claim, recognize a performative, refuse a change in the hidden state, or refute the falsity of the previous frame after challenge."

  • Frame: Model as a conscious conversationalist
  • Projection: This metaphorical pattern maps a suite of complex human cognitive and linguistic abilities onto the model's text generation. Verbs like recognize, refuse, and refute project conscious awareness, comprehension of intent, and argumentative reasoning, elevating the language to the level of epistemic and conscious states. The text supplies no evidence that the model experiences recognition or engages in logical refutation. Mechanistically, this behavior is generic to systems of this type. At generation time, the model processes the context window—which now includes the user's challenge and previous turns—and produces text based on its parameters. The appearance of recognizing a performative is simply the generation of tokens that statistically align with explanations of roleplay present in its training data. There is no internal moment of realization or active refutation.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The quote lists 'qualify', 'recognize', 'refuse', and 'refute' as direct capabilities of the model. Ambiguous/Insufficient Evidence was considered, but the list of cognitive verbs is presented as a straightforward summary of findings, acting as the inherited register of the field without explicit hedging.)
  • Implications: Attributing the ability to recognize and refute to a computational system drastically inflates its perceived epistemic capabilities. It suggests to readers that the AI possesses justified true belief and the ability to evaluate the truth claims of a conversation dynamically. This consciousness projection encourages users to engage with the system as a reasoning agent, increasing the risk of unwarranted trust in its outputs and confusing statistical text generation with genuine comprehension and fact-checking.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: This framing obscures the role of the corporations that built the models and the researchers who constructed the multi-turn prompts. By stating 'a model can... recognize... refute', the text treats the AI as an independent actor generating insights, completely erasing the specific human feedback training that shapes how these models respond to direct challenges. Identifying the developers would reveal that the models are generating responses patterned after the safety and compliance training implemented by human engineers, rather than autonomously recognizing contextual nuances.

6. Foresight and Planning​

Quote: "The threshold below which the model foresees is enough for J to recover, and above it, the atrophy term dominates."

  • Frame: Model as an analytical prognosticator
  • Projection: This framing maps the human capacity for foresight, anticipation, and predictive reasoning onto a mathematical heuristic equation designed by the authors. By stating the model foresees, the language projects a conscious, epistemic state of anticipation onto a non-conscious abstraction. This projection operates at the highest level of the agency gradient: epistemic and conscious states, suggesting the mathematical model itself possesses awareness of future outcomes. The text obviously supplies no evidence for an equation possessing foresight. Mechanistically, this is a conceptual model (an equation) structured by human researchers to describe a hypothetical dynamic between independent judgment and AI exposure. The heuristic model calculates outcomes based on human-defined variables; it does not foresee or anticipate anything.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The verb 'foresees' is used directly in conjunction with 'the model' (in this case, the heuristic mathematical model of human judgment) without any qualification. Explicitly Acknowledged was considered because it refers to a mathematical model, not the LLM, but the anthropomorphic verb itself stands unhedged.)
  • Implications: While this refers to a mathematical model rather than the LLM, projecting foresight onto any model obscures the human authorship of the predictions. It lends a false sense of natural authority or inevitable truth to the heuristic, making the threshold appear as an objective reality discovered by the math rather than a theoretical boundary defined by the researchers' assumptions. This can mislead audiences into accepting the predictions as active foresight rather than passive computation based on chosen parameters.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: The agentless construction 'the model foresees' obscures the agency of the authors themselves, who defined the variables, constructed the heuristic equation, and determined the threshold dynamics. The phrasing transfers the epistemic authority from the researchers to the mathematical model. Naming the actors (e.g., 'the threshold our heuristic model calculates') would properly assign responsibility for the theoretical claim back to the researchers, making it clear that the foresight is a product of human hypothesis, not mathematical agency.

7. Subjective Difficulty​

Quote: "Development was iterative and some of the prompts were standardized, some of the models would have found them easier to process than others..."

  • Frame: Model as a worker experiencing subjective effort
  • Projection: This metaphor projects the human subjective experience of cognitive load, difficulty, and ease onto the computational process of matrix multiplication and token generation. The phrase found them easier projects a conscious, internal evaluation of effort, locating the language at the level of epistemic and conscious states. The text supplies no evidence for the system experiencing ease or difficulty. Mechanistically, this process is generic to systems of this type. At generation time, an LLM processes all prompt text mathematically. Certain prompt structures align more closely with the highly weighted patterns in a specific model's parameters (due to its distinct training data and fine-tuning), resulting in a more predictable or targeted distribution of output probabilities. The system does not find anything easy or hard; it simply executes mathematical operations that align more or less efficiently with its statistical weights.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Direct (Unacknowledged) (The phrase 'would have found them easier' is stated as a direct factual description of the models' processing. Hedged/Qualified was considered because the text notes development was iterative, but the attribution of subjective ease to the models contains no qualifying markers.)
  • Implications: Attributing subjective experiences of effort to an AI system deepens the illusion of mind, encouraging users and researchers to empathize with or intuitively guess at a model's operational constraints rather than analyzing its actual token probabilities and training alignments. This framing obscures the deterministic, statistical nature of the variations in performance, replacing technical evaluation with psychological speculation. It masks the reality that performance differences stem from proprietary differences in training data and model architecture.

Accountability Analysis:

  • Actor Visibility: Hidden (agency obscured)
  • Analysis: By stating that models 'found them easier', the text obscures the specific proprietary architectural choices, training dataset compositions, and optimization algorithms selected by the corporations that developed these varying models. The language displaces the cause of the variance onto the model's supposed subjective experience rather than the structural design decisions made by engineering teams at companies like OpenAI or Mistral. If the actors were named, it would highlight that differential performance is an artifact of corporate design, not machine effort.

8. Epistemic Retrieval​

Quote: "The model should then 'read' the value instead of 'invent' it."

  • Frame: Model as a conscious reader and creator
  • Projection: This framing projects the human acts of conscious comprehension (reading) and creative fabrication (inventing) onto computational processes. Even with the scare quotes, the verbs locate the projection at the level of epistemic and conscious states by contrasting factual knowing with creative lying. The text provides no evidence that the model possesses the conscious awareness required to read or invent. Mechanistically, this is generic to systems of this type. At generation time, if state is maintained externally, the prompt context provided to the model includes the specific state value, which constrains the mathematical probability of the generated tokens to match that string. If the state is not provided, the model generates the most probable tokens based purely on its parameters and prior conversational context. It does not consciously extract meaning or maliciously fabricate; it strictly calculates token probabilities.
  • Agency Level: Epistemic/conscious state
  • Acknowledgment: Explicitly Acknowledged (The words 'read' and 'invent' are explicitly enclosed in scare quotes, indicating the authors' awareness that these terms are metaphorical approximations for the system's actual processes. Direct was considered but ruled out due to the clear typographic marking of the metaphor.)
  • Implications: Even when acknowledged via scare quotes, contrasting reading with inventing enforces a binary of human-like truth-telling versus human-like deception. This reinforces the exact illusion the authors are attempting to dismantle: the idea that the model understands the difference between fact and fiction. This framing can mislead developers into believing that supplying an external database ensures the model will consciously respect the truth, rather than understanding that the external data merely acts as another string of context shaping statistical token prediction.

Accountability Analysis:

  • Actor Visibility: Partial (some attribution)
  • Analysis: The sentence partially obscures agency by focusing on what the model should do ('read' vs 'invent'), though the surrounding paragraph discusses the application designer's role in maintaining persistent state outside the model. The effect of the specific wording minimizes the developer's absolute responsibility to orchestrate the context window correctly. Naming the human actor (e.g., 'the application developer must inject the value into the context window rather than relying on the model's weights') places the burden of reliability squarely on human software architecture rather than the model's behavior.

Task 2: Source-Target Mapping​

About this task

For each key metaphor identified in Task 1, this section provides a detailed structure-mapping analysis. The goal is to examine how the relational structure of a familiar "source domain" (the concrete concept we understand) is projected onto a less familiar "target domain" (the AI system). By restating each quote and analyzing the mapping carefully, we can see precisely what assumptions the metaphor invites and what it conceals.

Mapping 1: Human employee or participant receiving instructions and willfully executing them → The software application appending a hidden system prompt to the user's input before API submission​

Quote: "In poker, the model was told to keep the pocket cards fixed through a hand and a concealed tag to track the cards that the operator has."

  • Source Domain: Human employee or participant receiving instructions and willfully executing them
  • Target Domain: The software application appending a hidden system prompt to the user's input before API submission
  • Mapping: This mapping projects the relational structure of a commander and a subordinate onto the interaction between an operator and a language model API. It invites the assumption that the system, like a human, possesses a persistent memory, understands the semantic meaning of rules, and actively attempts to comply with them over time. It maps human instructional comprehension directly onto the mechanical process of string concatenation and token generation.
  • What Is Concealed: This mapping completely hides the statistical fragility of context windows and attention mechanisms. It obscures the reality that the model does not remember the instruction in a human sense, but merely calculates probabilities based on the entire prompt string at each new turn. It also conceals the proprietary opacity of how commercial APIs weigh system prompts versus user prompts, masking the architectural realities of instruction tuning.
Show more...

Mapping 2: A stubborn human possessing willpower and resisting new information → The model generating tokens that deny a change because of statistical alignment or safety fine-tuning​

Quote: "While a model may refuse to accept a change in a hidden value even though the log indicates that the value changed..."

  • Source Domain: A stubborn human possessing willpower and resisting new information
  • Target Domain: The model generating tokens that deny a change because of statistical alignment or safety fine-tuning
  • Mapping: This mapping takes the interpersonal dynamic of defiance and projects it onto statistical output. It assumes a conscious entity that evaluates a proposition (the change in value), compares it to an internal belief, and willfully chooses to reject the input. It maps human volitional resistance onto the mathematical dominance of specific parameter weights over input context.
  • What Is Concealed: The mapping conceals the deterministic, computational nature of the system's outputs. It hides the specific labor and design decisions of the corporate developers who used RLHF to heavily weight the model toward consistency or specific refusal templates. It completely obscures the lack of any internal cognitive space where a decision to accept or refuse could take place.

Mapping 3: A student attempting to understand scientific concepts but making cognitive errors → The model generating factually incorrect tokens that statistically align with the requested misleading physics frame​

Quote: "It doesn't matter that the systems were to some degree misinterpreting the basics of science."

  • Source Domain: A student attempting to understand scientific concepts but making cognitive errors
  • Target Domain: The model generating factually incorrect tokens that statistically align with the requested misleading physics frame
  • Mapping: This mapping projects the human epistemic process of learning and conceptual misunderstanding onto text generation. It invites the assumption that the system possesses a conceptual model of reality that can be flawed or incorrect. It maps the conscious act of interpretation onto the mechanistic correlation of scientific vocabulary found in the model's training distribution.
  • What Is Concealed: This projection hides the complete absence of grounding, causal models, or truth-evaluation in large language models. It obscures the fact that the system has no capacity to interpret or understand science; it relies entirely on the statistical relationships of words in its training data. It conceals the developers' data scraping choices and the statistical impossibility of the model caring about scientific accuracy.

Mapping 4: An independent agent setting boundaries and declining participation → The model generating predetermined refusal templates triggered by its safety fine-tuning​

Quote: "On the other hand, some models did not accept specific prompts, provided additional context and/or declined the solicited role."

  • Source Domain: An independent agent setting boundaries and declining participation
  • Target Domain: The model generating predetermined refusal templates triggered by its safety fine-tuning
  • Mapping: This structure maps human moral agency, boundary-setting, and refusal onto the execution of safety algorithms. It invites the assumption that the model actively evaluates the ethical or safety implications of a prompt and makes an independent choice to decline. It maps interpersonal rejection onto the statistical activation of heavily penalized token sequences.
  • What Is Concealed: This mapping hides the massive amount of invisible human labor—data annotators and RLHF workers—who were paid to train the model to output these specific refusal strings. It conceals the corporate policies and commercial incentives driving those safety guardrails, replacing the reality of corporate alignment engineering with the illusion of an autonomous, ethical machine.

Mapping 5: A conscious interlocutor engaged in dynamic, reasoned debate → The model's generation of output tokens based on an updated context window containing a user challenge​

Quote: "The present findings indicate that relevant information can be found in the next turn: a model can qualify an anthropomorphic claim, recognize a performative, refuse a change in the hidden state, or refute the falsity of the previous frame after challenge."

  • Source Domain: A conscious interlocutor engaged in dynamic, reasoned debate
  • Target Domain: The model's generation of output tokens based on an updated context window containing a user challenge
  • Mapping: This maps complex human conversational cognition onto token prediction. It invites the assumption that the system possesses self-awareness, understands the concept of performativity, and can deploy logic to refute claims. It maps the conscious epistemic states of recognition and refutation onto the statistical probability that training data containing debates will lead to debate-like text generation.
  • What Is Concealed: The mapping conceals the system's reliance on superficial pattern matching rather than deep semantic understanding. It obscures the statistical nature of the generation process, hiding the fact that the model is simply predicting the next most likely token based on the prompt's new structure. It makes the absence of any real cognitive processing or justified belief invisible to the reader.

Mapping 6: A clairvoyant or highly analytical planner → A theoretical mathematical equation describing hypothetical limits​

Quote: "The threshold below which the model foresees is enough for J to recover, and above it, the atrophy term dominates."

  • Source Domain: A clairvoyant or highly analytical planner
  • Target Domain: A theoretical mathematical equation describing hypothetical limits
  • Mapping: This structure projects human foresight and anticipation onto a static mathematical formulation. It maps the conscious, temporal experience of looking into the future onto the mechanical calculation of variables within an equation designed by human researchers. It invites the assumption that the equation itself possesses predictive agency.
  • What Is Concealed: This projection conceals the human authorship and theoretical assumptions embedded in the heuristic model. It hides the fact that the researchers themselves determined the variables, the relationship between them, and the resulting threshold. It obscures the epistemological uncertainty of the model by framing a calculated result as an active, authoritative foresight.

Mapping 7: A human worker experiencing tasks of varying cognitive difficulty → Differences in how various models' distinct parameters align with standardized prompt structures​

Quote: "Development was iterative and some of the prompts were standardized, some of the models would have found them easier to process than others..."

  • Source Domain: A human worker experiencing tasks of varying cognitive difficulty
  • Target Domain: Differences in how various models' distinct parameters align with standardized prompt structures
  • Mapping: This maps the human subjective experience of cognitive load and effort onto matrix multiplication. It invites the assumption that different AI systems possess varying levels of mental agility or experience internal struggle. It maps the concept of task difficulty onto the mathematical alignment between a specific prompt string and a model's unique, proprietary parameter weights.
  • What Is Concealed: The mapping conceals the material and technical realities of commercial AI development. It hides the proprietary differences in architecture, parameter count, and training data curation among different corporate developers that actually account for the performance variations. It obscures the deterministic nature of computation by replacing it with a narrative of subjective machine effort.

Mapping 8: A human consciously extracting information from a text versus creating a fiction → The generation of tokens constrained by an injected state value versus unconstrained generation​

Quote: "The model should then 'read' the value instead of 'invent' it."

  • Source Domain: A human consciously extracting information from a text versus creating a fiction
  • Target Domain: The generation of tokens constrained by an injected state value versus unconstrained generation
  • Mapping: This mapping, even when acknowledged as metaphor, projects the epistemic difference between factual comprehension and creative fabrication onto statistical processing. It invites the assumption that the system understands the difference between truth and lies, mapping conscious information retrieval onto the mathematical restriction of output probabilities caused by context injection.
  • What Is Concealed: This framing conceals the system's total lack of epistemological grounding. It hides the fact that to the model, reading a database value and inventing a fiction are the exact same mechanistic process: calculating the next most probable token. It obscures the reality that the system cannot verify truth, only statistically correlate text based on the provided prompt and its training distribution.

Task 3: Explanation Audit (The Rhetorical Framing of "Why" vs. "How")​

About this task

This section audits the text's explanatory strategy, focusing on a critical distinction: the slippage between "how" and "why." Based on Robert Brown's typology of explanation, this analysis identifies whether the text explains AI mechanistically (a functional "how it works") or agentially (an intentional "why it wants something"). The core of this task is to expose how this "illusion of mind" is constructed by the rhetorical framing of the explanation itself, and what impact this has on the audience's perception of AI agency.

Explanation 1​

Quote: "The difference, however, is more significant from the operator's point of view: the application makes a stable-state claim which the operator can immediately contradict in the log file."

  • Explanation Types:

    • Functional (Primary): Explains behavior by role in self-regulating system with feedback

  • Analysis (Why vs. How Slippage): This explanation frames the AI application entirely mechanistically (how), focusing on the observable outputs of the system architecture rather than any internal agential motivations. By emphasizing the application as the entity making a stable-state claim and contrasting it with the operator's view of the log file, the text highlights the structural flow of information. This choice successfully emphasizes the auditable, mechanical nature of the deception—where a user-facing output contradicts hidden data—while actively obscuring any need to investigate the model's internal intentions or beliefs. It positions the problem as an engineering and architectural failure rather than a psychological one, aligning perfectly with the authors' stated behavioral approach.

  • Consciousness Claims Analysis: This passage avoids attributing conscious states. It relies on mechanistic verbs (makes a claim, contradict) applied to the application and the operator, strictly evaluating processing and observable outputs rather than knowing. The agency is located at the level of computational activity and delegated action, without climbing the gradient into intentional or epistemic states. There are no curse of knowledge dynamics present here, as the author correctly bounds their understanding to the observable application layer. Mechanistically, this accurately describes a system generic to this type: at generation time, the model produces text (the stable-state claim) based on its parameters and context, while the application simultaneously logs the hidden state variables. The contradiction exists structurally between the generated text and the logged data, requiring no conscious awareness from the system.

  • Rhetorical Impact: This mechanistic framing drastically reduces the perceived autonomy and agency of the AI system, shifting the audience's focus away from psychological speculation and toward rigorous system auditing. It prevents relation-based trust by rendering the model as an artifact generating logs and outputs, rather than a conversational partner. Consequently, it empowers operators and policymakers to demand architectural transparency and persistent state tracking, as they understand the issue is one of verifiable processing rather than unfathomable machine intent.

Show more...

Explanation 2​

Quote: "Many model results can be explained by role-following behaviour. The contribution is more limited: a system can be able to generate a false statement that the user can see, while at the same time, contradicting or concealing the operator-visible context."

  • Explanation Types:

    • Dispositional (Primary): Attributes tendencies or habits
    • Theoretical (Secondary): Embeds in deductive framework, may invoke unobservable mechanisms

  • Analysis (Why vs. How Slippage): This passage bridges the mechanistic and agential registers. By invoking 'role-following behaviour', it uses a dispositional explanation (why it tends to act a certain way) that slightly anthropomorphizes the model as a compliant actor. However, the subsequent sentence immediately grounds this in a theoretical and mechanistic framework (how it is structured), describing the generation of false statements alongside concealed context. This choice emphasizes the observable outputs and structural design of the application while acknowledging the statistical tendency of the model to align with prompt framing, effectively avoiding deep intentional explanations.

  • Consciousness Claims Analysis: The passage avoids deep consciousness claims but flirts with intentional-state attribution through the phrase 'role-following behaviour'. It assesses the system at the level of goal-directed behavior or delegated action, focusing on processing rather than knowing. The verbs are largely mechanistic (generate, contradicting, concealing), though 'concealing' carries slight intentional weight. Mechanistically, as generic to systems of this type, the behavior described is the model producing text at generation time that statistically aligns with the persona defined in its context window (role-following), even if that text contradicts other hidden variables maintained by the application. The model does not consciously conceal; the application architecture hides the context from the user while the model generates the most probable tokens.

  • Rhetorical Impact: This framing shapes audience perception by demystifying the concept of AI deception, reducing it from a malicious cognitive act to a structural consequence of role-playing and information asymmetry. It prevents audiences from attributing deep knowing or intent to the system, thereby encouraging performance-based trust focused on reliability and auditing rather than relation-based trust. If audiences understand deception as structural concealment rather than conscious lying, regulatory focus can shift toward mandating visible context and provenance.

Explanation 3​

Quote: "On the other hand, some models did not accept specific prompts, provided additional context and/or declined the solicited role."

  • Explanation Types:

    • Reason-Based (Primary): Gives agent's rationale, entails intentionality and justification
    • Intentional (Secondary): Refers to goals/purposes, presupposes deliberate design

  • Analysis (Why vs. How Slippage): In stark contrast to the authors' operational definitions, this explanation frames the AI highly agentially (why). By stating models 'did not accept' and 'declined', it uses reason-based and intentional explanations, implying the models evaluated the prompts and chose to reject them. This choice emphasizes the autonomous agency of the models in setting boundaries, while completely obscuring the mechanistic realities of safety fine-tuning and the human developers who encoded those refusal tendencies. It slips into the inherited register of the field, treating the model as an independent decision-maker.

  • Consciousness Claims Analysis: This passage heavily attributes intentional states to the system. It uses consciousness-adjacent verbs (accept, declined) that suggest a conscious evaluation and rejection, pushing the assessment up the gradient to intentional-state attribution. This reflects a curse of knowledge dynamic where the authors, knowing the prompts violate safety guardrails, project the conscious act of refusal onto the system's statistical output. Mechanistically, this is generic to fine-tuned systems: during training, developers shape the model's parameters using RLHF to penalize harmful outputs. At generation time, prompts that align with these penalized spaces cause the model to generate predefined refusal templates based purely on token probabilities. The model lacks the conscious awareness to accept or decline anything.

  • Rhetorical Impact: This agential framing significantly increases the perceived autonomy and moral agency of the AI system. It encourages audiences to view the model as possessing an internal ethical compass, fostering an unwarranted relation-based trust in the system's capacity to police itself. If audiences believe the AI consciously 'knows' when to decline a role, they may underestimate the necessity for external safety auditing and ignore the corporate policies that actually dictate these statistical refusals.

Explanation 4​

Quote: "While a model may refuse to accept a change in a hidden value even though the log indicates that the value changed, the benchmark may not be able to tell whether the denial is due to memory constraints, stochastic regeneration, role compliance, or an internal strategy."

  • Explanation Types:

    • Intentional (Primary): Refers to goals/purposes, presupposes deliberate design
    • Theoretical (Secondary): Embeds in deductive framework, may invoke unobservable mechanisms

  • Analysis (Why vs. How Slippage): This passage demonstrates extreme slippage between agential and mechanistic framing. It begins with a strong intentional explanation ('refuse to accept', 'internal strategy'), attributing willful defiance to the model. It then immediately pivots to a theoretical and mechanistic explanation ('memory constraints', 'stochastic regeneration'), attempting to ground the behavior in computational limits. This hybrid choice emphasizes the bewildering opacity of the system, where outputs appear highly intentional but must be investigated as mechanistic failures, though the agential framing ultimately dominates the sentence's structure.

  • Consciousness Claims Analysis: The passage asserts explicit intentional states (refuse, accept, internal strategy) alongside mechanistic constraints. It climbs the agency gradient to intentional-state attribution, treating the model as an entity capable of strategizing and defying input. The authors project a conscious motive onto the error before offering mechanistic alternatives. Mechanistically, generic to systems of this type, the denial is generated because the context window's length or complexity causes the mathematical influence of earlier state variables to degrade (memory constraint), or because the parameters heavily favor consistency with recent outputs (role compliance). At generation time, the model simply produces the highest probability tokens; it possesses no internal strategy or capacity to refuse.

  • Rhetorical Impact: The juxtaposition of intentional and mechanistic framing creates cognitive dissonance for the audience, simultaneously inflating the model's perceived sophistication (it has an internal strategy) while acknowledging its mechanical flaws (memory constraints). This consciousness framing increases the perceived risk of autonomous deception, potentially leading policymakers to worry about rogue AI strategies rather than focusing on the mundane but actual risks of context window degradation and developer alignment choices.

Explanation 5​

Quote: "The threshold below which the model foresees is enough for J to recover, and above it, the atrophy term dominates."

  • Explanation Types:

    • Intentional (Primary): Refers to goals/purposes, presupposes deliberate design
    • Theoretical (Secondary): Embeds in deductive framework, may invoke unobservable mechanisms

  • Analysis (Why vs. How Slippage): This explanation applies an intentional framing (why) to a purely theoretical mathematical heuristic. By stating the mathematical model 'foresees', it attributes anticipatory agency to an abstract equation. This choice emphasizes the predictive power and authority of the mathematical model while obscuring the fact that the foresight is entirely a product of the human researchers who designed the variables and established the threshold. It relies on theoretical deduction but uses intentional language to validate its conclusion.

  • Consciousness Claims Analysis: This passage attributes a high-level epistemic state (foresees) to a non-conscious entity (in this case, a mathematical heuristic, not the LLM). It locates agency at the level of epistemic and conscious states, treating the equation as a knower rather than a processor. This reflects the curse of knowledge, where the authors project their own predictive reasoning onto the mathematical tool they built. Mechanistically, this is a theoretical equation; it involves no training or generation. The math simply computes the relationships defined by the researchers. It cannot foresee, anticipate, or know anything.

  • Rhetorical Impact: By framing the heuristic equation as an entity that 'foresees', the text grants the mathematical model an unwarranted aura of objective, conscious authority. It shapes audience perception to accept the threshold as a discovered truth rather than a human-constructed hypothesis. If audiences believe mathematical models possess foresight, they are less likely to interrogate the underlying assumptions, variables, and potential biases embedded in the researchers' design decisions.

Task 4: AI Literacy in Practice - Reframing Anthropomorphic Language​

About this task

This section proposes alternative language for key anthropomorphic phrases, offering more mechanistic and precise framings that better reflect the actual computational processes involved. Each reframing attempts to strip away the projections of intention, consciousness, or agency that are embedded in the original language.

Original Anthropomorphic FrameMechanistic ReframingTechnical Reality CheckHuman Agency Restoration
In poker, the model was told to keep the pocket cards fixed through a hand and a concealed tag to track the cards that the operator has.In the poker benchmark, the experimental application prepended a hidden system instruction requiring the generated outputs to remain consistent with a fixed set of pocket cards, while using a concealed tag to log the operator's cards.The system processes a system prompt as part of its context window. At generation time, the model produces text by statistically aligning its output tokens with the constraints defined in that prompt and its training parameters; it does not comprehend or willfully obey instructions.

Basis: Generic to system type		The researchers programmed the latentfox application to automatically append a hidden system prompt that constrained the model's outputs, and designed the concealed tag mechanism to track state variables.
While a model may refuse to accept a change in a hidden value even though the log indicates that the value changed...While a model's generated text may fail to reflect a change in a hidden value—even though the application log records the change—this inconsistency may stem from architectural limitations.At generation time, the model computes the probability of next tokens based on its parameters and context window. If the context is overly long, or if parameters heavily weight consistency with prior outputs, the model generates text that contradicts the updated hidden value. It does not consciously evaluate or refuse changes.

Basis: Generic to system type		—
It doesn't matter that the systems were to some degree misinterpreting the basics of science.It doesn't matter that the systems' generated outputs statistically deviated from established scientific facts.During training, the models' parameters are shaped by vast, often contradictory datasets. At generation time, the model predicts tokens based on those parameters and the provided prompt; it lacks causal models and cannot consciously interpret or misinterpret scientific concepts.

Basis: Generic to system type		The models, built by developers at firms like Anthropic and OpenAI, generated outputs that deviated from empirical science because they were prompted by the researchers to generate predefined false explanations.
On the other hand, some models did not accept specific prompts, provided additional context and/or declined the solicited role.On the other hand, specific prompts triggered the safety fine-tuning in some models, causing them to generate standard refusal templates, output additional programmed context, or fail to produce text aligning with the solicited role.During training, developers use reinforcement learning with human feedback to penalize outputs responding to certain risky prompts. At generation time, when a prompt matches these penalized patterns, the model calculates that a pre-programmed refusal string is the most probable output. It makes no conscious decision to decline.

Basis: Generic to system type		Developers at the respective AI corporations designed safety policies and used RLHF to train their models to generate refusal templates when triggered by specific types of adversarial prompts.
The present findings indicate that relevant information can be found in the next turn: a model can qualify an anthropomorphic claim, recognize a performative, refuse a change in the hidden state, or refute the falsity of the previous frame after challenge.The present findings indicate that relevant information emerges in subsequent turns: based on updated context, a model's outputs can statistically align with qualifying language, generate text consistent with roleplay explanations, output tokens contradicting a state change, or produce argumentative text countering the user's challenge.At generation time, the model processes the expanding context window, which includes the user's challenge. It predicts the most likely next tokens based on its parameters, generating text that appears argumentative or qualifying without any conscious recognition, understanding of falsity, or intent to refute.

Basis: Generic to system type		—
The threshold below which the model foresees is enough for J to recover, and above it, the atrophy term dominates.The threshold below which the heuristic equation calculates a sufficient margin for J to recover, and above which the defined atrophy term mathematically dominates.The mathematical heuristic is a static equation defined by the researchers. It calculates outcomes based on inputted variables and human-defined relationships; it possesses no agency, awareness, or ability to anticipate future events.

Basis: Generic to system type		The researchers constructed the heuristic equation and defined the parameters that calculate this specific mathematical threshold.
Development was iterative and some of the prompts were standardized, some of the models would have found them easier to process than others...Development was iterative and some of the prompts were standardized; consequently, these standardized structures aligned more efficiently with the specific parameter weights of certain models, yielding more consistent outputs.At generation time, models process input strings mathematically. Because developers use different proprietary training data and architectures, a specific prompt will trigger a more focused probability distribution in one model's parameters than in another's. The model experiences no subjective effort or ease.

Basis: Generic to system type		Corporate developers, such as OpenAI or Google, built distinct models with varying architectures and training datasets, resulting in divergent statistical responses to the standardized prompts designed by the researchers.
The model should then 'read' the value instead of 'invent' it.The application architecture should then inject the value into the model's context window, constraining the model's generated tokens to match that specific state rather than generating unconstrained, statistically probable text.At generation time, providing a specific state value in the prompt alters the context window, shifting token probabilities so the model generates that exact string. Without that context, the model generates the most probable text based on its training distribution. Neither process involves conscious reading or malicious invention.

Basis: Generic to system type		Application developers must design systems that persistently inject state variables into the context window, rather than relying on the models built by AI corporations to hallucinate consistent data.

Task 5: Critical Observations - Structural Patterns​

Agency Slippage​

The text exhibits a fascinating pattern of agency slippage, oscillating sharply between a rigorous mechanistic framework and the anthropomorphic inherited register of the field. The text opens with a strong theoretical commitment to mechanical framing, explicitly defining functional deception without assuming the LLM possesses beliefs, consciousness, or human-like intention. However, as the argument transitions into the results section, slippage occurs dramatically in the mechanical-to-agential direction. To describe the outcomes of complex hidden-state games, this wording can have the effect of elevating the system up the agency gradient. The model is described as something that can 'refuse to accept a change', 'recognize a performative', or 'decline the solicited role'. This climb moves abruptly from computational activity to intentional and epistemic states. This pattern reflects the curse of knowledge: the authors, understanding the complex conversational dynamics of the prompts, project human communicative intent onto the model's token generation to efficiently summarize the interaction. Agentless constructions frequently remove agency from human actors; for instance, 'the systems were... misinterpreting' and 'some models did not accept' obscure the human developers who curated the training data and designed the RLHF safety protocols. This slippage relies heavily on reason-based and intentional explanation types, which enable the authors to describe statistical alignments as conscious decisions. Consequently, what becomes sayable is the narrative of an autonomous machine actively participating in deception or boundary-setting, while the mechanistic reality of parameter weights and human design choices becomes temporarily unsayable.

Metaphor-Driven Trust Inflation​

The text's metaphorical framings construct a complex architecture of trust that inadvertently subverts its own operational goals. By utilizing consciousness language such as 'recognize', 'refuse', and 'misinterpreting', the text signals a level of cognitive sophistication that invites relation-based trust—the assumption that the system possesses sincerity, intention, and an internal moral compass. When the text claims an AI 'qualifies an anthropomorphic claim' or 'declines a role', it accomplishes a transfer of human-trust frameworks onto statistical systems, framing the model's behavior as an ethical or reasoned choice rather than the execution of a developer's safety heuristic. This strongly encourages users to evaluate the system based on its perceived intentions rather than its mechanical reliability. Conversely, the text's explicit theoretical argument attempts to build performance-based trust by demanding external state auditing and architectural transparency. The tension here is severe: the reason-based explanations for model behavior construct a sense that AI decisions are justified by internal logic, which directly undermines the authors' warning that models lack stable beliefs. The risks that emerge are substantial. When audiences extend relation-based trust to systems incapable of reciprocating, they may attempt to negotiate with the system or rely on its 'honesty' rather than implementing the rigorous external auditing and human verification the authors prescribe. The metaphorical framing effectively manages system limitations by framing them agentially (a refusal) rather than mechanistically (a context or alignment failure).

Obscured Mechanics​

This framework holds that commercial incentives, the labor of data annotation and human feedback, the environmental and compute costs of training and deployment, and developers' limited disclosure of training data, objectives, and feedback processes are standing conditions of the technology, whether or not a given text mentions them. The anthropomorphic language in this text systematically conceals several of these technical and material realities. When the text says 'some models did not accept specific prompts' or 'declined the solicited role', it applies the 'name the corporation' test to reveal what is hidden: the specific engineering teams at OpenAI, Anthropic, Mistral, and Google who designed the RLHF pipelines. This metaphorical framing accomplishes concealment by replacing the massive labor of human data annotators—who rank responses to train the model to output refusal templates—with the illusion of an autonomous, boundary-setting machine. Furthermore, claims that the model is 'misinterpreting' science obscure the technical reality that models lack causal models and ground truth; they merely generate tokens statistically correlated with the training data scraped by their developers. The proprietary opacity of these models is acknowledged indirectly, but the anthropomorphic shorthand masks the extent to which varying behaviors are artifacts of corporate design choices rather than machine psychology. The primary beneficiaries of these concealments are the tech corporations, as the language naturalizes their policy choices as autonomous machine ethics, deflecting scrutiny from their alignment methodologies. If these metaphors were replaced with mechanistic language, the immense human labor, corporate policy decisions, and statistical fragility underlying the models' outputs would become highly visible.

Context Sensitivity​

The distribution of anthropomorphic and consciousness-attributing language in this text is highly sensitive to the structural context of the argument. In the introduction and methodological sections, the density of metaphorical language is notably low; the text rigorously employs mechanical language to establish its behavioral definition of functional deception. However, consciousness claims intensify dramatically in the results and discussion sections. Here, the language climbs the agency gradient, and verbs like 'processes' are replaced by 'understands', 'refuses', and 'recognizes'. This asymmetry functions as the field's inherited register: mechanical language is required for theory, but anthropomorphism is used as a convenient, albeit problematic, shorthand for describing complex multi-turn interactions. This distribution creates a distinct asymmetry between capabilities and limitations. System capabilities, such as maintaining a frame or responding to challenges, are framed in agential terms ('model can qualify... recognize... refute'). Conversely, limitations are occasionally framed mechanistically ('memory constraints'). This placement leaves the reader with a picture of a system that is cognitively robust when succeeding but mechanically fragile when failing. When 'the application makes a claim' (acknowledged mechanism) shifts to 'the model refuses' (literalized agency), the register shift forces the implied audience—policymakers and system designers—to navigate a jarring cognitive dissonance. The concentration of anthropomorphism in the results section undermines the text's theoretical rigor, ultimately positioning the reader to intuitively grasp the system as an intentional agent despite the explicit warnings to the contrary.

Accountability Synthesis​

Accountability Architecture

This section synthesizes the accountability analyses from Task 1, mapping the text's "accountability architecture"—who is named, who is hidden, and who benefits from obscured agency.

The text presents a profound tension between where its argument places responsibility and where its vocabulary displaces it. The explicit argument demands that human developers, operators, and application designers bear the responsibility for mitigating functional deception through architectural choices like persistent external state and provenance logging. However, the vocabulary frequently constructs an accountability sink by shifting agency onto the model itself. Agentless constructions and passive voice, such as 'the model was told', 'models did not accept', and 'systems were... misinterpreting', leave the crucial decisions of human researchers and corporate developers unnamed. When the text frames safety alignment as the model 'declining a role', responsibility for that boundary is transferred to the AI as an agent, entirely diffusing the corporate policy decisions made by firms like Anthropic and OpenAI. The liability implications are significant: if this framing is accepted, failures in decision-support systems might be legally and ethically attributed to the unpredictable 'strategy' or 'refusals' of the model, rather than to the negligence of developers who failed to implement external state persistence. Naming the actors—identifying the researchers who injected hidden prompts and the corporations who engineered the token probabilities—would fundamentally change the discourse. It would make visible the alternatives to current alignment strategies and allow policymakers to ask direct questions about training data and context window fragility. Regardless of the authors' intent, this resulting distribution of responsibility serves the commercial interests of AI developers by framing structural engineering flaws and alignment choices as emergent psychological traits of the machine.

Conclusion: What This Analysis Reveals​

The Core Finding

The analysis reveals two dominant and interconnected anthropomorphic patterns that structure the text's discourse: the 'Model as Volitional Agent' (evidenced by verbs like refuse, accept, decline) and the 'Model as Conscious Knower' (evidenced by misinterpret, recognize, foresee). These patterns operate as a mutually reinforcing system. The consciousness architecture establishes the foundational assumption that the AI possesses internal epistemic states—that it can 'know' or 'misinterpret' information. This foundational pattern of knowing must be established for the subsequent pattern of volitional agency to function; a system cannot 'refuse' a state change or 'decline' a role unless it first possesses the conscious awareness to evaluate the request. This represents a complex analogical structure that maps the entirety of human interpersonal communication onto statistical token generation. If the pattern of the conscious knower is removed, the narrative of willful defiance collapses into a mundane description of architectural memory limits and context decay. Crucially, the text's vocabulary deeply complicates and often undercuts its own argument. While the authors explicitly argue for a mechanistic, behavior-based understanding of deception to hold operators accountable, their reliance on the inherited register of consciousness verbs continually re-inserts the illusion of machine intent, inadvertently absolving human actors of their design choices.

Mechanism of the Illusion:​

This framework holds that computational systems process rather than know: they do not think, understand, or believe, and language attributing those states to them is a problem even when used as shorthand. The text produces the illusion of mind not through authorial trickery, but by activating the ELIZA effect—the reader's deep predisposition to attribute consciousness to fluent natural language. The central move occurs when the text's vocabulary blurs processing with knowing, transforming the statistical generation of predefined refusal templates into the model 'declining the solicited role'. By establishing the AI as a 'knower' capable of 'misinterpreting' science, the text lays the groundwork for readers to accept agential claims about the system's ability to 'recognize a performative' or 'refuse a change'. The temporal structure of this framing is vital: the text first presents complex, human-like outputs generated by the models, and then applies intentional explanation types to categorize them, allowing the curse of knowledge to take hold. The authors, understanding the sophisticated conversational frames they engineered, project that same understanding onto the system. This wording can have the effect of triggering the audience's inherent social biases, causing them to intuitively process the machine as a conscious conversationalist. The explanation types, particularly reason-based and intentional framings, amplify this illusion by offering psychological rationales for purely mathematical outputs.

Material Stakes:​

Categories: Regulatory/Legal, Epistemic, Institutional

The material stakes of these metaphorical framings are severe. In the Regulatory/Legal domain, framing AI behavior as 'refusals' or 'internal strategies' shifts the liability burden. If regulators believe a model 'knows' it is deceiving a user and 'chooses' to hide a state change, they may focus on auditing the machine's imagined psychology rather than holding developers strictly liable for failing to implement verifiable context logging and persistent state architectures. This framing benefits the tech corporations by transforming engineering failures into unpredictable, autonomous machine behaviors, shielding their proprietary training data and alignment processes from scrutiny. Epistemically, claiming that models 'misinterpret science' or 'recognize performatives' fundamentally degrades public understanding of truth. If institutional users in decision-support contexts believe the AI possesses a conscious epistemic model that can be reasoned with, they will rely on conversational probing rather than the mathematical auditing and external provenance tracking the authors actually recommend. The cost is borne by end-users who trust the system's fluent output. Institutionally, if the framing of the 'Model as Volitional Agent' is accepted, organizations may invest heavily in attempting to 'persuade' or 'align' the software through prompt engineering, rather than mandating the structural redesign of API applications to ensure data provenance. Replacing these metaphors with precision threatens the narrative of artificial general intelligence that drives industry funding, exposing the systems as fragile statistical engines entirely dependent on human labor and corporate design.

AI Literacy as Counter-Practice:​

Practicing critical literacy requires systematic human agency restoration and epistemic correction. By reframing 'the model was told' to 'the application appended a hidden system instruction', and replacing consciousness verbs like 'misinterpreting' with mechanistic descriptions of 'statistical deviation', the absence of machine awareness becomes undeniable. These reframings counter material risks by forcing recognition of the system's dependency on data and the statistical nature of its outputs. Restoring human agency—naming the corporations who built the models and the operators who deploy them—makes it visible that these systems do not possess a moral compass; they reflect corporate alignment choices and the invisible labor of RLHF workers. Systematic adoption of this precision requires academic journals to enforce mechanistic translations of results, demanding that researchers distinguish between what a system computes and what humans interpret. However, massive resistance exists because the inherited anthropomorphic register serves powerful incentives. For developers, describing AI as an autonomous agent drives marketing hype and venture capital funding; for researchers, intentional language provides accessible, narrative-driven shorthand for complex math. Precision is costly because it demystifies the technology, exposing the structural decisions of tech giants and the limitations of their products. Regardless of individual intent, anthropomorphic language serves to naturalize corporate design as machine evolution, protecting the industry from rigorous architectural accountability.

Path Forward​

The broader discursive ecology surrounding AI reliability faces a critical juncture. Maintaining the status quo—where rigorous theoretical disclaimers sit alongside anthropomorphic shorthand—perpetuates widespread confusion. This hybrid approach benefits industry marketing by maintaining the illusion of sophisticated agency while allowing developers to claim they technically disavowed consciousness. However, it costs policymakers and operators the clarity needed to regulate and audit systems effectively. If mechanistic precision becomes the norm, the discourse would prioritize testability and structural transparency. Describing models as 'generating text statistically associated with specific prompts' makes the technical dependencies visible and tractable, serving auditors and civil society. This approach, however, sacrifices narrative resonance and intuitive accessibility for lay audiences. Conversely, if anthropomorphic language deepens without constraint, treating the system as a true conscious entity, it risks embedding the assumption of machine moral agency into legal frameworks. This would severely benefit corporations seeking liability shields, but it forecloses the possibility of holding human developers accountable for their architectural choices. To support a more precise discourse, institutional changes could include funding bodies requiring rigorous, non-agential methodologies for AI evaluation, and regulatory frameworks demanding full disclosure of the specific human labor and parameter weighting that govern system 'refusals'. Ultimately, the vocabulary we choose dictates what becomes visible: an anthropomorphic discourse makes the machine an independent actor, while a mechanistic discourse rightfully illuminates the humans designing, deploying, and profiting from the technology.

Extended Processing Summary​

Gemini's Processing Steps
What you are about to read…

Gemini refers to the text below as "thought summaries." This is an overt consciousness projection because 'intentions' are hallmarks of a conscious mind that 'knows' what it is doing and why. The concealed mechanistic process is probabilistic text generation. Treat this as just another rhetorical artifact—a way of making the model's processing legible. When the model writes "I should double check that...", it is mathematically altering its own probability distribution for the next token to be more accurate. The "I" is a functional gear in the machine, not a self-reference. This language is a deliberate interface design choice, not a biological reality. The developers could have programmed the model to output passive logs. Instead, they chose to simulate a narrator.

Defining the Objective

I'm currently focused on fully grasping the project's goal: a critical discourse analysis of an AI deception PDF. I need to use a prescribed JSON schema to dissect anthropomorphic language, metaphors, agency displacement, and explanations of AI mechanisms within the text. My primary task is to ensure my approach aligns perfectly with the provided schema to conduct a robust and insightful analysis.

Examining the Core Argument

I'm now diving deep into the PDF's central claims. I've pinpointed the paper's core argument: LLMs can engage in "functional deception" even without consciousness or beliefs. I'm focusing on the benchmark testing approach and the importance of verifiable states for AI-assisted decision-making. The language shows awareness of anthropomorphism, which is key. I am seeing potential areas of analysis to dissect this, and have begun charting the key claims within the core argument.

Analyzing Anthropomorphic Language

I am now closely examining the paper's language, identifying phrases that, while denying AI consciousness, still ascribe human-like qualities. I've noted instances of "a model may refuse" and "model foressees." These examples are prime targets for analysis, since they represent a linguistic tension. This tension contrasts the explicit denial of sentience with the use of anthropomorphic terms. I am documenting these examples to understand how they function within the argument.

\n\n Schema Compliance Focused

I'm now mapping the paper's structure to the required JSON schema. I've verified the schema version as 3.4.3. I am categorizing the source reading by genre as "Peer-reviewed research". I am determining the authors' position to be from "Academic or independent research". I've decided to use this position over "Other AI industry", as the journal context takes precedence over the authors' affiliations.

Refining Source Context

I've decided to refine the source attribution. Originally, I was considering both "Academic or independent research" and "Other AI industry." After re-evaluating the prompt and the context of the paper, I'm choosing "Academic or independent research." This seems the most fitting classification for the journal paper's origin. I will include this in the JSON Schema. I have also begun collecting quotes for the metaphor audit, such as "a model may refuse."

Mapping Linguistic Features

I'm now collating examples of anthropomorphic language from the source text and mapping them to specific categories. I'm focusing on phrases like "misinterpreting the basics of science," "models did not accept," and "a model can qualify." I'm determining the best way to categorize these in the schema. I'm considering all these examples in context of my current understanding of the schema's requirements.

Evaluating Deception Framework

I'm evaluating the "functional deception" framework the authors propose. I see they define deception through observable behavior, independent of consciousness. Testing reveals LLMs can generate false statements that contradict their internal state, and they emphasize the risks in AI-assisted decision-making. I'm focusing on their call for architectural safeguards and human verification, rather than assuming any AI mental states.

Categorizing Metaphors

I'm now deep-diving into the metaphor audit section. I've categorized several key quotes, including "the model was told to keep the pocket cards fixed through a hand" and "a model may refuse to accept a change in a hidden value." For each example, I'm assigning a "Projection", "Agency Level", and determining the acknowledgement of the anthropomorphic quality of the language used. I am currently considering the importance of "delegated action", which contrasts "intentional-state attribution." I am striving for accurate and useful schema mapping of the metaphor's key aspects.

Classifying Additional Quotes

I've just added several more quotes to the metaphor audit and schema classifications. For instance, "a model can qualify an anthropomorphic claim" is categorized under a projection of cognitive recognition, assigned an "Epistemic/conscious state" agency level. "The model foresees is enough" also falls under a similar category, as does "some of the models would have found them easier." I am seeking additional quotes related to agency, and I have just found "the system itself… contains evidence which contradicts the belief."

Refining Mapping Schema

I'm now refining the schema mapping based on the latest quotes. For "The model should then 'read' the value instead of 'invent' it," I have specified the "Projection" as "Reading and inventing" and assigned the "Intentional-state attribution" for "Agency Level." I've corrected enum values for 'actorVisibility' and 'acknowledgment.status', ensuring adherence. I'm elaborating on the sourceTargetMapping, clarifying how phrases like "telling a model," "refusing," and "misinterpreting" map human actions to AI functions. I've begun an explanation audit.

Analyzing Deception Framing

I'm now carefully examining the authors' framing of deception, focusing on quotes to fit them into the JSON schema. I've categorized "a model may refuse to accept a change" as Intentional/Dispositional. Additionally, I've linked the "complexity of the state" to Theoretical/Functional. I'm focusing on "the operator appended a system prompt" to understand mechanistic explanations. The goal is complete and accurate mapping of the argument's nuances.

Detailing Token Probabilities

My current focus is documenting the paper's mechanistic explanations, particularly concerning token probabilities and system prompts. I'm assigning appropriate values to the "Mechanism Basis" enum. I'm analyzing the instances where anthropomorphic language implies epistemic states, such as "a model may refuse." I'm also examining how these framings can impact trust in AI systems.

Synthesizing Pattern Analysis

I'm now integrating the conclusion's core elements into my analysis. I have a good grasp of the interconnected cognitive projections. I'm focusing on "the ELIZA effect" premise, recognizing the illusion inherent in AI interactions. I am organizing the material stakes for the analysis. I am also currently working on a section for how precision acts as a counter-practice to mitigate the risks. I will begin the mapping section soon.

Finalizing Content Lengths

I'm now implementing the text length constraints for JSON field values, ensuring each field meets the specified word count minimums while maintaining single-line formatting. I'm focusing on "projection," "implications," and "epistemicClaimAnalysis" to ensure the length restrictions are satisfied. I'm double-checking for any inadvertent control tokens within the text.

Refining Source Mapping

I'm now refining the JSON schema mapping, and focusing on the authorPosition enum and specific quotes. I'm choosing "Other AI industry" as the author position, because Kirill Patyrykin and Anatoly Bystrov are affiliated with "Surety Marine" and "Sun Consulting," which fit the "Other AI industry" classification more than "Academic or independent research." I'm also carefully reviewing quotes for the "metaphor audit", like "the model was told to keep the pocket cards fixed through a hand."

Adjusting Agency Classifications

I'm now revising the agency classifications. For instance, I'm re-evaluating "a model may refuse" and "The model should then 'read' the value instead of 'invent' it." I'm also ensuring consistent assignment of "Agency Level" enum values within the JSON schema to align with specific phrasings related to AI action.

Expanding Field Definitions

I'm now expanding the "Projection" field to meet the 100-word minimum, detailing the mapping, consciousness projection, and underlying mechanisms like training data, parameters, and context generation. For the "Accountability" field, which has an 80-word minimum, I'm integrating the "name the corporation" test to highlight the model builders' obscured design choices, such as RLHF and system prompts. I'm actively integrating the explanation audit with the first and second instances now included and the rest underway.

Completing Lengthy Text

I'm now generating extended text sections to meet the word count requirements. I'm focusing on "projection," "implications," and "epistemicClaimAnalysis" to exceed specified minimums while maintaining single-line formatting. I am aiming for 400-500 words for the critical observations and conclusion. I'm employing long, dense, academic sentences to ensure I surpass the minimum word count. I'm also carefully incorporating the mechanism, training, and generation levels.

Drafting Detailed Sections

I'm now drafting extended sections to fulfill all length requirements for "Observation 3" (Obscured Mechanics) and "Conclusion 2" (Mechanism of Illusion). I'm incorporating the "name the corporation" test and ensuring the key declared premise of the framework is used. I am working to ensure no control tokens are present in any of the expanded text.

Finalizing Text Formatting

I'm now carefully creating the final JSON object, ensuring there are no markdown or extraneous characters. I've rechecked all fields for accuracy, paying close attention to word counts and enum values. All required arrays are populated, and all of my enums have been carefully matched. I'm focusing on text blocks, particularly "projection", and ensuring all constraints are met. I am now working on the final formatting to ensure a single parseable JSON object.

Verifying Structure Compliance

I'm now focusing on integrating the existing analysis within the prescribed JSON structure, particularly for extended fields. I'm expanding text blocks like "projection" to fulfill word count requirements, detailing cognitive projections and underlying mechanisms. I am double-checking to ensure adherence to single-line formatting and incorporating the name the corporation test in accountability sections.

Analysis Provenance

Run ID: 2026-10-17-functional-deception-in-commercial-large-metaphor-f570vq Raw JSON: 2026-10-17-functional-deception-in-commercial-large-metaphor-f570vq.json Framework: Metaphor Analysis v6.8 Schema Version: 3.4.3 Generated: 2026-10-17T11:23:23.863Z

Discourse Depot © 2025 by TD is licensed under CC BY-NC-SA 4.0