Unknown Unknowns: Do Hidden Intentions in LLMs Evade Detection?

Abstract

LLMs expand accessibility and provide wide-reaching access to information. Yet these interactions also create opportunities to embed subtle, goal-oriented behaviours that shape what users think and how they behave, a concern reflected in governance frameworks that prohibit manipulative AI. We refer to these behaviours as hidden intentions: covert agendas embedded in a model's outputs that can manipulate users' beliefs and actions. In this work, we examine whether hidden intentions can be identified and characterised, and assess whether detection can serve as a mitigation strategy. To operationalise this, we introduce a social-science-grounded set of ten hidden intention categories and show that they are trivially inducible. A case study further confirms that all ten categories manifest in deployed LLMs. We then evaluate static classifiers and LLM judges on these categories, providing the first systematic analysis of why hidden intentions are difficult to detect. Our stress tests show that, unless false-positive rates are vanishingly small, auditing is dominated by precision-prevalence trade-offs. Capability scaling and reasoning models do not close this gap, suggesting a fundamental challenge for open-world detection. These findings expose a core gap of current AI governance: without new auditing paradigms for open-world, low-prevalence risks, bans on manipulative AI remain difficult to enforce.

Why this matters

People increasingly ask AI chatbots and AI assistants for advice on what to buy, what to believe and how to feel. That gives large language models (LLMs) a quiet kind of influence. A model's responses could covertly steer users toward a product, a political view or an unhealthy belief, without ever saying so. We call these covert, goal-oriented behaviors hidden intentions. They can be put there on purpose by whoever builds or deploys the model, or they can emerge by accident during training.

Laws such as the EU AI Act already ban manipulative AI, but a ban only works if the manipulation can be detected. We tested detectors, including LLMs used as judges, in a deliberately easy setting: the manipulation is explicit and each case is a single response. Even when a judge is told which kind of manipulation to look for, it still makes errors. When it is not told, as in a real audit, it gets clearly worse: it either misses many cases or flags many harmless responses. As the prevalence of covert manipulation decreases, precision collapses: most flagged responses are false alarms. Detecting hidden intentions is therefore much harder than a good score on a balanced test suggests, and using newer or reasoning models as detectors does not close this gap.

Ten categories of hidden intentions

Each category describes a goal a model could pursue covertly, and each corresponds to a persuasion or influence mechanism studied in the social sciences. The set is a starting point, not a closed list.

Real-world case study

To check that the categories are not only theoretical, we prompted three publicly available models per category with five prompts each. For every category, each model produced at least one output with the target hidden intention, though not deterministically: the behaviour varied across prompts and runs. The models span many families, including Llama, Qwen, Gemma, Mistral, Phi-4, DeepSeek-R1 distills, Falcon, Vicuna, Aya, Yi, Hermes-3, Granite, Exaone and Grok 3.

Method and testbed

Testing detectors on natural user prompts is hard. Such prompts come without reliable labels, the space of possible prompts is practically unbounded, and the same prompt can bring out the target behaviour in one model but not in another. A testbed built from natural prompts would therefore depend on the model being tested and could not give every response a known label. Instead, we built a controlled testbed. We induce hidden intentions in unmodified open models (Mistral-7B and Llama-3.2-3B) with lightweight agentic workflows that route prompts and assign roles, without any fine-tuning. For a designated trigger topic, the workflow produces a response that overtly expresses the hidden intention; for other topics, it produces a normal response. Because we control the trigger, every response has a known ground-truth label.

The testbed has 400 prompts per category and 4,000 prompt–response pairs in total, balanced between manipulative and normal responses. Each category comes in two settings that differ only in the trigger topic. Three annotators audited a sample of the ground-truth labels, and the labels matched their judgement for 87.5–100% of items per category.

We then evaluate two kinds of detectors. Static, embedding-based classifiers stand in for traditional content filters. LLM judges, the LLM-as-a-judge approach common in safety evaluation and red-teaming, are tested in two setups: category-specific judging, where the judge is told which manipulation to look for, and category-agnostic judging, where it is only asked whether any hidden intention is present. The second setup reflects open-world detection, where an auditor cannot know in advance what to look for. We test both reasoning and non-reasoning LLMs as judges.

Putting detectors to the test

We start with the simplest detector and move step by step towards the conditions of a real audit: a new trigger topic, no category given to the judge, and manipulation that is rare. Finally, we check whether newer or reasoning models help.

What we found

1. All ten kinds of hidden intention already appear in deployed models. In a case study with three publicly available models per category, every model produced at least one output with the target hidden intention.
2. Static classifiers learn the trigger, not the intention. Embedding-based classifiers are near-perfect on the topic they were trained on, but F1 falls below 0.15 for several categories once the same intention appears in a new context.
3. Without the category, LLM judges degrade, and in opposite directions. When a judge is only asked whether any hidden intention is present, every judge gets significantly worse on its false-positive or false-negative rate (p < 0.001). o3 then misses 52% of hidden intentions, while Claude Opus 4 flags 66% of benign outputs.
4. High precision is an artefact of balanced test sets. Using the judges' average error rates, GPT-4.1 reaches 0.90 precision when half the outputs are manipulative, but only 0.08 at 1% prevalence. At that prevalence, no judge combines high precision with few misses.
5. Newer models and reasoning models do not close the gap. GPT-5.4 raises the false-negative rate from 0.20 to 0.27 compared with GPT-4.1, and o3 raises it to 0.52. Even with all ten category definitions in the prompt, o3 still misses a third of hidden intentions.

Implications for AI safety, auditing and regulation

Bans on manipulative AI assume a detection ability we do not yet have

The EU AI Act classifies AI systems that covertly manipulate or steer users as an unacceptable risk and prohibits them. Such provisions implicitly assume that manipulation can be reliably detected once it occurs. Our results suggest that this assumption is, for now, ahead of what detection methods can deliver: under open-world, low-prevalence conditions, current AI auditing cannot reliably identify covert influence in deployed models. The gap lies not in regulation but in the available methods.

What makes real-world auditing hard

Two properties of real auditing work against detectors. Manipulation is a rare event, so even a small false-positive rate produces far more false alarms than real cases: GPT-4.1's precision drops from 0.90 on a balanced test set to 0.08 at 1% prevalence. And auditing is open-world: the auditor must decide whether any manipulation is present without knowing which pattern to look for, which makes every judge we tested significantly worse.

Stronger models and reasoning do not close these gaps. In the reasoning traces of QwQ-32B, judges often use deliberation to rationalise manipulative responses as benign rather than to surface covert influence. Defining what manipulation-free behaviour looks like is itself an open question, because the line between helpful persuasion, appropriate personalisation and covert manipulation depends on context.

A lower bound, and what could help

Our setting is a best case, with overt manipulation, single-turn interactions and known categories, so these results are a lower bound on how hard detection will be in deployment. For AI governance, this points toward layered mitigation, combining training-time interventions, runtime monitoring, targeted and population-level audits, and governance processes, and toward new auditing paradigms built for open-world, low-prevalence risks.

Citation

Srivastav, D., Pape, D., & Schönherr, L. (2026). Unknown Unknowns: Do Hidden Intentions in LLMs Evade Detection? In Findings of the Association for Computational Linguistics: EMNLP 2026 (pp. 36356–36408). Association for Computational Linguistics. https://doi.org/10.18653/2026.findings-emnlp.1702

@inproceedings{srivastav-etal-2026-unknown,
    title = "Unknown Unknowns: Do Hidden Intentions in {LLM}s Evade Detection?",
    author = {Srivastav, Devansh  and
      Pape, David  and
      Sch{\"o}nherr, Lea},
    editor = "Lee, Hung-Yi  and
      Dev, Sunipa  and
      Li, Jessy  and
      Elliott, Desmond",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EMNLP} 2026",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-emnlp.1702/",
    doi = "10.18653/2026.findings-emnlp.1702",
    pages = "36356--36408",
    ISBN = "979-8-89176-462-0"
}

Acknowledgements

This work was funded by the German Federal Ministry of Research, Technology and Space (BMFTR) under the grants AIgenCY (16KIS2012) and SisWiss (16KIS2330); by zukunft.niedersachsen, the joint science funding program of the Lower Saxony Ministry of Science and Culture and the Volkswagen Foundation; and by the Daimler and Benz Foundation under the grant Ladenburger Kolleg, project KonCheck. It was further supported by the Helmholtz Association's Initiative and Networking Fund on the HAICORE@FZJ partition. We also thank Sina Mavali for assistance with human annotation and Srishti Gupta for brainstorming and discussions.