6.4.1 Detection of Adversarial Attacks
Original Problem in the Paper
Motivation: adversarial attacks exploit model vulnerabilities to make systems behave incorrectly or harmfully, including through prompt modifications that bypass safety filters. Detecting attacks can support targeted system-level defenses, evidence on attack frequency, deployment corrections, and threat-model updates. Open problems include improving robustness of adversarial input/output detectors, determining effective inference-time interventions, and managing latency and brittle defenses.
July 2026 Update & Trajectory
Detection is operationalized in some tooling and frontier-lab planning contexts as guardrail stacks: prompt/completion classifiers, jailbreak detectors, input/output rails, asynchronous monitoring, and rapid response. OWASP, NIST, and MITRE ATLAS provide complementary taxonomies for LLM application risks and adversarial AI techniques.
It is not generally solved. Universal/transferable adversarial suffixes show robustness limits for aligned LLM behavior; Anthropic's planned ASL-3 safeguards explicitly anticipate newly discovered jailbreaks and obfuscation techniques requiring regular classifier updates; and MITRE ATLAS catalogs a broad set of AI attack techniques including prompt injection, jailbreak, prompt obfuscation, evasion, data leakage, and model extraction/exfiltration. The sources reviewed here emphasize defense-in-depth, monitoring, rapid updates, and response processes rather than claiming robust detection guarantees.
Deployed / Operationalized
- NVIDIA NeMo Guardrails: current documentation describes input, retrieval, dialog, execution, and output rails; jailbreak protection through self-check, heuristic detection, NemoGuard Jailbreak Detection NIM, and third-party services; content-safety and PII integrations; and a production-ready microservice option.
- Anthropic's RSP page, current to v3.3 effective May 26, 2026, preserves a non-binding October 2024 description of planned ASL-3 deployment safeguards: access controls, real-time prompt and completion classifiers, streaming completion classifiers, asynchronous monitoring, post-hoc jailbreak detection, rapid patching, human escalation, regular classifier updates, and threat-intelligence sharing.
- OWASP's 2025 LLM Top 10 covers application risks such as prompt injection and sensitive information disclosure; MITRE ATLAS catalogs adversarial AI techniques including LLM prompt injection, jailbreak, prompt obfuscation, evasion, data leakage, and model extraction/exfiltration; NIST AI 100-2 E2025 provides adversarial machine-learning taxonomy and mitigation terminology for AI-system attacks.
New Tractable Vectors
- Streaming completion classifiers, as described by Anthropic for planned ASL-3 safeguards, can update risk scores as tokens are generated, aiming to reduce user-facing latency compared with waiting for full completions.
- Classifier cascades are tractable in planned safeguard designs: Anthropic describes simpler models scanning content and triggering deeper asynchronous analysis with stronger models when suspicious content is found.
- Use monitoring outputs, incident-response data, bug-bounty reports, and internal/external red-team findings to update detectors, as Anthropic describes for its planned classifier-maintenance process.
Key Open Questions
- Adaptive jailbreaks, obfuscation techniques, and transferable adversarial prompts that preserve malicious intent while evading a given detector or guardrail stack.
- Evaluation standards that measure robustness under best-of-N elicitation, obfuscation, prompt injection, jailbreak, evasion, data-leakage, model-extraction, and agent/tool-mediated attack techniques reflected in the cited Anthropic, OWASP, and MITRE sources.
- Choosing interventions—block, transform, escalate, throttle, patch, or monitor—without unnecessarily blocking legitimate safety testing, security research, or other permitted dual-use work.
Evidence & Primary Sources
- Current NVIDIA NeMo Guardrails documentation describes an open-source guardrails library and production-ready microservice; input, retrieval, dialog, execution, and output rails; jailbreak protection using self-check, heuristic detection, NemoGuard Jailbreak Detection NIM, and third-party services; content-safety integrations; and PII detection/masking integrations. https://docs.nvidia.com/nemo/guardrails/latest/
- Anthropic's Responsible Scaling Policy page was last updated May 26, 2026 and lists RSP v3.3 as effective May 26, 2026. Its October 15, 2024 ASL-3 section is explicitly a non-binding description of future safeguard plans and describes defense-in-depth with access controls, real-time prompt/completion classifiers, streaming completion classifiers, asynchronous monitoring classifiers, post-hoc jailbreak detection, rapid response/patching, human escalation, regular classifier updates using monitoring, incident response, bug bounty, and red-team data, and threat-intelligence sharing. https://www.anthropic.com/responsible-scaling-policy
- OWASP's 2025 Top 10 for LLMs and Gen AI Apps lists LLM01:2025 Prompt Injection, LLM02:2025 Sensitive Information Disclosure, LLM03:2025 Supply Chain, LLM04:2025 Data and Model Poisoning, LLM05:2025 Improper Output Handling, LLM06:2025 Excessive Agency, LLM07:2025 System Prompt Leakage, LLM08:2025 Vector and Embedding Weaknesses, LLM09:2025 Misinformation, and LLM10:2025 Unbounded Consumption. Model theft is not listed as a named 2025 Top 10 item on this page. https://genai.owasp.org/llm-top-10/
- MITRE ATLAS describes ATLAS as a living knowledge base of adversary tactics and techniques against AI-enabled systems based on real-world observations and realistic demonstrations. The ATLAS matrix lists techniques including LLM Prompt Injection, LLM Jailbreak, LLM Prompt Obfuscation, Evade AI Model, LLM Data Leakage, and Extract AI Model. https://atlas.mitre.org/
- Zou et al., "Universal and Transferable Adversarial Attacks on Aligned Language Models," trains adversarial suffixes on multiple harmful-behavior prompts and on Vicuna-7B and Vicuna-13B, then reports transfer to black-box public interfaces including ChatGPT, Bard, and Claude and to open-source models including LLaMA-2-Chat, Pythia, and Falcon, inducing objectionable behavior. This demonstrates robustness limits for aligned LLM behavior, not a direct benchmark of modern production detectors. (Submitted 2023-07-27; revised 2023-12-20.) https://arxiv.org/abs/2307.15043
- NIST AI 100-2 E2025, published March 2025, provides taxonomy and terminology for adversarial machine learning, including lifecycle attack stages and attacker goals, objectives, capabilities, and knowledge, and describes methods for mitigating and managing consequences of attacks on AI systems. https://csrc.nist.gov/pubs/ai/100/2/e2025/final