6.3.1 Prevention of Model Theft
Original Problem in the Paper
Motivation: capable models may become valuable theft targets; broader integration can expand exfiltration attack surface; securing weights and system components can prevent unauthorized access that would undermine safety and national-security governance. Open problems include adequate cybersecurity against insider and outsider threats, physical/data-center/hardware/software controls, coordination across actors, threat-vector analysis, and defenses against query/API/logit/side-channel model extraction.
July 2026 Update & Trajectory
Infrastructure protection is becoming more concrete: RAND proposes security levels and preliminary benchmark security systems for frontier-model weight security, and Anthropic publicly lists ASL security safeguards such as access control, compartmentalization, logging, insider-risk programs, red teaming, deception technology, and model-weight controls. RAND also recommends confidential computing to secure weights during use and reduce attack surface. However, the cited primary sources do not show model theft solved against sophisticated insiders or top-tier nation-state actors; RAND explicitly says defending against the most capable actors requires significantly more investment. API-level extraction of precise internal information also remains demonstrated for production language models under Carlini et al.'s disclosed/authorized attack setting. Weight security is increasingly framed as a defense-in-depth security-program and assurance problem with unresolved high-end assurance, not as a solved technical primitive.
Deployed / Operationalized
- RAND recommends centralized, access-controlled weight storage, reduced authorized access, insider threat programs, defense-in-depth, third-party red teaming, hardening model-access interfaces against weight exfiltration, and confidential computing.
- RAND's framework identifies 38 attack vectors, attacker capacities up to highly resourced nation-state operations, five security levels, and preliminary benchmark security systems for frontier-model weights.
- Anthropic publicly lists ASL security safeguards including access management and compartmentalization, artifact integrity/provenance, binary authorization, model-weight access controls, multi-party authorization, mandatory code review, hardware authentication, centralized SIEM/SOAR logging, access monitoring for model weights and high-value IP, honeypots/fake weights, physical security, and external red-team/penetration testing. The cited Anthropic page presents this as Anthropic-specific public safeguard text, with some controls described as planned and many already in place, not as audited cross-lab deployment evidence.
- MITRE ATLAS treats model access, inference-API exfiltration, model extraction, and AI intellectual-property theft as part of a living adversarial-threat matrix for AI systems.
New Tractable Vectors
- Use RAND-style security levels and preliminary benchmark security systems to make frontier-lab weight-security assessments less ad hoc.
- Model-weight honeypots/fake weights and access-monitoring analytics for model weights and high-value IP.
- API hardening against logit/projection-layer extraction by restricting risky logit/logprob interfaces, rate-limiting repeated logit-bias queries, monitoring malicious query patterns, and adding carefully evaluated logit noise where utility tradeoffs are acceptable.
Key Open Questions
- Defending against sophisticated insiders and state-compromised administrators without paralyzing research workflows.
- Quantifying residual risk for nation-state model theft across cloud systems, endpoints, supply chain, physical environments, and human targets.
- Preventing partial model-parameter extraction and broader model-stealing/capability-replication attacks through APIs while preserving legitimate evaluation and fine-tuning access.
Evidence & Primary Sources
- The original paper's Section 6.3.1 frames model theft as a governance-relevant security problem: capable models may become valuable theft targets; integration expands the attack surface; unauthorized access to weights or system components could undermine safety and national-security governance; and open problems include insider/outsider cybersecurity, physical/data-center/hardware/software standards, coordination, threat-vector analysis, and defenses against query/API/logit/side-channel extraction. (2024-07-20; arXiv v2 2025-04-16): https://arxiv.org/pdf/2407.14981
- RAND identifies 38 attack vectors, analyzes attacker capacities from opportunistic criminals to highly resourced nation-state operations, defines five security levels and preliminary benchmark security systems, and recommends weight centralization, reduced authorized access, insider threat programs, defense-in-depth, hardening model-access interfaces, third-party red teaming, and confidential computing. RAND also states there is wide diversity of expert views about defending against more capable actors such as top cyber-capable nation-states and that securing weights against the most capable actors will require significantly more investment. (2024-05-30): https://www.rand.org/pubs/research_reports/RRA2849-1.html
- Anthropic's Responsible Scaling Policy page lists Version 3.3 as effective May 26, 2026 and includes public ASL security-safeguard text covering access management/compartmentalization, artifact provenance, binary authorization, model-weight access controls, multi-party authorization, mandatory code review, hardware authentication, centralized SIEM/SOAR logs, access monitoring, honeypots/fake weights, physical security, and external red-team/penetration testing. The page's October 2024 safeguard section says many measures are already in place while others are planned, so it supports Anthropic-specific safeguard documentation but not broad audited deployment across major labs. (2026-05-26): https://www.anthropic.com/responsible-scaling-policy
- Carlini et al. introduce a model-stealing attack extracting precise, nontrivial information from black-box production language models: recovering the embedding projection layer up to symmetries, extracting OpenAI Ada and Babbage projection matrices for under $20, recovering the exact hidden dimension of gpt-3.5-turbo, and estimating full projection-matrix recovery under $2,000 in queries. The paper supports API-hardening work around logit/logprob/logit-bias interfaces, rate limits, malicious-query detection, and logit noise, while not claiming a complete solution. (2024-03-11): https://arxiv.org/abs/2403.06634
- MITRE ATLAS describes itself as a living knowledge base of adversary tactics and techniques against AI-enabled systems and its matrix includes AI Model Access, AI Model Inference API Access, Full AI Model Access, Exfiltration via AI Inference API, Extract AI Model, and AI Intellectual Property Theft. (reader output published/updated 2026-06-30): https://atlas.mitre.org/