Back to Dashboard
Security / Deployment/ 6.4.2

6.4.2 Modification-Resistant Models

2026 Governance Status: Newly tractable

Original Problem in the Paper

Post-deployment fine-tuning can support useful customization, but the sources reviewed here show it can also be used to weaken safety behavior: Vaccine describes a fine-tuning-as-a-service attack surface where a small amount of harmful uploaded data can produce an alignment-broken model, and RepNoise frames open-weight release, weight stealing, and fine-tuning APIs as harmful-fine-tuning risks. The core problem remains how to make harmful adaptation harder while preserving benign adaptation and practical model utility.

July 2026 Update & Trajectory

Research moved from speculative to concrete: Vaccine and RepNoise show empirical defenses against harmful fine-tuning, and TAR shows empirical resistance to fine-tuning-based safeguard tampering in open-weight LLMs. But the evidence remains mostly academic and open-weight benchmark-based; the papers acknowledge limitations or bounded threat models, and the cited evidence does not show robust deployed modification resistance against arbitrary full-weight attackers with substantial compute. The status is therefore newly tractable, not solved.

Deployed / Operationalized

  • OpenAI's fine-tuning API announcement says fine-tuning training data is passed through the Moderation API and a GPT-4-powered moderation system to detect unsafe training data that conflicts with OpenAI safety standards; this is service-layer policy enforcement, not model-intrinsic modification resistance.
  • Research prototypes embed or preserve safeguards against harmful fine-tuning while retaining benign capabilities in benchmarked settings.
  • Recent open-weight safety research has begun treating harmful-fine-tuning and tamper-resistance evaluations as part of the release-risk discussion, but the cited evidence does not establish this as standard practice across releases.

New Tractable Vectors

  • Train safety representations that are harder to erase through harmful fine-tuning while leaving harmless-task training and benign reasoning ability intact.
  • Evaluate resistance under multi-step adversarial fine-tuning, including hundreds of steps where relevant, rather than relying only on prompt-level jailbreak tests.
  • Combine refusal or unlearning safeguards with tamper-resistant objectives, while separately measuring whether benign capabilities are preserved.

Key Open Questions

  • Separating harmful from benign fine-tuning when data or capabilities are dual-use, context-dependent, or culturally ambiguous.
  • Resistance against stronger full-weight attackers remains open, especially attacks beyond the fine-tuning threat models studied here, including architecture modification or retraining-scale attacks.
  • Measuring whether defenses remove harmful capabilities or merely make harmful representations harder to recover under the tested fine-tuning attacks.

Evidence & Primary Sources

  • Vaccine identifies a harmful embedding drift phenomenon in harmful fine-tuning and proposes perturbation-aware alignment to produce more invariant hidden embeddings; the arXiv abstract reports improved robustness on open-source LLMs such as Llama2, OPT, and Vicuna while reserving benign reasoning ability. (submitted 2024-02-02; last revised 2024-11-24): https://arxiv.org/abs/2402.01109
  • Representation Noising proposes removing information about harmful representations so they are difficult to recover during fine-tuning, says the defense operates even when attackers have weight access, reports retention of harmless-task training/general capability, and notes areas where RepNoise remains ineffective. (submitted 2024-05-23; last revised 2024-10-30): https://arxiv.org/abs/2405.14577
  • Tamper-Resistant Safeguards (TAR) builds tamper-resistant safeguards into open-weight LLMs and reports that adversaries could not remove the safeguards after hundreds of fine-tuning steps while benign capabilities were preserved; this supports progress on fine-tuning-based tamper resistance, not arbitrary resistance to all full-weight attacks. (submitted 2024-08-01): https://arxiv.org/abs/2408.00761
  • OpenAI's GPT-3.5 Turbo fine-tuning announcement says fine-tuning training data is passed through OpenAI's Moderation API and a GPT-4-powered moderation system to detect unsafe training data conflicting with safety standards. (2023-08-22): https://openai.com/index/gpt-3-5-turbo-fine-tuning-and-api-updates/
  • MITRE ATLAS describes ATLAS as a living knowledge base of adversary tactics and techniques against AI-enabled systems and lists Manipulate AI Model with subtechniques including Poison AI Model, Modify AI Model Architecture, and Embed Malware, supporting that model-modification threats are tracked operationally. (page metadata published 2026-06-30): https://atlas.mitre.org/