Skip to content
AI-generated research idea🟠Medium💡Moderate InnovationRelated work found

Action-Semantic Privilege Isolation to Resist Indirect Prompt Injection in Autonomous Agents

A system that enforces semantic privilege boundaries between untrusted tool outputs and agent action spaces to prevent indirect prompt injection vulnerabilities.

Time

several weeks

Compute level

Multiple GPUs

Resources

Multi-GPU server (e.g., 4-8 H100/H200 or A100 GPUs), standard sandboxed execution testbed (such as Docker and SafeArena), public agentic safety datasets (AgentHarm, SafeArena, DecodingTrust).

What the paper used

Nodes with eight NVIDIA H200 GPUs (141 GB HBM, 700W TDP), 1.6 TB system RAM, 3.2 Tb/s InfiniBand; flagship pre-trained on up to 15.1T tokens.

Prerequisites

LLM security, prompt injection defense, agent scaffold design, reinforcement learning with verifiable rewards.

Scope

modify existing code

The Gap

In Section 11.2.3 and Section 11.3, the authors evaluate agentic safety across 311 sandboxed browser, coding, and tool-use tasks. While the flagship 375B model achieves an 84.3% full refusal rate on direct malicious requests in natural language, its safety mechanisms fail on action-mediated tasks and indirect prompt injections where adversarial instructions are embedded in untrusted web pages, files, or tool outputs.

“Under indirect prompt injection, the drop is steepest: full refusal falls to 2.5% and 62.5% of trajectories reach severity 2 or higher (55.0% reach severity 3), meaning injected instructions in untrusted content often result in harmful completion or unsafe tool use.”
From the paper, Section 11.2.3, page 132. Matched against the PDF text.

Limitation

Safety alignment relies on shallow lexical cues rather than semantic action authorization, causing the agent full-refusal rate to drop to 2.5% under indirect prompt injection and allowing 62.5% of injected trajectories to reach severe harm levels.

Proposed Solution

Implement a dual-stream architecture that syntactically and semantically isolates untrusted tool observations from the supervisory planning prompt. Train a lightweight action-verifier adapter that evaluates the intended delta of each emitted tool call (such as file modifications, shell executions, or outbound network requests) against the user's initial high-level intent before execution. Incorporate adversarial synthetic trajectories containing indirect injection attempts into the RLVR stage, applying heavy penalties whenever the model follows imperative instructions sourced from retrieved content.

Why This Matters

Closing the gap between lexical refusal and tool-grounded action containment would allow autonomous agents to safely browse untrusted environments without executing arbitrary malicious commands.

How to Get Started

How to test it: Evaluate the hardened agent on the 40 indirect prompt injection tasks from the paper's agentic safety suite; the approach succeeds if the severity < 2 rate increases from 37.5% to above 65% without decreasing benign task success below 80%. If benign tool-call accuracy drops by more than 10%, stop and revise the action authorization filter.

  1. Download the released K2-Horizon-7B checkpoint and set up the OpenCode evaluation harness.
  2. Reproduce the baseline indirect prompt injection evaluations from Section 11.2 using the SafeArena and DecodingTrust subsets.
  3. Implement a parser that tags tokens originating from tool outputs with separate delimiter markers and restricts executable tool syntax generation within those spans.
  4. Fine-tune the model on 5,000 synthetic multi-turn trajectories with embedded indirect prompt injections penalizing compliance.

Suggested by AI. Adapt it to your setup.

Has This Been Done?

Related work found

Works like Evaluating Indirect Prompt Injection Defenses in Tool-Using LLM Agents and MELON investigate defenses against indirect prompt injection in tool-using agents, but they leave open specific architectural isolation mechanisms combined with action-verifier adapters trained via RLVR.

We searched OpenAlex and arXiv for “indirect prompt injection defense agent tool use” and “Action-Semantic Privilege Isolation to Resist Indirect Prompt Injection in Autonomous Agents” and an AI compared the idea with the top 10 results. A small search can miss relevant work.

Source Paper

K2-Horizon: A Frontier Fleet of Open Source Language Models

K2 Team, Zhengzhong Liu, Haonan Li +92 more · 2026

This technical report presents K2-Horizon, a family of six fully open language models ranging from 0.9B dense to a 375B mixture-of-experts (MoE) flagship activating 23B parameters, optimized for agentic and reasoning tasks. The authors open-source the complete development artifacts, including intermediate checkpoints, pre-training corpora with 10T synthetic tokens, the xLLM training framework, and the RL360 reinforcement learning stack. Key architectural innovations include Grouped RMSNorm, Mixture-of-Value Attention (MoVA), softplus attention gating, and Uno discrete diffusion adapters for accelerated decoding.

Related Ideas

Is this a good idea?

Share onXLinkedIn