Researchers Unveil AgentFuzz Framework to Detect Vulnerabilities in LLM-Based Agents

Researchers from a security team (including Curry, an AI web security engineer, and Family Lou, a PhD student at Fudan University) presented 'Make Agent Defeat Agent,' a study on detecting prompt-style vulnerabilities in LLM-based agents. Their work introduced AgentFuzz, a fuzzing framework designed to identify critical flaws like remote code execution (RCE) and server-side request forgery (SSRF) in agents by combining static analysis, dynamic validation, and natural language prompt mutation. The tool addresses challenges in traditional fuzzing—such as indirect function calls and semantic prompt generation—by using feedback-driven seed scheduling and sink-guided mutation to trigger vulnerable sinks (e.g., eval). Evaluated on 20 popular open-source agents (each with over 1,000 GitHub stars), AgentFuzz uncovered 34 vulnerabilities, including 23 with assigned CVEs or issue IDs, in high-profile projects like Auto-GPT (180,000+ stars). The approach outperformed baseline tools like LLM-Smith, achieving 33x higher precision and 3x better recall by validating risky code chains dynamically. Key attack vectors demonstrated included prompt injection leading to unchecked tool execution, such as Elasticsearch queries or Jinja2 template rendering. The presentation included a live proof-of-concept showing a reverse shell exploit.