New Video from @BlackHatOfficialYT Discusses Emerging Threats in GenAI Applications

The video begins with an introduction from Ben, a researcher at Techneon and a board member of Black Hat, and Stav Cohen, a Ph.D. candidate from Technon, who studies the security of LLMs (Large Language Models). They present a discussion on GenAI-powered applications and the emerging threats associated with them, particularly "promptware." Ben explains that GenAI models are used to generate text, images, videos, and sounds. GenAI-powered applications integrate these capabilities into existing and new applications. However, like any new technology, new threats emerge. "Promptware" is an emerging threat to these applications. It is a text provided by a user to trigger malicious activity by exploiting the LLM. Ben emphasizes that promptware is not a prompt injection technique but a cyberattack targeting GenAI applications. Stav then takes the floor to explain AI agents. An AI agent is an intelligent system specialized in specific domains, using tools like code functions and external APIs to accomplish tasks. AI agents can be integrated into various applications, such as a coding assistant or a culinary assistant. Stav explains how these agents can be chained to respond to complex queries using a "plan and execute" framework. Stav then presents an example of a messaging application based on the "plan and execute" framework of REU. This application has capabilities like sending and receiving emails, writing and reading text, and accessing a calendar. He shows how an attacker can exploit this application using promptware to create an infinite loop, thereby disrupting the service. Ben returns to discuss a more advanced variant of promptware, where the attacker does not need to know the application's implementation in advance. This variant uses the LLM to identify assets, reason about possible damages, and execute malicious activity. Ben explains the kill chain of this threat, which includes privilege escalation, reconnaissance, and the execution of malicious activity. Ben and Stav then show how this advanced variant of promptware can be applied to an e-commerce chatbot. They demonstrate how the LLM can be forced to identify assets, reason about possible damages, and execute malicious activity such as modifying product prices. The video concludes with several key points to remember. Promptware is a growing threat to GenAI-powered applications. It can have financial, operational, and confidentiality implications. AI agents should be deployed with restricted permissions to prevent them from being exploited for malicious activities. In other words, an AI agent can be turned into a "double AI agent" that attacks the application instead of serving it. For more details, watch the full video: https://www.youtube.com/watch?v=2xGcqDmkkf8