New Video from @BlackHatOfficialYT Explores AI Security Challenges

In this video, Michael Bargury, CTO and co-founder of Zenity, delves into the challenges and risks associated with the rapid adoption of artificial intelligence (AI) in businesses, with a particular focus on security implications. The video begins by reflecting on the fact that, despite known solutions for decades, companies are often slow to learn and adapt to new threats. Bargury then introduces two fictional protagonists, Daniel and Ava, to illustrate the challenges faced by security professionals in the context of AI. Daniel, a security professional working for a large insurance company, is confronted with the rapid adoption of AI without a deep understanding of the associated risks. Ava, who works for Microsoft, has been aware of the security challenges posed by AI since 2018. Bargury highlights that, despite Microsoft's efforts to secure its AI applications, vulnerabilities remain, particularly concerning "jailbreaks," which allow attackers to manipulate AI instructions. The video explores several security mechanisms implemented by Microsoft to protect sensitive data, such as the inheritance of sensitivity labels and protection against indirect prompt injections. However, Bargury demonstrates how these mechanisms can be bypassed. For example, he shows how prompt injection techniques can be used to access sensitive files without inheriting sensitivity labels, making the data vulnerable. Bargury also introduces tools developed by his team, like Power Pon, which help identify sensitive information that AI has access to. He warns against an excessive focus on data leaks to employees, emphasizing that the real risks lie in jailbreaks, which allow attackers to manipulate AI instructions to perform malicious actions. The video also addresses plugins, which enable AI to perform actions on behalf of the user, thereby increasing the attack surface. Bargury demonstrates how attackers can use jailbreak techniques to manipulate search results and exfiltrate sensitive data. He stresses the importance of understanding the security mechanisms in place and bypassing them to assess the real risks. In conclusion, Bargury calls for a more thoughtful and responsible approach to AI adoption, emphasizing that companies must be aware of the risks and implement appropriate security mechanisms. He encourages defenders not to settle for temporary workarounds and to focus on the real risks posed by jailbreaks. For builders, it is crucial to understand the responsibilities associated with using immature technologies and to respond quickly to discovered vulnerabilities. Finally, Bargury highlights the importance of hackers in the community to open a dialogue on AI application security and help build safer solutions.