Black Hat 2024 Presentation on AI Agents for Cybersecurity Forensics

The presentation at Black Hat 2024, delivered by a data science lead at Red Canary, focuses on building AI agents for cybersecurity forensics analysis using OSQuery. The speaker details a four-step recipe for agent development: defining narrow goals, preparing and cleaning data, enforcing structured JSON output, and orchestrating multiple agents via Langraph, an open-source durable execution framework. Red Canary processes 350,000 agent calls daily, with some agents conducting over a million investigations, and demonstrates a use case automating Windows endpoint forensics across 39 OSQuery tables. Key techniques include chain-of-thought prompting, tool calling for enrichment (e.g., VirusTotal), and model selection (GPT-4o, GPT-4.1, Sonnet 4, GPT-5) based on speed, cost, and accuracy, with GPT-4o completing a full workflow in 36 seconds for $0.34. The talk emphasizes trust-building through semantic similarity algorithms, LLM-as-a-judge validation, and Yara signatures to ensure consistent, accurate outputs. All code, prompts, and synthetic OSQuery data are open-sourced for community use.