Black Hat Presentation Reveals 0% Detection Rate by SAST Tools in New Security Benchmark

The presentation at Black Hat introduces a safe code detection benchmark developed by researchers, including Irina (a bachelor student at Masaryk University) and Andrew, to evaluate Static Application Security Testing (SAST) tools. Their benchmark, consisting of 33 realistic vulnerability examples across five subcategories (source, authentication, authorization, cardinality, and normalization confusions), revealed a 0% detection rate by existing SAST tools, despite OWASP benchmarks claiming 80-85% true positive rates. The team demonstrated three case studies—coupon reuse, negative tipping, and cross-tenant access—highlighting how modern web development practices (e.g., refactoring, abstraction) obscure vulnerabilities from SAST’s taint analysis. While AI tools like GitHub Copilot and Gemini achieved 100% detection in their tests, the presenters cautioned that SAST remains more scalable and deterministic, though current tools lack interactive, developer-friendly custom rule capabilities. They proposed integrating SAST more closely with IDEs (e.g., as language servers) and using custom rules to enforce security invariants, noting that convoluted code often resists both SAST and AI analysis. The talk concluded that SAST vendors must improve framework coverage and rule-writing flexibility to address complex, multi-layer vulnerabilities. November 2022 marked a shift with AI’s emergence, but the benchmark’s saturation limits its generalizability.