SiftRank: List-Wise Document Ranking Algorithm for Vulnerability Discovery in Large Codebases

Caleb Gross presents SiftRank, a list-wise document ranking algorithm for vulnerability discovery that addresses the attention management problem in large codebases by externalizing attention management from language models. The technique was demonstrated on a WattBox IoT device with a 55-megabyte runtime binary containing 5,700 decompiled functions, successfully identifying a command injection vulnerability in a super admin diagnostic endpoint that was patched in March. SiftRank works by having an LLM reorder batches of code snippets by relevance to a query, then uses statistical analysis including T-tests to identify functions that consistently rank highly across multiple samples, ultimately producing a short list of 3-4 functions on average for manual review. Evaluation on the BinPwn dataset of 94 CVEs across 28 CWEs showed SiftRank outperforms zero-shot classification by over 2x in precision, with smaller models like GPT-4o Nano matching or beating larger models like GPT-4o when using SiftRank. The algorithm also outperformed RankGPT and its permutation self-consistency variants across all measured metrics, with the advantage increasing as corpus size grows and vulnerability prevalence decreases. The tool is open source and applicable beyond vulnerability research to any problem involving large search spaces with fuzzy evaluation criteria and limited attention resources.