CAN AI ACTUALLY FIND REAL SECURITY BUGS? TESTING THE NEW WAVE OF AI MODELS

Summary from the TLDR InfoSec newsletter

The author of ai-security-analyzer decided to test the new generation of reasoning LLMs for their ability to detect vulnerabilities in a test project. A comparison of OpenAI o1, o3-mini-high, Gemini 2.0 Flash Thinking and Pro, and Deepseek R1 is presented, with the reasoning models all successfully identifying some issues while Gemini 2.0 Pro mostly identified false positives. The author concluded that the AI models are promising tools but still imperfect and also compared the test against Semgrep and found that they both identified different vulnerabilities