EVALUATING AI AGENTS IN SECURITY OPERATIONS

Summary from the TLDR InfoSec newsletter

Cotool tested an AI agent's ability to solve security operations tasks using the Splunk BOTSv3 dataset and frontier models from Claude, OpenAI, and Gemini. Cotool found that while GPT-5.1 and Opus 4.5 achieved the top accuracy at 63%, Opus was 3x more expensive than GPT-5.1 but also completed tasks in half the wall-clock time of any other model. The Gemini models performed notably poorly on the tasks and failed to complete several tasks.