CYBERSOCEVAL: BENCHMARKING LLMS CAPABILITIES FOR MALWARE ANALYSIS AND THREAT INTELLIGENCE REASONING

Summary from the TLDR InfoSec newsletter

Researchers from Meta and CrowdStrike introduced the open-source CyberSOCEval benchmark to assess large language models (LLMs) on two critical SOC tasks: malware analysis and threat intelligence reasoning. Technical findings show that current models score only ~15-28% on malware analysis and 43-53% on threat-intelligence reasoning, highlighting significant gaps in interpreting JSON logs, mapping attack chains to MITRE ATT&CK, and reasoning about multi-hop adversaries. Security teams should be cautious about off-the-shelf LLM reliance, instead using these benchmarks to validate models, augment human analysts rather than replace them, and prioritize training with domain-specific cyber data.