ON APPLES, ORANGES, AND CLASSICAL ML VERSUS LLM SECURITY PERFORMANCE

Summary from the TLDR InfoSec newsletter

When classical ML-based security tools are compared against LLM-based security tools, the classical ML tools often have several advantages that skew the results. Classical ML tools are frequently trained on a portion of the dataset and then tested on the same data, whereas LLM tools are tested in a one-shot fashion. The ground truth may contain labeling errors that classical ML models replicate, and classical ML models require continuous retraining to maintain accuracy. Moving forward, we should develop security-focused foundational models to help overcome scalability issues and hallucinations.