ATLAS: EVALUATING AGENTS ON SEARCH-INTENSIVE TASKS

Summary from the TLDR AI newsletter

ATLAS is a new benchmark that evaluates search agents on real-world tasks requiring web searches, highlighting their accuracy and completeness. The benchmark reveals that even high-effort agents miss significant golden answers, indicating major room for improvement in search engines. ATLAS differs from existing benchmarks by focusing on non-memorized tasks, requiring extensive multi-domain searches, and offering a cost-effective grading process.