MICROSOFT'S GUI-ACTOR ENABLES AI AGENTS TO NAVIGATE SCREENS WITHOUT COORDINATES

Summary from the TLDR AI newsletter

GUI-Actor is a new approach that lets AI agents interact with computer interfaces by using attention mechanisms instead of predicting precise pixel coordinates based on screenshots. The method achieves state-of-the-art results on GUI benchmarks while being more efficient - fine-tuning only 100M parameters while keeping the underlying vision model frozen delivers comparable performance to much larger models.