PROGRAMBENCH

Summary from the TLDR AI newsletter

ProgramBench challenges agents to recreate software executables without source code, using only documentation and experimentation. The tasks range from terminal utilities to complex software like compilers and libraries, offering over 248,000 behavioral tests across 200 tasks. Agents must design and implement entirely from scratch in a secure, sandboxed environment, emphasizing software architecture skills without external aids or decompilation.