I'VE PUT A COMPLEX CODEBASE INTO A SINGLE 120K-TOKEN PROMPT, AND ASKED 7 QUESTIONS GPT-4 AND GEMINI 1.5. HERE ARE THE RESULTS!

Summary from the TLDR newsletter

This Reddit user recently put two complex codebases into GPT-4-Turbo-128K and Gemini 1.5 and asked them questions. The codebases implemented a parallel inet runtime, so it involved some hard compiler stuff. Gemini 1.5 completely outperformed GPT-4-Turbo-128K in the task of understanding the codebases. Most of the questions that GPT-4 got wrong are ones it would have answered correctly in a smaller context. Gemini still struggled to create a complete mental model of the system, but it is extremely good at locating existing information, making long-range connections, and doing some limited reasoning on top of this information.