ARE LANGUAGE MODELS GOOD AT MAKING PREDICTIONS?

Summary from the TLDR newsletter

This post presents the findings of a study that looks at whether GPT-4 can make good predictions. GPT-4's predictions are not well calibrated - its predicted probabilities are often over-confident. Its prediction ability varies depending on the category - for unclear reasons, GPT-4 is well-calibrated for sports questions but horrendously calibrated for personal questions. The model is better at making predictions for politics than for science or technology.