HOW WE BUILT BETTER GENAI WITH PROGRAMMATIC DATA DEVELOPMENT

Summary from the TLDR Web Dev newsletter

Snorkel AI used weak supervision to programmatically curate an open source prompt/responses dataset with a 24% increase in response win-rate against ChatGPT. Because the curation was done programmatically, it could be done by two devs in one day as opposed to hundreds of annotators over the course of weeks/months. This article details the problem with manual annotations and the techniques used for dataset filtering and evaluation in a scalable way.