TLDR Web Dev · MISCELLANEOUS
WHY WORDFREQ WILL NOT BE UPDATED
Summary from the TLDR Web Dev newsletter
wordfreq provided snapshots of language usage up to 2021 - it will not be updated anymore due to several reasons. First, the rise of generative AI has contaminated online data with AI-generated text, making it difficult to accurately represent human language usage. Second, previously free sources of data, like Twitter and Reddit, have become inaccessible or expensive, making it impractical to collect new data. Finally, its developer has become disillusioned with the current state of natural language processing.