I WAS WRONG ABOUT ROBOTS.TXT

Summary from the TLDR Web Dev newsletter

This dev initially blocked all web crawlers using robots.txt to protect his data, but this inadvertently broke link previews on his LinkedIn posts. LinkedIn's bot couldn't access the necessary Open Graph meta tags to generate previews because it was blocked by the robots.txt file. To fix this, he updated his robots.txt to specifically allow the LinkedInBot to crawl his website.