CRAWL ENTIRE WEBSITES WITH A SINGLE API CALL USING BROWSER RENDERING

Summary from the TLDR newsletter

Cloudflare's Browser Rendering API renders pages in a headless browser and returns the data in multiple formats, including HTML, Markdown, and structured JSON. It is now available in open beta for both the Workers Free and Paid plans. Users can configure crawl depth, page limits, and wildcard patterns to include or exclude specific URL paths. The crawler honors robots.txt directives.