BUILDING AND SCALING NOTION'S DATA LAKE
As Notion grew exponentially, it had to build a scalable data lake. Its solution involves incrementally ingesting updated data from Postgres to Kafka, then using Hudi to write to S3 for processing. Spark is used for complex tasks like tree traversal and denormalization. This approach has resulted in cost savings, improved data freshness, and has unlocked new possibilities for AI and search features.