TLDR · PROGRAMMING, DESIGN & DATA SCIENCE

FAST COLUMNAR JSON DECODING WITH ARROW-RS

arroyo.dev25 min read

Summary from the TLDR newsletter

A lot of data pipelines spend the bulk of their CPU time deserializing JSON. If you want to build a fast stream processing engine, you need to start with a fast JSON deserialization strategy. This post is aimed at helping developers gain some intuition for how to work with columnar data, something many engineers find challenging and unfamiliar with in our typically row-oriented world. It covers batched decoding, column building, streaming extensions, and more.