SPARK RELEASE 4.1.0

Summary from the TLDR Data newsletter

Spark 4.1.0 adds declarative pipelines, real-time structured streaming mode, Arrow-native UDF and UDTF support, SQL scripting and VARIANT as GA features, and Spark ML on Connect for Python. The release targets lower-latency streaming and a cleaner PySpark and SQL experience while improving caching, memory management, and large-workload stability.