KAN: KOLMOGOROV-ARNOLD NETWORKS

Summary from the TLDR AI newsletter

Multi-Layer Perceptrons are used widely in AI today, including in the Transformer between Attention layers. However, they use fixed activation functions. This paper suggests using learned activation functions on edges using the Kolmogorov-Arnold representation (functions can be represented by a superposition of simpler functions). In this case, the researchers replace weights with splines. The architecture is much more complicated, but has some interesting properties that potentially make it more interpretable.