TLDR AI · ENGINEERING & RESOURCES
1300 TOKENS PER SECOND ON MAC
Summary from the TLDR AI newsletter
Implementing batch parallel KV cache in MLX leads to dramatic inference time speed up for synthetic data generation and model completions.
TLDR AI · ENGINEERING & RESOURCES
Implementing batch parallel KV cache in MLX leads to dramatic inference time speed up for synthetic data generation and model completions.