GPU UTILIZATION IS A MISLEADING METRIC

Summary from the TLDR AI newsletter

While commonly monitored, GPU utilization may not accurately reflect true GPU performance in ML workloads, as it doesn't account for whether the GPU's computational power is fully leveraged. Trainy discovered this while only achieving ~20% model FLOPS utilization (MFU) despite 100% GPU utilization during LLM training. It recommends tracking SM efficiency for a better performance indicator and achieving a 4x speedup in training time by optimizing with fused kernels and the right model parallelism level.