HOW TO JOINTLY TUNE LR AND WD IN ADAMW

Summary from the TLDR AI newsletter

AdamW is commonly used to decouple learning rate and weight decay. However, the common implementation in Pytorch doesn’t explicitly do this. This post discusses how to tune these parameters.