THE FEATURE THAT WAS 100% CORRELATED WITH THE ANSWER — BECAUSE IT WAS THE ANSWER

Summary from the TLDR Data newsletter

Target encoding, rolling averages, and shifted features can silently leak the answer when computed before a proper split or across mixed entities. The safe workflow is to split first, calculate training-set encodings out of fold, fit other statistics only on training data, use group-aware transforms, and give unseen categories honest fallbacks.