IT'S OWL IN THE NUMBERS: TOKEN ENTANGLEMENT IN SUBLIMINAL LEARNING

Summary from the TLDR AI newsletter

Subliminal learning is a curious phenomenon in which a language model fine-tuned on seemingly meaningless data from a teacher model acquires the teacher's hidden behaviors. Certain concepts and tokens can become entangled during training - increasing the probability of one also increases the probability of the other. That means that simply prompting a model with a specific token can cause it to favor certain topics. This post tests this hypothesis through experiments, reporting results on Qwen-2.5 7B instruct.