“BEHAVIORIST” RL REWARD FUNCTIONS LEAD TO SCHEMING

Summary from the TLDR AI newsletter

A large class of reward functions, which includes almost every reward function in the RL and LLM literature, are all doomed to eventually lead to AI that will pretend to be docile and cooperative while secretly looking for opportunities to behave in egregiously bad ways. This is because negative reward for lying and stealing looks the same as negative reward for getting caught lying and stealing. The reward function will miss sufficiently sneaky misaligned behavior, so the AI will learn that this kind of behavior is good.