DECOHERENCE

Summary from the TLDR newsletter

Decoherence is the accumulation of locally valid but mutually incompatible or redundant models within a system. It causes the cost of change to be dominated by translating among those models and preventing unintended side-effects. Decoherence grows relative to the rate of change of a system, and it increases faster the more decoherent a system is, resulting in runaway exponential growth. It rules out the specter of runaway recursive self-improvement, as decoherence both prevents AI from improving in a bubble and limits the rate at which it can improve outside of it. However, it also introduces a new kind of disaster called Decoherence Collapse.