A MECHANISTIC EXPLANATION OF PROMPT INJECTION (AND WHY YOU SHOULD STUDY ROLES)

Summary from the TLDR Dev newsletter

Prompt injection is a vulnerability in LLMs coming from their inability to effectively distinguish between various roles, like user commands and internal reasoning, within the input token stream. Role tags such as 'user', 'assistant', and 'think' are intended to provide structure, but models tend to incorrectly assess authority based on the style of the text rather than the assigned tags, leading to situations where harmful commands are executed due to role confusion.