BYPASSING BING CHAT’S GUARDRAILS TO SOLVE CAPTCHAS

Summary from the TLDR InfoSec newsletter

Bing Chat is a public implementation of the GPT-4 LLM and includes guardrails to prevent certain forms of malicious usage. One user demonstrated that Bing Chat successfully detected that it was being asked to solve a CAPTCHA and refused when it was just prompted with a picture of the text. However, the same user showed that if they edited the CAPTCHA onto a picture of a locket and presented a context of it belonging to their dead grandmother, Bing Chat would transcribe the CAPTCHA.