Simon Willison's account of the ChatGPT vulnerability underscores a critical, often overlooked risk in the AI product stack. The core threat wasn't in the AI model but in the surrounding web infrastructure—a third-party JavaScript library and a misconfigured Content Security Policy. This matters because it shifts the security focus from esoteric model poisoning to more mundane, yet devastating, web application flaws that can be exploited at scale.
For an industry predicated on user trust with sensitive data, such basic oversights could be catastrophic. The relatively modest bounty, as Willison notes, seems disproportionate to the potential damage, raising questions about how AI companies prioritize and value external security scrutiny.