ChatGPhish: How ChatGPT Turned Into a Phishing Machine
I’ve been saying it for months now: the way we use AI tools is creating security holes we don’t even see yet. This week brought another proof point, and it’s not some theoretical risk buried in a research paper. It’s a phishing bug called ChatGPhish, and it affects the way users get answers from web summaries inside ChatGPT.
Here’s what happened. A vulnerability was discovered in how ChatGPT renders summarized web pages, specifically in its Markdown conversion process. When ChatGPT gives you a quick summary of a web page, it strips away a lot of the original HTML and replaces it with its own condensed format. That summarization is useful, but it also means cues you rely on to judge a link’s trustworthiness, like the real URL or page branding, can get lost or misrepresented. Attackers figured out they could upload or reference malicious pages that, when summarized by ChatGPT, would produce phishing content that looked like legitimate information delivered by the AI itself.
Why does this matter? Because most users have learned to trust AI outputs as filtered, cleaner versions of the web. If the AI gives you a link, you’re less likely to scrutinise it than if you opened a random web page yourself. That trust is exactly what ChatGPhish targets. The attack surface isn’t a single company server anymore. It’s every AI-assisted search, every research assistant, every tool that condenses the web on your behalf.
For Australian businesses, this should raise specific alarms. Many teams rely on AI assistants to speed up research, draft client summaries, or scan partner websites. If the AI silently redirects intended destinations to phishing pages, one bad summary becomes a compromise entry point. Traditional security controls often don’t cover this path because the AI acts as an intermediary between the user and the actual web request.
The practical takeaway: treat AI-generated links the same way you would treat any link from an untrusted source. Hover over them whenever possible. Where you can, require human verification before users act on AI-supplied content. Better yet, segment how front-line staff access AI summarisation tools, especially when those tools pull from external web sources.
What’s most frustrating here is the cycle keeps repeating. First cloud apps, then email, then messaging platforms, then AI assistants. Every new convenience layer becomes a new exploitation layer. The attackers aren’t inventing new techniques as fast as we’re creating new surfaces for them to attack.
Trust in the tool is not a security control. The moment AI summarisation becomes a default path for users, it becomes a default target for phishers.
Related Reading:
– https://philiphall.com/ai-safety-net-holes-security-2026
– https://philiphall.com/ai-dethroned-stolen-passwords-hackers-verizon-2026
– https://philiphall.com/ai-just-broke-the-19-year-record-heres-what-it-means-for-your-business
Related Reading
- 300,000 ChatGPT Accounts Got Hacked Last Year. Here’s What It Means for Your Business.
- Scammers Are Now Hosting Malware on chatgpt.com. Yes, the Real One.
- Cyrillic Phishing Attacks: How Homograph Domain Spoofing Works and How Businesses Can Defend
The views expressed on this site are my own and do not represent those of any current or former employer. Articles are based on publicly available information and are provided for general educational purposes.
