Gmail’s Gemini-Powered Summaries Can Be Tricked by Hidden Prompts

There’s no denying that GenAI tools help us save time and handle tasks efficiently. One such example is Gmail’s AI-powered email summarization feature, which is powered by Gemini. It saves time by condensing long messages into quick summaries. But the question is, can it be trusted to detect or avoid phishing?

The attackers can use basic HTML to game Gmail’s Gemini-powered summaries

A newly discovered flaw in Gmail’s Gemini-powered summaries could be used to trick users into falling for phishing scams. According to Marco Figueroa (via Bleeping Computer), attackers can embed hidden instructions in emails that Gemini will follow when generating a summary.

In simple terms, the attacker hides prompts within the email body using basic HTML and CSS. By setting the font size to zero or changing the text color to white, the message becomes invisible to the user but remains readable by Gemini. As a result, when the user asks Gemini to summarize the email, these hidden prompts are included in the summary.

Figueroa also shows where an email contained a concealed message warning the user that their Gmail password had been compromised. The Gemini summary included this warning along with a phone number for “support” which actually led to a fake helpline. It’s a straightforward trick that takes advantage of users’ trust in AI-generated content. Since it doesn’t rely on links or attachments, it’s also more likely to slip past spam filters and land directly in the inbox.

While many might notice something off, there’s always a risk for those who rely heavily on AI summaries. A quick glance at a Gemini-generated message could be all it takes to fall for the scam.

Google told Bleeping Computer that it is aware of the issue and is working on mitigation strategies. “We are constantly hardening our already robust defenses through red-teaming exercises that train our models to defend against these types of adversarial attacks.”

Do you use Gemini-powered email summaries in the Gmail app?