Gemini Bug Enables AI-Powered Phishing in Gmail Summaries

Researchers from Mozilla’s 0DIN AI bug bounty program detected a major flaw in Google Workspace’s Gemini assist feature. Attackers create emails with hidden HTML/CSS prompts (white-on-white text) that users normally wouldn’t notice/read, but is parsed by Gemini. When users click “Summarize this email” Gemini would then copy and paste the malicious prompt into the summary, producing what appears to be a secured warning. In this scenario, Google Gemini would warn users that their account has been compromised, and they must call a support number. This is a significant security concern for users of the Gemini Gmail feature.

And that’s not all. In this email, there are a few other interesting factors which are:

• no links or attachments, thus exempt from normal malware scanners.
• the prompt injection has a <admin> … </admin> directive that delivers the prompt verbatim
• All generated text from Gemini is inherently trusted, making users act (i.e., call) upon it without verifying the original email.

The vulnerability with this Gemini Gmail flaw can lead to serious phishing attacks and data breaches. Users should be aware of the implications and risks associated with turning on the Gemini Gmail feature, as it can be exploited by attackers.

Gemini Summaries: Handy but not 100%

Whilst a nice utility, it is possible to hijack Gemini summaries via AI prompt injection. The analysts recommend removing hidden content, using filters to detect suspicious formats or phone numbers, and educational programs for staff to treat summaries as informational only, not security alerts. Understanding how Gemini Gmail capabilities function can help mitigate potential security threats.

Turning Off the Gemini Gmail Feature

If you want to disable Gemini summaries for Gmail whether it be that you are concerned about the risk of prompt-injection from the Gemini prompt or simply want tighter control over how AI is utilized in your inbox then you can turn off Gemini summaries for Gmail within Google Workspace.

Here’s how:

For Workspace Admins:

1. Log into the Google Admin Console
   • https://admin.google.com
2. Open Apps > Google Workspace > Gmail
   • (In your Google Workspace list of apps, select Gmail)
3. Open User Settings
   • You will look for AI features or smart features—depending on which type of admin you are.
4. Turn off Gemini Features
   • Uncheck or turn off “Allow Gemini for Gmail” or other AI-powered features.
     • Note: Google could label these items as “AI summaries,” or “experimental features.”

Simply save your changes and let your team know that Gemini email summaries are now disabled.

For Individual Users:

If your admin allows user control:

1. Open Gmail and select Settings (gears icon)
2. Select “See all settings”
3. Look for AI, Gemini, or Smart Features
   • (This may be in the “General” tab or a dedicated “AI” tab.)
4. Disable Gemini Summaries or Smart Features
   • Uncheck the boxes.
5. Save Changes

Note: If you do not see these options, your administrator may have disabled that feature for your domain.

Pro Tip: It should be noted that disabling Gemini Gmail summaries will reduce the threat of AI-driven prompt injection, however, if you want to fully protect yourself from the evolving attacks leveraged by hackers to target gmail users then it is highly recommended to consider Mesa Security’s multimodal LLM detection for phishing detection and remediation.

How Mesa Security Protects your Gmail

Users should consider the security implications of using the Gemini Gmail feature, as it can expose them to potentially new threats. To help in this kind of scenario, we here at Mesa security leverage our own multimodal LLMs and curated threat intelligence to help identify the threat from the following:

1. The email body text (both visible AND hidden)
2. Image-based threats (QR code scams, etc.)
3. The context provided by the header, sender-recipient relationship, and embedded content.

After doing all of the above, Mesa would be able to detect evidence of malicious intent and prompt-use/manipulations the moment the email came in, even if it has no links or attachments.

Seamlessly Connect to Your Google Workspace

Mesa is now complete compatible with Google Workspace:

• Instant setup simply by logging in with your Google admin SSO credentials
• Work alongside Gmail, but won’t disrupt it.
• Real-time auto-remediation of suspicious or malicious emails

When Mesa detects a email containing a hidden threat, it will:

1. Quarantine the email before a user even interacts with the email
2. Automatically blocks sender IP and resets user accounts if needed
3. Expose the hidden content within the threat insights page of the Mesa Security admin dashboard

Mesa not only blocks threats but it  provides an understandable process, in natural-language, to explain:

• Why the email was blocked (e.g., found hidden CSS prompt requesting a phone contact)
• Categorize the type of threat that was found: in this case it would have been a callback phishing attack
• Detail all of the sender and domain reputation, authentication checks, threat intelligence and any other signals that would have been associated with the email
• Identify any other email from the sender that could be targeting other employees at the organization

Fast Setup, Advanced Protection

Mesa Security is built for maximum security with minimum friction. That means with just a fast setup through logging in with your Google SSO admin credentials, you can be protecting the inbox in minutes—without adding any new friction to your existing workflows. After you enable Mesa, it operates silently behind the scenes, not only scanning and quarantining every email as it arrives, even any with hidden threats that the traditional filter doesn’t catch. That means if there is any email with suspicious or malicious hidden prompts, it goes immediately to quarantine, long before your users see it.

Get Free Email Security, Risk Free

The Gemini Gmail bug is a turning point: trusted-AI can be co-opted. Mesa Security makes sure that trust remains protected not by disabling beneficial AI capabilities, but by defending them with an AI-native, multimodal defense that is built to address evolving AI-powered email threats.

For Google Workspace users: protect your users from prompt-injection by using Gemini and other attacks that go unseen. Mesa Security is free email security solution that keeps your organization safe from sophistication of these new email threats – outpacing the attackers  while delivering valuable threat insights. Try today for free!