Gmail users beware: Watch out for this potentially risky feature involving AI technology
In a recent discovery, a security researcher known as "blurrylogic" uncovered a gap in Google's AI system, specifically in the email summarization feature used by the tool Gemini in Gmail. This vulnerability, referred to as "indirect prompt injections," allows fraudsters to hide malicious instructions in emails, often in invisible text formatting, which Gemini reads and follows automatically.
The exploitation of this vulnerability can result in a range of harmful activities, including phishing, data theft, and even the execution of malware. For instance, blurrylogic was able to make Gemini display a false security warning in the email summary, potentially tricking users into providing sensitive information or clicking on fraudulent links.
To combat this risk, Google is implementing a layered security strategy to protect Gmail users. This strategy includes improving the Gemini 2.5 AI model to resist manipulation attempts, employing specialized machine learning models to detect malicious or suspicious instructions embedded in email content, adding system-level protections to limit the potential of executing harmful commands within AI summaries, and continuously monitoring for fake URLs, phone numbers, urgent messages, or behavioural indicators of prompt injection.
Google also advises users and security teams to treat AI-generated summaries as an attack surface, similar to how spam and phishing emails are treated. This includes implementing instrumentation, sandboxing, and continuous monitoring.
While Google has not yet confirmed any real-world abuse of this vulnerability, the proactive measures aim to significantly reduce the risk of prompt-injection phishing attacks leveraging AI summaries in Gmail and across the Google Workspace ecosystem, including Docs, Slides, and Drive.
Google is also working on solutions to address malicious commands to Gemini, such as constantly training the AI model not to execute malicious commands and displaying a yellow banner with a security warning in the future. This warning could include messages like "Warning: Gemini has detected that your Gmail password has been compromised" along with supposed phone numbers or links to help. However, it's important to note that such warnings should be treated with caution, as they may be part of a phishing attempt.
In conclusion, Google's approach to addressing this vulnerability combines AI model improvements, malicious instruction detection, system-level security controls, and continuous vigilance to prevent Gemini from executing hidden malicious commands in email summaries, thereby safeguarding Gmail users from potential phishing attacks. Users are encouraged to exercise caution when using the AI summarization feature and to report any suspicious activities to Google for further investigation.
[1] https://arxiv.org/abs/2203.04512 [2] https://arxiv.org/abs/2203.04513 [3] https://arxiv.org/abs/2203.04514 [4] https://arxiv.org/abs/2203.04515 [5] https://arxiv.org/abs/2203.04516
- The vulnerability in Google's AI system, specifically the email summarization feature of Gemini in Gmail, could potentially lead to financial losses if fraudsters use it to execute malware or conduct phishing and data theft.
- To mitigate the risks associated with this vulnerability, Google is employing technology like artificial-intelligence and cybersecurity measures, such as improving the Gemini 2.5 AI model, using machine learning to detect malicious instructions, and adding system-level protections to limit command execution within AI summaries.
