Artificial Intelligence and Phishing Threats

These days, artificial intelligence (AI) seems to be everywhere—it’s in our phones, cars, and even in things like washing machines. I saw one recently that had built-in AI, and it felt a bit futuristic, honestly. Still, there’s no denying how much AI has simplified our lives.

It helps boost productivity and unlocks new creative avenues. Think about the chatbots you might have come across, like ChatGPT, which is popular nowadays. But despite these advancements, AI isn’t free from issues.

If you’re using Google Workspace, you might have encountered Gemini, Google’s AI model that’s integrated into apps like Docs, Sheets, and Gmail. Unfortunately, researchers have found that malicious actors can exploit this feature by manipulating email summaries to inject deceptive phishing prompts.

How AI is Aiding Hackers

Researchers from Mozilla’s 0Din discovered a gap in Google’s Gemini that lets attackers insert hidden instructions into email summaries. This method, showcased by Marco Figueroa, uses a tactic where subtle, invisible commands are embedded in an email’s body. When Gemini summarizes these messages, it unwittingly incorporates these covert prompts.

The attack doesn’t rely on shady links or attachments. Instead, it cleverly uses a mix of HTML and CSS, rendering certain elements invisibly by setting the font size to zero and using white text. These hidden commands may not show up in Gmail’s standard view but are still read by Gemini. Thus, when users request a summary, they risk receiving fake security alerts or urgent messages that appear to be from Google.

In one demonstration, Gemini falsely alerted users that their Gmail passwords had been compromised, even providing a fake support number. Given that Gemini is integrated into Google Workspace, many might instinctively trust that information, making this tactic particularly dangerous.

Google’s Response to Security Flaws

Google started implementing rapid infusion protection back in 2024, but it seems this strategy can be bypassed. The company stated that updating safeguards is an ongoing process.

A Google spokesperson commented, “Combatting attacks like rapid injection is a priority for us, and we have implemented various defenses to guard against harmful or misleading reactions.” They also confirmed that they haven’t observed any active exploitation of this specific technique.

Staying Safe from AI Phishing Scams

So what can you do to shield yourself from phishing scams using tools like Gemini? Here are six practical steps to consider:

1. Don’t Automatically Trust AI-Generated Content

Seeing a summary in Gmail doesn’t mean it’s safe. Be cautious with AI-generated suggestions, alerts, or links. Always verify key information, like security alerts, with trustworthy sources.

2. Skip the Summary Feature for Suspicious Emails

If you suspect an email isn’t legitimate or if it’s from an unknown sender, avoid using the AI Summary feature. Instead, read the full original email. This helps prevent falling for misleading summaries.

3. Be Wary of Urgent Emails and Messages

Watch out for emails that create a sense of urgency or prompt you to check account details. Attackers often use AI to craft convincing alerts for sensitive information requests. Always take a moment to scrutinize suspicious messages before acting.

To protect your devices from malicious links, ensure you have antivirus software installed. This can safeguard your personal data and warn you about phishing attempts.

4. Keep Your Apps and Extensions Updated

Regularly update Google Workspace and your web browser. Google frequently releases security updates to stave off new threats. Additionally, avoid unofficial extensions that interact with Gmail or Docs.

5. Consider Data Removal Services

AI phishing attempts often stem from stolen personal information, which could come from data breaches or the details you share online. Data deletion services can help by scanning and requesting the removal of your data from broker sites. While they can’t erase everything, reducing your digital footprint makes it harder for attackers to tailor phishing scams.

6. Disable AI Summaries for Now

If you’re concerned about AI-generated phishing, think about disabling Gmail’s Gemini summary feature until stronger protections are in place. You can still read emails the traditional way, which limits risks associated with manipulated summaries.

How to Disable Gemini on Desktop

• Open Gmail on your desktop.
• Click the Gear icon in the top right.
• Select See all settings.
• Scroll to “Google Workspace Smart Features”.
• Toggle off Google Workspace Smart Features.
• Click Save.

How to Disable Gemini on Mobile

On iPhone:

• Open the Gemini app.
• Tap your Profile photo.
• Select Gemini Apps Activities.
• At the top, tap Erase.

On Android:

• Open the Gmail app.
• Tap the menu icon (three horizontal lines) in the top left.
• Scroll down and tap Settings.
• Select your Email account.
• Scroll down to Google Workspace Smart Features and uncheck the box to disable.

Key Takeaways

This flaw serves as a reminder that phishing tactics are evolving alongside AI. Now, attackers are focusing on manipulating messages instead of relying solely on visible red flags. This trend raises questions about our comfort level in trusting AI to summarize or filter emails. It’s a compelling, if unsettling, aspect of how technology is changing.

How do you feel about relying on AI for email summaries? Share your thoughts.