How AI Hacks Humans

A digital marketing specialist opens his inbox. There's a message about one of his Google Ads accounts. A settings change needs his attention. He manages Google Ads, Google Business profiles, and SEO campaigns for a roster of clients, so he gets messages like this all the time. This one referenced his actual work, named the kind of account he manages, and asked him to log in and adjust a setting.

Everything checked out. Except the URL.

The domain in the login link was close. One character off, maybe two. The kind of thing you'd only catch if you were squinting, and why would you squint at something that sounds exactly like your job?

He caught it. Barely.

What Changed

The thing is, the danger was never the crude stuff. The "you've won a prize" emails, the obviously fake invoices. We all learned to spot those. The danger is a message so well-fitted to your actual work that it reads as routine.

This kind of attack used to take real effort. Someone had to research the target, learn their professional vocabulary, and build a pretext that matched his daily workflow. That cost time and talent, which meant attackers mostly went after people worth the investment.

AI collapsed that cost to almost nothing. Researchers at Oxford found that large language models could generate tailored spear phishing for over 600 UK members of Parliament at a fraction of a cent per message.¹ A team at TrendAI went from a LinkedIn profile to a fully customized attack in 30 minutes, and the whole system was built in 24 hours using off-the-shelf tools.²

Six hundred tailored attacks for the price of a large coffee.

And it works. AI-generated phishing emails hit a 54% click-through rate in controlled testing, compared to 12% for the generic kind we've trained ourselves to ignore.³ The AI-gathered intelligence about each target was accurate 88% of the time, and the whole operation performed on par with human experts at about one-thirtieth the cost.³

The Cover Story

But the AI is only half of it. The other half is the platforms we use every day.

Google changes something every week. A new policy notification, a revised settings page, an alert that your account needs attention. If you manage client accounts for a living, you are swimming in legitimate requests to log in and change things. The attacker's message didn't need to be clever. It just needed to sound like one more notification in the pile.

The pile never stops growing. Familiarity is the actual vulnerability here. The constant churn of platform updates trained this specialist to comply with exactly the kind of request the attacker was making. Google's own notification habits provided the cover story for free.

What to Do About It

The FBI puts it plainly: be alert to hyperlinks that may contain misspellings of the actual domain name.⁴ That's the tell that saved the specialist in this scenario. The message was flawless. The URL could not be.

Business email compromise cost $2.77 billion in 2024, and total cybercrime losses hit $16.6 billion, up 33% from 2023.⁵ The attacks driving those numbers don't look like scams. They look like work.

Every year the attacks get smarter. The fix is still a phone call.

So when a message asks you to log in and do something, verify the request through a different channel. Call the sender. Open the site yourself instead of clicking the link. The message can be perfect, but the domain can't fake being real, and neither can a voice on the other end of a phone.

Footnotes

1. LLMs for Spear Phishing: GovAI/Oxford Study ↩

2. From LinkedIn to Tailored Attack in 30 Minutes ↩

3. AI-Supported Spear Phishing Fools More Than 50% of Targets ↩ ↩²

4. FBI IC3 PSA: Business Email Compromise ↩

5. FBI IC3 2024 Report via Abnormal AI ↩