NJEI

Originally published on my blog.

I was facilitating an AI workshop at SEED Inc. One of the women in the room said ChatGPT was her favourite tool, and when I asked why, she didn't say anything about writing emails or summarising documents. She said it understood her. She'd been taking relationship advice from it.

So I asked her the obvious follow-up. If someone had your account open right now, they'd know everything about you — wouldn't they?

She said yes. It took her a second, and it came out awkwardly, and the room got quieter than it had been a minute before.

Nothing she was doing was careless. That's what makes it interesting. She was using the thing exactly as designed, and the design produces a searchable, first-person record of what you're worried about, who's in your life, and how you talk when nobody is editing you.

Social engineering was always mostly homework

The old version of this attack looks like patience. Someone reads your LinkedIn, works out your manager's name, learns that your company says "Concur" and not "the expense system," notices you're travelling next week. Then they call a colleague and sound like they belong.

The call is the easy part. Sounding like you belong is the expensive part, and it used to take days of scraping and guessing.

An AI account skips all of it. It isn't a profile assembled from fragments — it's a briefing you wrote yourself, in order, in your own phrasing, including the things you'd never publish anywhere.

The account isn't the target. The account is the reconnaissance, already finished.

This is already happening, quietly

Group-IB found over 101,000 ChatGPT credentials sitting in infostealer logs on dark web markets between June 2022 and May 2023, peaking at 26,802 in a single month. Their count for the following year was around 225,000 log files. Almost none of that came from breaking into OpenAI. It came from ordinary malware on ordinary laptops, harvesting whatever the browser had saved.

The leaks that involve no attacker at all are just as instructive. In January 2025 a DeepSeek database sat exposed on the open internet with over a million chat logs in it. That same year, people using Meta AI tapped "Share" thinking it meant "save" and published medical questions, custody disputes and confessions to a public feed under their real names. OpenAI pulled its own discoverable-link option in August 2025 after shared conversations started appearing in Google results.

When the Mixpanel incident hit OpenAI users in November 2025, no chat content leaked at all — just names, emails, coarse location. OpenAI still warned people to expect convincing phishing off the back of it. That is how little you need.

Meanwhile the performance half got cheap too. The FBI's May 2025 advisory describes attackers using AI-cloned voices to build rapport, get into someone's personal accounts, and then use what they find inside to go after that person's contacts. The bureau logged more than 22,000 AI-related fraud complaints in 2025, with reported losses above $893 million.

The gateway problem

I'm in Cameroon, and here one Android phone is the whole gateway. Mobile money, WhatsApp, email, the AI app, and the OTP that guards all of them. There's no second device, no work laptop that stays at the office.

That collapses a security model most advice quietly assumes. "Use a separate device for sensitive work" isn't a tradeoff here, it's a non-option. The phone isn't one factor among several — it's the single point where everything either holds or doesn't.

I don't think most people have priced this in. A bank account has a fraud department and a reversal process. A chat history has neither, and it's worth more, because it's the thing that makes the fraud believable in the first place.

What I changed

Small things, none of them clever:

  • A unique password and 2FA on the AI accounts specifically. I'd done this for email and banking years ago and somehow never for the app I talk to most.
  • I stopped pasting anything that identifies someone else — client names, real subdomains, other people's numbers. Placeholders cost me four seconds.
  • Memory off by default, temporary chats for anything I'd mind hearing read back in a stranger's voice. Then I went and read what was already stored, which was a strange half hour.
  • A verification word with the two or three people who could be talked into moving money on my say-so. It sounds ridiculous until you've heard how good the cloned audio is.

What I still don't know is how anyone detects this. A stolen card leaves a transaction. Someone reading two years of your conversations at 3am leaves a login from an unfamiliar city, on a settings page almost nobody opens.

The woman at that workshop wasn't wrong to trust it. She just hadn't been asked the question out loud before, and neither had I.

I write about building software solo, AI tooling, and what technology actually looks like from a secondary African city — at njei-blog.vercel.app.