Talking AI Deception with ex-Meta/Google/Elevenlabs Investigator

Transcript

There may be transcription errors: we apologize for those in advance.

Rob: Welcome to Won't Fix, where we have conversations about AI-driven deception, abuse, and scams, and why they're so hard to stop.

This is a good one — we get into some of the practical details of scam fighting. I talk with Assaf Kipnis. He's a former Meta e-crime investigator, ex-Google Trust & Safety, and the founder of KTLYST Labs. We get into why AI isn't really changing scams so much as scaling them, why adversaries collaborate while defenders don't, and why the number of accounts taken down is possibly the most useless metric in trust and safety. Plus, what actually works when you stop playing whack-a-mole.

One unexpected thing we talked about: why bad metrics happen to these teams, and why sometimes the goal of a trust and safety team isn't to stop the bad guys — it's just to make the platform annoying enough to abuse that they go to someone else's platform instead.

I think you'll enjoy this one. And please, as always, send us your feedback.

Rob: Good to see you, Assaf. How are you?

Assaf: You too.

Rob: How and where did we meet? I'm trying to remember.

Assaf: I was on the e-crime team at Facebook, and you were in ads integrity, I think.

Rob: Business integrity — most of which was ads integrity, yes.

Assaf: Right. And I started working on pages and things like that, which is how we got more into each other's domains.

Rob: That's right. So, e-crime — that's a pretty cool name for a team. Is that something people can talk about, or have talked about, externally?

Assaf: I don't think so. I don't think people really talk about the Facebook teams. At least when I was there, there was a larger investigative org that was the tip of the spear on threat intelligence, and one of the teams inside it was e-crime, which I was on. We specifically went after financially motivated actors. Not child safety, not espionage — specifically actors trying to make money off Facebook users.

Rob: Well, every advertiser we worked with was trying to make money off Facebook users. But your part was where they were using very devious schemes, or misrepresenting who they were, or things of that nature.

Assaf: Yes. Back then there were scams, but they were less sophisticated than they are now. And there was abuse of Facebook and Instagram products in order to make money — down to literally every product they could abuse. Groups, pages, jobs, everything.

Rob: And what have you done since then? You left Meta, as I did, and went to Google. Where else have you been the last few years?

Assaf: After Meta I was at Google for a year and a half, working in Ads Trust & Safety, specifically on scams. After that I spent a very short time at ElevenLabs, a voice AI company, running their intelligence and investigations org. And then I went out on my own. I started a consulting company where I help with threat intelligence and AI deployments, and I started a startup, more in the security space, working on operationalizing threat intelligence.

Rob: That's super relevant background for the kinds of things I'm trying to talk about. I talk to a lot of people, and I think you've said this to me before — people think everything is totally different now because of generative AI and deepfakes. Some of it is new, but a lot of it isn't. Is that fair? How do you think about what's genuinely new in scams versus what's the same stuff in a different cover?

Assaf: When I think about abusive behavior geared toward generating money, it's always been about making the user believe something. Believe that you have a product, believe that you're doing something, believe in the narrative. AI right now is just adding another layer of authenticity.

In the past, one of the easiest things people would do was crop an image out of another article, put it in a new article, and that's your authenticity. Now you don't need to do any of that. You don't need to write anything of your own. You can make videos. So that last step — making people believe, making it sticky — AI made a lot more attainable.

Rob: And the other thing that strikes me is that it should make things cheaper. It should let more people get into the business of being a bad actor if they want to, just because it's more attainable. That's what I've always worried about — script kiddies, or whatever the old term was. Anyone with a browser can access some pretty powerful tools now.

Assaf: They can. But I think there's still a lot of methodology that's tribal knowledge inside that community — sharing playbooks, sharing how you orchestrate the plot.

From a defensive perspective, or if you're not really in the know, you think, oh, they just put a piece of content out there and make money off it. With AI, yes, that's easier to create and easier to create at scale. But those tools still don't get you past the guardrails companies have — stopping bad content, catching you when you're not where you say you are, the general defenses.

So yes, you can have a script kiddie doing this and making a little money, which I see more as a nuisance. What's happening with the larger organizations — and they're basically businesses doing this constantly — is that they've been at it a while. They have really good mechanisms. They have really good evasion. And they don't have a mandate to deploy AI everywhere, so they deploy it if and when they need it. My assumption is they deploy it at the tip of the spear — how does this reach a person — and in making more content, faster.

Rob: You alluded to playbooks. So is it the case that the bad guys share information with each other? I used to tell people we needed to break down our mistakes — that we needed deep dives when things got through — because I'd heard bad actors collaborated. They'd get on Skype and talk to each other, back when Skype still existed. Is that true, that they collaborate a lot?

Assaf: It's very true. There's the collaborative aspect of just sharing stuff. They have conferences, which is very weird. They have Telegram channels where they share information. It's two parts, really: there's the sharing industry and there's the tooling industry. I'll make a tool, it works really well, and I'll sell it to you. It's the same as an exploit tool. I'll just sell it.

And you're kind of alluding to the whole point of the company I created, KTLYST Labs. The idea from the beginning is that adversaries talk to each other. They share knowledge. If one person learns something, they'll go around and say, hey, I learned this, change everything you're doing because I learned about it. Meanwhile, we as companies don't talk to each other at all. We don't even talk to each other within the same organization. When one team learns something from an investigation, no other team finds out and deploys it in their space.

My background — I always say threat intelligence, but really it's thinking like an adversary and understanding how they operate. And they love to share, while we absolutely hate sharing.

Rob: I was actually talking to someone about this recently. As ads and other online content moved from images to video, some of the tooling companies used — ours and others — would grab frames out of videos and put them in front of reviewers, or feed them to machine learning models. And at first we didn't randomize the gaps between the keyframes we were grabbing.

It turns out multiple companies had this happen and didn't realize it at the time, and didn't tell each other: randomize your frame sampling, because otherwise the bad guys will just put good images in the keyframes, since they know which ones you're going to grab. That could have been a very simple fix ahead of time, if we'd known — if we'd actually talked to each other about it.

Assaf: And you know what? I bet in one of those companies, one of the threat intel teams knew about it.

Rob: I'm sure. And there's still a lot of relationships and anecdotal, ad hoc sharing that happens. But I think you're right — it just isn't happening even within companies to the extent that it should.

Assaf: We operate in silos because that's what organizational structures create. Adversaries don't have silos, and they know our infrastructure better than we do. It was always, yep, they figured out how this product works with this product and with this product — three completely different teams, one of the products built ten years ago and who knows how it works. They do.

Rob: So tell us a bit more about what you're working on now with KTLYST Labs. Who are you working with, and how's it going?

Assaf: Right now I'm working with a friend of mine who also spent time at Meta, and before that at McAfee. We're building a product that essentially lets teams use the intelligence they generate.

I'm looking at it more in the cybersecurity space right now. If threat intel gets information, how does that information immediately become something the hunting team can go look for? Or something the red team can run an exercise on? And then those two teams produce reports — so how do those reports turn back into intelligence you can push out to all the other teams?

At Meta the question was always: how do we become proactive? My idea of proactive is sharing information so that we're ahead of the adversaries and can understand their journey and cut them down earlier — before they attack us and we block it.

Rob: So essentially you're getting to build the things you always wanted to exist. Talking to a lot of people who worked at these companies, sometimes it's "I'd like to recreate the tool we had, because it worked really well." But there's always an "and wouldn't it have been nice if we also had this thing, and that thing."

Assaf: That's exactly it. I see people recreating tools like Maltego and doing it really well, and a lot of investigative tools. When you move from one company to another you always have that itch of, oh, I really liked that product, let me make it here.

But having the ability to walk away from that and ask, okay, what's the actual problem? What's the core of the problem in the system? That's what let me create something that isn't anywhere. It just doesn't exist.

Rob: As someone who is an investigator — taking one step back — there are a lot of discussions where people look at something and say it's going to replace this thing, and therefore that whole job category doesn't make sense anymore. But there's always higher-level work, at least in my estimation. I'm curious how you see these tools — language models, agents — replacing some of this work, or making it higher-leverage. What do you see as the impact?

Assaf: I've seen a lot of discussion about how AI can't replace creative writing, can't be creative at the level of a human. I find that to be true, and I see it everywhere.

Even in an investigation. I have an investigative bot I built for myself. It does open source investigations. It goes to a bunch of places and finds things I didn't even think about. But then it comes back and says, okay, how do we synthesize this? Or it gives me its current idea of what's going on. Without my expertise and my creativity, what it gives me is very high level — not generic, because I built it not to be generic, but high level. It can't think like an adversary the way I do. It can't ask the questions. It answers the questions you bring it. It'll postulate a few more, answer those, and try to make sense of it.

Which is great, because an investigation that would have taken me a week now takes two hours. I don't have to spread my mind thin going to all these websites, collecting things out of URLs, finding more websites and more tools to look at. It does all of that. The busy work of finding where to look, identifying and copying or scraping pieces of information — that's gone. And that makes me a much more effective investigator, much faster, and lets me make much more educated decisions.

But I still need to make those decisions. I don't believe right now — and I don't know if it will ever happen — that AI is going to make these really granular calls, drawing on experience it just doesn't have.

Rob: One of the interesting things in the discussion about AI taking jobs is what happens at the entry level of any profession — how do people actually get in and get that experience? So I'm curious about two things. One, how did you get into this line of work originally? And two, what do you tell people who find this interesting or have some aptitude for it?

Assaf: The way I started down this route: I was about 29. It was after I was in the military and I wanted to get a degree. I thought I was going to get a computer science degree, and then I went to Stevens Institute of Technology and they told me about this newish degree they had — cybersecurity. They started talking to me about it and I thought, this is very reminiscent of stuff I did in the military.

In the Israeli military I wasn't in a tech unit — I was in artillery, doing a bunch of things. But as an officer I did a lot of border defense. So it really resonated with me: how to protect perimeters, how to understand attackers.

So I did that. And then, by chance, I got an opportunity to go into a SOC at VMware, where I worked the swing shift, 4 p.m. to midnight. I found that I really enjoyed finding the attacker. I didn't enjoy writing scripts or writing detections — I enjoyed finding them. From there I rolled into a role at LinkedIn, more in the threat intel space, doing investigations. And I found I really had what they call the investigative mindset: I'm seeing something, this is weird, let's go dig and find what's going on, and how they're doing it, and how the products are built around what they're doing. That's how it rolled into where I am now, learning from each of those roles.

For new people starting out, I'll say it's a little scary, because it starts with: if you're interested in investigations, where do you go? Trust and safety? The entry roles there are usually more customer-support-type work. A threat intel vendor? That might be more interesting. Security? So that's difficult.

And on top of that there's this whole layer of assumption that we're going to take away all the lower-grade jobs, automate operations, and so on. Yes, you can automate a lot of things. But that creates another problem: if you don't have low-level people, nobody ever grows into something else.

It's funny, because people try to put what I'm doing into that bucket — automating security operations. And my answer is always no, no, no. We're not automating anything. You still need to make the decisions. You still need someone who sits there and first builds the tool for themselves — there's the tool for the company, but then you augment it to what you do and what you need. And then you need the judgment calls: what am I stopping? What am I blocking? How do I keep moving forward and look for the threat actor's next hop?

When I talk about this with people, a lot of them say, well, threat actors are going to use AI and we're going to use AI and it'll just be AI versus AI. Which is a very lazy argument, because it means nothing. What I see, both in the industry and with threat actors, is that people who understand how to use AI use it — they don't replace things with it. You take the people who do the job and you give them superpowers. Instead of being bogged down writing a document — which people like me struggle with; I remember getting 70% into an investigative document and going, I'm done, here we go, it's fine — I do that now in a minute. Or trying to figure out which threat intel feed actually has the information you need. You can aggregate things a lot faster.

You're still the linchpin making the decisions. But instead of replacing your lower-level people, you teach them to use it really well to augment what they're doing. They produce a lot more, and honestly they're happier and less burnt out by the menial work that we all know low-level analyst work involves.

Rob: That makes a lot of sense. So I started working on online ads about 22 years ago. My first ads job was at a company called NexTag, a comparison shopping company, and they started doing lead generation ads. We had an affiliate program — which I'd never do again, because affiliate programs for lead gen are very, very problematic. People can generate fake information.

We were using a platform called Be Free, which was later acquired by Commission Junction. And I remember getting an email from the CEO, who obsessively looked at all the dashboards over the weekend, saying we'd suddenly gotten all these leads from the affiliate program and I needed to look into it immediately.

So I looked. Be Free had this feature where an affiliate could change their name, their URL, their payment information. We went back and found a bunch of affiliates we'd approved who had applied in the names of various companies — we were basically auto-approving everyone — and one of the names one of these affiliates had recently applied under was one of our competitors, which should never have been approved. And we noticed that all of the ones with leads had changed their addresses to a bunch of P.O. boxes in Syracuse, New York.

So we thought, okay, this is actually pretty easy to find. They're all in the same city, and their names were changed. We shut the whole program down. We weren't paying out in net two days, so we looked at it and didn't pay them. But that was one of my first experiences of: oh, there are these folks who figure out the edges of a thing, and then all of a sudden they attack it over a weekend. If you have something broken, it can get taken advantage of very quickly.

Assaf: That's exactly how it works. They find a hole, and then all of a sudden everybody's using it. Everybody's found this loophole. All right, this works, let's use it. And they use it fast, because the assumption is that you'll plug it eventually.

Rob: I've seen many examples of that, most of which I probably can't talk about. Are there any you can talk about, or obfuscate, that would be interesting and informative for people to hear?

Assaf: I remember working with an operations team. There were signals saying, this plus this equals bad, look into it. Very operational: you do this, then you do this, and you approve or disapprove. Their job was to sit there with a hammer and say, nope, this is bad, take that down, take that down.

They came to us and said, we're noticing the same actors, but they're circumventing us. They've figured out how we're stopping them and now they're doing something else. We kind of understand what's going on, but as an operations team, we don't have the policy to do anything. We can only follow the policy. We can't investigate it.

So we investigated, and we found a group of actors — not one specific actor, but a cluster doing the same thing — spreading misinformation for profit. They were all using a specific tactic that worked until something in operations changed, and then they moved to this new tactic. Mostly it came down to whether they were located in the US or not.

What we identified was that they were able to manipulate certain assets and use those assets to propagate the information. So we asked: what's different about these assets? What we noticed was that the new assets didn't have a location at all.

To put it simply — sorry if it's a little obfuscated — if I want to advertise in the United States, and you say that if you're located in another country you can't advertise in the United States, then I say, fine, I'll use a different asset. And nothing stops me, because that asset doesn't have a location. It doesn't say I'm in the United States or anywhere else.

So we investigated how they were doing it, which assets they were using, and the scale of it. First we were able to identify it. Second, there was a specific process that let us put the fix into production immediately. We found that you could be the admin of that asset without it affecting the asset's location — so the product changed. From then on, the asset takes on your location. If you're in a different country, now the asset is in that country. And then there was a policy change: if you see these assets running this misinformation from another country, you shut them down.

What was really different about how we did that as an organization — not just my team, my team only investigated and found which pieces went together — was seeing the policy change, the product change, and the operations change together. They had been really using that gap, using it to promote as much as they could. Once we closed it, it didn't matter what they did next.

A lot of the time when people find bad actors, it's: well, just remove all these accounts and we win. And maybe we'll send law enforcement. What I'm saying is the opposite. You can do that. But if you change the policy, change the product, and change the operations, then any new account, any new asset they create gets shut down immediately.

What happened is that within a week those actors moved to a different platform — because we didn't just shut down their accounts, we shut down their pipeline for using accounts. That's the thing I always talk to people about: how working cross-functionally can be extremely useful, and specifically how not to come at this from the myopic view of just taking down bad accounts.

Rob: There's always a temptation to fix the immediate problem instead of the systemic one. The other thing — I was talking to someone about this yesterday — when X/Twitter started showing account locations, you saw that certain accounts you'd expect to be US-based were posting from another country. They were saying, wow, this is great for trust and safety. And I said, look, there's a big difference between an unexpected product change and what it exposes, versus how people adapt their behavior afterward.

And frankly, sometimes the people quickest to adapt are the worst ones. If you add friction, sometimes you're just adding friction for everyone who's good, because the bad guys figure out a way around it. So you have to think about this stuff not just as a steady state, or an initial state, or an initial response — you have to think about the counter to the response, and then the counter to that.

Another place this came up recently was Australia, and what's happened since they required kids off social media. Sometimes when you don't hear anything about a policy, it's because everyone has figured out ways around it. That's what I heard from some Australian friends — you don't hear about it anymore, not because the kids are happy to be off social media, but because they found a way to stay on by some other means.

Assaf: From my jaded perspective, both of those were mostly a show. With the Twitter one, I remember people on LinkedIn and Facebook going, oh my God, Twitter did this thing. And in my mind — they're doing what every other social platform has been doing for a decade. So first of all, that was just bull. And second, how hard is it for me to use a VPN to change my location? I wasn't expecting it, so for a moment I got caught with my pants down. But people have very short memories. Oh, yeah — now I'm in America.

I was actually having something of an argument with someone in Australia who was touting that legislation. I don't understand it. What are you doing here? Yes, you're removing the people with the fewest resources and the least know-how to bypass it. But it's so easily bypassable that to me it's just a show. It's a piece of paper that says, hey, look, we did a thing — while literally everybody can get around it extremely easily.

Rob: Exactly. That's the discussion when people say things like, we need to verify all the advertisers. There's value there, and it's a discussion worth having. But when you expect these things to be magical solutions to your problems, you're going to be disappointed, because you're making things harder for a bunch of ordinary people. You're making things harder for the bad guys initially, but they're going to figure out the workarounds — and because the financial incentives are there, they'll come up with workarounds the average person never would.

I don't know if you ever saw the stories — slightly different topic — about people hiring homeless people to set up merchant accounts to accept payments. The kinds of people who would do that, unfortunately there are many of them, and they'll do all kinds of things to keep doing what they're doing.

Assaf: That's a very interesting perspective to me, because I just don't think that way.

We always have this idea of block lists and allow lists. There was one thing at Meta where actors kept creating a logo that looked like Meta's, and there was this consistent effort to keep finding that logo. And it's like — dude, they just need to change one pixel. Just one. Because it's a hash.

And I agree with you. They don't just have the incentive to make money, they have the incentive not to change the way they do business. They have a process. What you did was take one piece of their very robust process and mess it up. So for a minute the process no longer works, and their whole focus becomes: crap, how do I get from the point before you broke it to the point after, so I can continue my process?

Meanwhile, companies are celebrating. Someone very senior once said to me, "But I thought we stopped it." No. You're not going to stop crime. We're not stopping crime here. We're making crime expensive and difficult to do. But there's this perspective that the one thing we're going to do will solve it.

It reminds me of the COVID conversation about Swiss cheese. A mask isn't going to stop it. Staying home isn't going to stop it. The vaccine isn't going to stop it. But if you do all of them together, the chances drop a lot. Identifying the images, blocking some accounts, blocking IP addresses — each one is a layer. But there's a very strong focus on finding the one thing that makes it all stop, which is why companies continuously chase their tail.

Rob: So to that point — is the best case for companies to reduce the amount of time their assets are abused, and push the abuse to someone else's platform? Or is that too cynical a view of what success against scams looks like?

Assaf: The job is always to make them go somewhere else. The job is literally: get off my platform.

Until companies get their act together legally and otherwise, and are able to collaborate for real — not the signal sharing I'm seeing, which is mostly pomp and circumstance, but actually collaborating against threat actors — the job is to make my platform inhospitable. Go do something else. We would always say: go to Twitter, man. Leave me alone. Go to Parler. Don't be here.

It's very cynical and very jaded. But in the end, if you're sitting inside a platform, all you can do is make your platform the worst place to be if you're a threat actor.

Rob: Is there any way for countries or states to take that approach — to make it harder or more expensive for threat actors to do business in their country or state? Like, California does something, and now scammers are less likely to scam Californians and go scam Texans instead. Is that a real thing?

Assaf: That's deluding yourself too — although if you're a government, and your residents are being victimized, that's different, because you have a responsibility. Especially at the federal level.

The way I think about this is that regulations only have teeth if they make someone hurt, if they make the company hurt. We saw this with security. Nobody cared about security until several regulations came in and said, if this happens to you, you're going to pay a lot of money. And in security I feel like it was applied really well. Now you usually won't see a company come up without a security team. Is it a show or not? Doesn't matter — you won't see a company without one.

But the regulation I'm seeing right now on scams is very, very high level. Here's where it's not high level: in banking, in fraud, banks have a huge incentive for their customers not to get scammed, because they get hit with everything.

With social media companies and others, it's different, for a few reasons. One, regulators, for some reason, are very happy when a large company gives them a nice presentation about what they're going to do. Cool, you did what you needed to do. Is it really happening? Nobody knows. So that's one thing I'd expect from regulators: get people who know what's going on and go figure out whether the thing was actually deployed, and how. These companies are obviously going to do everything to avoid a fine worth 10% of their profits. But you need to verify — actually go in and see how it's being done, and understand whether your regulation is even deployable. You might have created a regulation this company can't implement. They'll tell you they're doing it.

The other difficulty, especially at large companies, is that actors spread their exploitation chain across multiple companies, and the actual money exchange happens somewhere else entirely. So each of these companies can rightfully say: this is not happening on my platform. A piece of it is. And there's no real recourse from a government perspective to say, that piece is a problem — and as a government, we can put all the pieces together and say that together, you are all culpable in this. Maybe that's the thing that would make companies look at it more deeply.

Although even as I say it, I don't think holding companies' feet to the fire in that way is all that helpful, because in the end they're mechanisms to make money. That's what they're here for. If they don't make money, they don't exist. So they'll have bad actors on them.

I don't know if I got anywhere with that argument, but it's such a complex one. It's not happening on my platform, it's happening across multiple platforms, I'm trying. And then there's growth — growth trumps everything, because that's how it is. But I do think government regulation could be a lot better here, and less performative: okay, what is actually happening on these platforms? Not trying to find out whether Zuckerberg is the devil. I don't care. Zuckerberg is trying to make money. Is he a nice guy? I don't know. But work with the platform and figure it out, instead of saying, hey, go do that and it'll all be better, so I can say I won.

Rob: The best I can come up with is transparency — giving some third party the ability to look at a subset of things, make a judgment, and share that judgment with the public.

The example I use is the Insurance Institute for Highway Safety. In the 1960s they started testing vehicles independently of the manufacturers. Insurance companies had a financial incentive to make sure cars were safe. Over time the manufacturers participated more and more in that testing — oh, you tested this, actually we should try this, how about this. And eventually it became part of some of those companies' marketing: look how safe our car is.

So I do think there's something there for some subset of these safety and security problems. But it takes a lot of work and thought to figure out the right set of things to do and how to get people to participate — because there's this tendency to think we can just regulate it into being fixed, which I'm not sure is true. You can regulate people into doing some work.

Assaf: I want to say something and see how it connects. The thing I constantly saw — and I'm sure you did too — is that every organization doing this has metrics. And the most prevalent metric is: how many assets did we take down?

In the age of AI that's a completely useless metric. It means nothing. It always meant nothing. Now it means even less.

I had a conversation with someone at one of the large companies about scams, and their framing was basically: if you see this thing, don't act — directed at the victim. And I said, that's what we're doing? We're telling victims not to click, instead of saying, here's what we're going to do to make this stop.

The answer, which really made me mad, was: well, I would love to have a SWAT team that goes and finds them and takes them down, but we can't do that. So in their mind you can block, or tell people not to click — and the only other option is to go arrest them. The fact that people who have been on these platforms for so long only see those two options is learned uselessness. Learned futility. Well, if I can't block, there's nothing I can do.

There's so much to do. There's so much more that can be done. You're just held hostage by metrics that mean nothing.

Rob: I think that's a great way to end it — unfortunately, on a semi-cynical note. But coming up with the right metrics and counter-metrics is something we should talk a lot more about.

Assaf: Yeah, I have feelings about metrics. I don't know if it comes across.

Rob: We're going to have to talk more about that one. Assaf, thank you for chatting with me. This was really fun, and I'm looking forward to many future discussions about this and similar topics.

Assaf: Same here. Thanks, Rob.