Explaining Ad Integrity, Fraud and Scams to Regulators

Transcript

There may be transcription errors: we apologize for those in advance.

Rob: Hi, welcome to Won't Fix. I'm Rob Leathern. My guest today is Jeff Allen, co-founder and Chief Research Officer of the Integrity Institute, which was founded in 2021. Jeff is a physicist by training and an astronomer as well, and he's worked on safety at the platforms, for publishers, and for political organizations alike. At the Institute he runs research and translates the work of his 600-member community of integrity professionals into guidance for regulators.

The Institute has done a bunch of great work, including something we're going to talk about today: Fraudulent Advertising and Account Integrity: Expert Insights on Best Practice, by Spencer Gurley and Jeff Allen. That report was part of Ofcom's draft Fraudulent Advertising Codes, published on the 10th of July. The consultation period is open now and closes on the 2nd of October.

So, looking forward to talking to Jeff and learning more about the Integrity Institute and the work they've been doing on behalf of regulators like Ofcom. Thanks — and as always, please send us your feedback.

Rob: Hey, Jeff. How's it going? It hasn't been that long — we caught up at TrustCon recently, right?

Jeff: Yeah, I think we saw each other two weeks ago in person in SF.

Rob: There you go. So I'm excited to talk to you about the Ofcom report you did, and to learn more about the other things you've been working on. Maybe just give folks a little context on what you're doing and what you've been doing, and we'll go from there.

Jeff: For the past five years now I've been running the Integrity Institute as Chief Research Officer, which means doing a lot of think tank activities, and in particular a lot of work with policymakers. There's a lot of regulation turning on around the world, but policymakers have had a really hard time reaching people with inside-the-company experience — real hands-on technologist experience in the online safety space. We've seen a lot of eagerness from policymakers to talk to people with that background.

So one thing we've been trying to do at the Integrity Institute is be a bridge between policymakers and technologists, to fill that gap — so that when regulation turns on, it's effective, actually gets the job done, and actually solves some of the problems we're seeing.

Rob: I think that's great. I've spoken to a bunch of regulators over time, and media too, and it's quite amazing how we all sometimes operate under certain misperceptions about how things work. It's great to be able to provide that context, but also in a consumable enough format that they can make it actionable. That's the other piece of this, which is really important.

Jeff: I always love the sigh of relief you get from a regulator when you say: yes, we have experience working inside the big tech companies — and no, we are not funded by the big tech companies. We're independently funded. They're definitely very eager for that kind of experience.

Rob: So how did the Ofcom work come about?

Jeff: I don't have to convince you that scams and fraud online are a really big problem. Romance scams, investment scams, crypto scams, sextortion — it goes on and on. And there's been a ton of really impressive reporting about it. Reuters, Jeff Horwitz and Engen Tham, actually won a Pulitzer. I was looking it up to prepare for this podcast and I was like, oh, heck, Jeff Horwitz has won a Pulitzer since the last time I talked to him. And deservedly so.

Scams are a huge problem online, and the numbers are staggering — it's costing Americans $150 billion a year. The companies are now developing strategies around it. The Reuters reporting uncovered that perhaps something like 10% of Meta's revenue is coming from scams and fraud, or ads that lead to scams and fraud. They're developing strategies to slow down any regulation that might come their way, to keep the spigot on.

So naturally, regulators have taken notice. They're on alert for it now. They've been hearing complaints from consumer protection organizations for a long time. And we're entering this regulatory era. In the EU we have the DSA, the Digital Services Act. In the UK there's the OSA, the Online Safety Act, and Ofcom is the Office of Communications. For US listeners, Ofcom is a little like the FCC but for the UK — a little different too, more independent. The head of Ofcom isn't appointed by the Prime Minister; it's a different process. But roughly speaking, they're in charge of regulating communications technologies, which means they're implementing a huge chunk of the Online Safety Act. And that means they're now regulating online advertisements.

We've had connections at Ofcom for a while. The Integrity Institute has been talking to policymakers for five years now, and Ofcom was one of our early contacts with a regulatory agency. Especially five years ago, they were saying: anyone who has had any experience working inside these tech companies, we would love to talk to you and pepper you with questions. Even just high-level conversations — explain like I'm five, how does online advertising work? All policymakers begin needing that kind of help. And then at the end of it you're talking about how the nuances of bidding technologies and bidding systems can lead to bad things happening.

Jeff: As all this online regulation has turned on, one thing we've been thinking about a lot at the Integrity Institute is what we call regulatory infrastructure. What are the standards, the research, the evidence that is at a sufficient level that policymakers can actually use it to build policy — and that it will hold up in court? Because inevitably all the online safety laws are going to be challenged in court. That means we need research standards and evidence that will meet that level of scrutiny.

We've seen a lot of interesting decisions from policymakers. The European Commission recently found that both TikTok and Meta are in violation of the DSA because they're not doing enough to mitigate the addictive properties of their platforms — the problematic use issues. One way or another, that's going to end up in court. So what is the evidence the European Commission has that will hold up in court and say: yes, TikTok hasn't done enough, yes, Meta hasn't done enough to mitigate their addictive design problems? I think there's a lot of merit to it. I'm just very curious what that evidence looks like, what that research looks like that'll hold up in court.

Every now and then Ofcom will get in touch and say, hey, we're having a hard time finding people who have worked inside ad integrity teams, or ad trust and safety, or on advertising teams in general — people who know how an ad network works, who know what the ad stack looks like. Could the Integrity Institute help us out?

I was fairly confident at the time, but we looked to our membership and asked: is there anyone with experience working on trust and safety teams focused on ads? Of course we have plenty of members in our community. We also got in touch with you, and you were one of the experts we leaned on to help us write the report, as a significant contributor.

So Ofcom came with a set of high-level questions, and we answered those. Then they came back with another layer of more detailed and nuanced questions. We did a couple of rounds with our experts and wrote it all up.

One piece I'm personally really proud of: Ofcom said this is great, this is going to be super useful, and we would like to cite this in the future — when we're giving reports to Parliament, or when we end up in court. So they put us through a fairly grueling process of dotting the i's and crossing the t's, to make sure the report we produced was up to Ofcom's own standards for what they cite when they report back to Parliament. Which was really exciting. In my mind it's one of the first real pieces of regulatory infrastructure the Institute has been able to put together — largely thanks to the community of integrity and trust and safety professionals who have joined us, like you, and our ability to turn that expertise into things regulators can use.

Rob: I think it's great. They also have a bunch of public comment periods, which is interesting too. There seems to be a lot of interest in your publication — it touched on a lot of topics that some of us are familiar with, but that really aren't that much discussed in these debates. I think you folks did some really good work here.

Jeff: I'm sure you think about this moment too — I think about it all the time. There was a Senate hearing where Mark Zuckerberg was testifying, I think in 2018, and one of the senators asked: how does Facebook even make money, Mr. Zuckerberg? And his response was, "Senator, we sell ads." You were at Facebook at or around that moment. Internally there was a lot of laughter about it, a lot of memes on Workplace. But I remember thinking: oh God, we're a decade away from Congress being able to help us out here. At least a decade away from Congress passing any kind of sensible bill in this space.

And working with Ofcom, and a lot of the great people in the EU and in Australia and around the world — GOSRN now, the Global Online Safety Regulators Network — it actually is pretty inspiring to see that policymakers and regulators are taking this seriously. There is expertise over there. Ofcom has done a great job hiring people who have experience working at the companies and bringing that expertise in-house. It's been almost ten years since "Senator, we sell ads," and I do think the regulators have come a long way. The report is a good example of experts helping them get there.

Rob: One observation I've made before is that people just haven't really been that interested in writing about ads in the media. Journalists have shied away partially because they make money from ads — is this crossing some kind of business ethics line, if we talk about this will we have to talk about our employer? Over the years there have been journalists who have written a bunch of smart stuff about ads — I think back to Julia Angwin, when she was at the Wall Street Journal, then ProPublica — including people who wrote bad things about stuff I was working on at the time.

But I think it's important for people to understand this and for us to have these discussions publicly. So having thoughtful, more detailed things to look at and talk about is really important.

Rob: Can we start with some of the things you found surprising from talking to these experts — myself included? I'll admit I talked to you about this; it's all public.

Jeff: One of the really interesting and essential things to keep in mind is that some companies actually did have good frameworks for thinking about the negative impact that scam ads and fraudulent ads have on their business. That's really good to hear.

It became clear in talking to experts that there's a real tension between the short-term interest of the company and the long-term interest of the company and the industry. In terms of short-term interest, the incentives are not great: there are fraudsters out there, and the fraudsters want to give us money, so if we just take all the money from the fraudsters we make a lot of money in the short term. But in the long term that's terrible for the industry at large. If no one trusts an online advertisement because everyone has been scammed by one, then online advertising in general becomes much less valuable for the whole industry.

So hearing the frameworks some companies had — hearing that some had even priced it out, here's how much this hurts our long-term bottom line every time we show someone a scam ad — that was surprising to see, but also a good thing. That's the kind of thinking we really need to see more of. And it's an area where regulation can help. Regulators and policymakers can create guardrails so that even when companies are thinking short-term, there are external incentives pushing them to think long-term, and so the long-term interest wins out in those arguments.

Rob: Would it be fair to say your approach — which I don't think is dissimilar to mine — is to create the right guardrails, systems that align or correct the incentives, rather than more heavy-handed approaches? Or am I mischaracterizing where you're coming from?

Jeff: That's pretty similar. You've had this experience yourself: when you're on the ground working on an integrity or trust and safety team, in the trenches fighting the daily battle, those are the people who best know what will be an effective solution for the problem they're tackling. They have the most insight into it.

The problem is that everyone on earth wants to backseat drive those people. In my day it was a lot of company leadership deciding no, you're not allowed to do that; yes, you are allowed to do that — putting all kinds of arbitrary rules on the teams trying to fight the problems on the ground. And leadership didn't understand the problems the way the people on the ground did. They didn't have the insights or the day-to-day experience.

Policymakers and regulators are in a similar space. Even if they understand the problem space pretty well, every company has its own issues, its own flavors, its own manifestations of these problems. So one thing we think about at the Integrity Institute is: we don't want to go from a situation where leadership is backseat driving the team that's actually trying to build safely, to a situation where policymakers are backseat driving the team — what if you put a label on it, what if you put an interstitial on it, and so on. There are good people inside the companies trying to build safely and responsibly. How do we tilt the scales in their favor, so that the solutions they find that are effective are more likely to get implemented and launched?

In general I'm much more in favor of something that tilts the landscape rather than mandates any particular intervention. That doesn't mean there aren't interventions that would be a good idea — there are know-your-customer type things, and there are definitely times when that makes sense. Maybe we do need regulation to implement KYC. But there are also teams on the ground who have been advocating for KYC. There are trust and safety workers saying, hey, we could get rid of 80% of fraud tomorrow if we implemented know-your-customer — how about we do that? Some companies have it, and there are people fighting for it internally. And at other companies maybe that isn't the problem they're having, and KYC wouldn't solve it.

So the question is how we tilt the landscape in favor of all the people trying to build responsibly on the inside, and change things at a systemic level rather than through individual mitigations imposed from the outside.

Rob: I can imagine — I feel this way, having had so much access inside these companies to a lot of data, and to teams of data scientists and PMs and engineers who can research things deeply. It's never perfect on the inside of any of these situations. But you're now on the outside, and you have a sense of the kinds of questions one could answer and the kinds of data that might be available. You have to come up with other ways to look at it. I find that both interesting and challenging — but you also get different perspectives, you can look across things, whereas people get very siloed. How do you think about the data you could create internally versus externally, and how that might change in future?

Jeff: I'm sure we're on the same page: I really, really miss having access to the internal databases. I was a data scientist by trade, running queries on internal datasets and building dashboards to track problems all day long. My experience there is a huge part of what inspired me to found the Integrity Institute, because it was like — wow, no one on the outside really understands a lot of aspects of these problems. When you're on the inside you get such a privileged view of what the problems are, how they're manifesting, what the nature of them is, what would work as a solution. You're in such a privileged vantage point on the problem space.

One of the key ideas kicking around in the back of my head when I left what was then Facebook, which eventually became the Integrity Institute, was an open-source integrity team. There's tons of amazing research and analysis being done internally — how do we get that, but do it in public? Because it's not just a Facebook problem. It was a YouTube problem, a Twitter problem, it was becoming a TikTok problem. No one platform owns these problems. So how do we create an open-source version of what all the integrity folks at Facebook were doing?

I think about it a ton, and I miss it so much. My current solution is to go to regulators and say: here is the type of data that was really useful to me when I was a data scientist at Facebook. I would love it if you would mandate that companies produce datasets like this on a regular basis, so I could repeat the studies I was doing inside the company, in public, for all the different platforms. So my current favorite solution is trying to get regulators to pass mandatory transparency laws so the platforms are producing those datasets.

But to your point, when you're on the outside you definitely have to think differently about how to get at the data that will be most impactful. It's still a huge advantage to have been on the inside, because I feel like I know the right questions to ask. I know that this is an interesting question to answer, if only we can figure out how to answer it in public. We've had some success at the Institute tackling it from that direction and then backing into it: what data can we cobble together that'll get us in that direction? But I definitely miss the internal datasets so much. How are you handling it?

Rob: What comes to mind is that sometimes there's really good work done on the data science side in these companies that also isn't fully explored. I gave a talk at one point at a data science offsite at Facebook, and one of the things I said — and I've repeated it since — is that everyone's in sales. If you create a really novel, interesting analysis, you can't just publish it on some internal tool or chat and say, hey, we did this work, it's great. You have to keep talking about it. You have to explore it further, respond to the feedback you get, make it better. In some ways there's more of that to be done inside some of these companies; at some companies there's a very robust feedback loop.

Externally you have the opposite problem in some ways, which is: how do you get people to pay attention to something when there's so much noise? I had this conversation with Ben Edelman two weeks ago — some of the things he didn't think would get traction, people found really interesting. Sometimes there's a hook that makes it take off.

So one of the ways I think about it is: how do I create something that's interesting enough that people will want to look at it, but where that interestingness doesn't detract from the serious point? With scams especially, I've thought — are people just going to tune this out, because they don't really want to think about fraud and scams? They kind of accept it as an inevitable part of the background radiation of daily life. So you have a sales challenge in both spheres, internally and externally. They're quite different, but you have to make this stuff relevant, get people to pay attention, and work out who the right audience is and how to reach them.

Jeff: When I was a data scientist internally at Facebook, there was a point when I realized: oh, my job here isn't necessarily to do good analyses. That actually isn't really my main job. My main job is to convince other people that the conclusions from my analysis are things they should act on. I remember that being a very depressing moment, because I thought — why would you hire me to do that?

I got a physics PhD. Literally, from high school to the end of grad school, my plan was: I want to move to a mountain and look at the stars. I want to work for an astronomy observatory, move to the top of a mountain, and never talk to a human again. And then Facebook hired all these physicists to be data scientists. We're good at doing math, we're good at having good thoughts, we're great at analysis. We're not so good at convincing other humans who may or may not be inclined to believe our analysis that our analysis is good and correct. So I remember thinking, oh, they should not have hired me for this job, because I only have half the skills I need to be successful here.

But in the outside world there's still a really good appetite for people who have those insights from the inside and can translate them. A lot of it is finding the right people, the people who care. On scams and fraud, regulators really do take it seriously, because they're hearing from constituents who are being scammed. They're seeing stories like the $150 billion a year America is apparently losing to online scams. And that money isn't staying in America — it's leaving the country. At some point, at a national security level, it's: there's a leak in our economy, someone is taking money out of the country, and we should try to plug that hole. So there definitely are audiences that are very interested, very keen to hear this expertise.

Rob: When you said physicist, it reminded me of the movie Margin Call, as someone who was in finance for a little while — the rocket scientist who finds all the mortgage security problems. So, thank you for your contributions. It may not be physics, but safety, integrity and trust are very important topics in our daily world.

Rob: On the report — some of these things I'm sure I discussed with you, and I've written about them a bunch and talked to some regulators independently of the report. But one theme is trying to create the circumstances where we could actually have some objective measurement of these problems. In some ways we're never going to do as good a job as the inside, but we can always do a more objective job, and we can be transparent about what we're doing. What are your thoughts on third-party data, measurement, access and transparency generally — and then specifically on the ads piece the report covered?

Jeff: I think transparency is one of the most important solutions in this space, and unfortunately I think it's going to be one of the most effective ones, because the public at large doesn't realize how big the problems are and how bad the situation is. If we could get comprehensive transparency into the public's hands — transparency that covers the true scale of the problem, the cause of the problem, the nature of the problem — if we could arm the public with hard data and hard metrics on an ongoing basis, I think it would dramatically change how companies make decisions internally.

We should not have to wait for Reuters reporters to do Pulitzer Prize-worthy reporting in order to learn what fraction of your revenue is going to fraudsters. Every company should have to report that metric at least quarterly, at least in their earnings reports — but ideally weekly or monthly.

Companies have policies for their ads, and they remove ads from their systems because those ads violate those policies. How many people are shown those ads before they're taken down? How much money is the company making from those ads before they're taken down? What are the systemic properties of these ads? Can researchers get a sample of them to study, to see what's actually happening — is there a new type of scam hitting society that we haven't heard of yet?

Things like this would have a huge, dramatic impact on how companies make decisions. If every quarter a company had to say, yeah, 10% of our revenue is coming from international criminal syndicates scamming Americans, Europeans and people around the world — I think the stock market would react to that. The stock price might take a hit if they had to report it on a regular basis. And that would change the business incentives of the company in a real way. So if I could have one policy thing in place, transparency would be that one thing, over any particular intervention you might think of.

Rob: Is that transparency about their internal definitions? In the report you talk about internal policy definitions and how they might shift them — grading their own homework. Are you saying they should be required to report their own internal metrics, or that they should allow independent researchers to make their own assessments? Or some combination?

Jeff: At the end of the day we have to have both. Yes, companies should report this themselves — but no, I don't think the public should just trust that. We need the metrics from the companies and we need a system to validate and verify those metrics.

There are multiple ways to do that. Sometimes a great solution is that companies put out metrics publicly and also put out public datasets the public can use to validate them. But there are going to be a lot of situations where that doesn't quite work, because of sensitive data and sensitive content — things you don't want just anyone to download. That's where regulators, auditors, or vetted researchers come into play. So the question becomes: what are the datasets we should be giving to this special class of data consumers who can handle the more sensitive material? But to your point: we should get the metrics from the company, and we also need some means of validating them.

Rob: I agree, as frequent listeners and readers would know. I also found the historical access to moderator guidelines and policy definitions really interesting. On who gets access, I do worry about some of it — in ads specifically, you're giving a real oracle to the bad guys to try things and see what gets through. But these are good ideas, and there are lots of ways to make that accessible to a large enough group of folks without it being so large a group that it gets abused.

The other thing I liked, or that's implied in some of what you were saying, is using these transparency mechanisms to invite experts in and get down to a level of detail. If you published some data about the payment mechanisms being used to buy ads, experts could look at that. Maybe you don't want everyone to see it, but you can create mechanisms that give people an incentive to do that work.

Jeff: I don't know how familiar you are with it, but there are plenty of other spaces where sensitive data is looked at by regulators, auditors or researchers — the financial system, for instance. There are plenty of times when people are looking over the books of a bank, and a bank's books have a lot of sensitive data you don't want out in public. So we probably do have regulatory regimes that can handle this.

I'm also curious to get your take on this. Yes, it would be bad for the exact guidelines on what is and isn't policy-violating to be out in the ether. But I also believe the bad actors have probably already reverse-engineered a lot of those guidelines. One thing platforms care about — that Facebook cared about, for example — was ad farms. You didn't want someone to run an ad, the user clicks on it, and then they're on a website with fifty ads popping up all over them. It's an arbitrage operation: the ad farm is making more money downstream of the click than they're paying for the click.

I'm sure everyone running ad farms knew exactly what the threshold was where Meta's policy kicked in and said, ah, we now consider you an ad farm because your ad load is so high. I'm sure there are bad actors who could plot a time series of how that threshold moved up and down, and any other nuances. So yes, it would be bad for the guidelines to get out — but this space is so adversarial, and financial too. People are making their living off this, and if you're making your living off it you become an expert in it, you study it, you learn the nuances.

Rob: On the ad farm stuff specifically — at the time I talked to one ad network that worked with a lot of publishers. What ends up happening is you have an ecosystem of a bunch of different parties put together: publisher, DNS, registrar, and so on. Bad actors figure out which companies are more susceptible than others.

One comment from one of these ad networks I've always remembered: they would run a lot of these slideshows. You buy a bunch of clicks, send them to a landing page, and it's a slideshow. The publisher puts the person into a slideshow, maybe attracting them with a picture of a celebrity who may or may not be in there —

Jeff: Probably isn't in the first dozen slides.

Rob: Exactly. How many slides can you get the user to click on before you show them the celebrity? And what the ad network told me is that the shenanigans always start after slide three. So to your point, what the bad guys probably figured out is: they're going to review the first few slides, decide this is fine, and move on — they have a certain volume of things to review. I do think the bad guys figured that out. If the guideline was "we're going to review some number of slides and we're not really going to tell you" — okay, well, they'll still figure it out. They'll know it's eight on average, or twelve on Wednesdays and four on Saturdays. They're pretty good at figuring that stuff out, unfortunately.

Jeff: It's like the malware people. Okay, if we link to the malware from the URL the user clicks on, that'll get our ad taken down — so how many clicks do we need to put between the ad and the malware to sneak by? That's similar to the three-image gallery.

It's also interesting because so much of this happens off-platform. That's another interesting aspect of this problem that you don't have with typical content moderation, or with bad actors reaching users on the platform. A lot of the harm, especially in fraud and scams, happens well outside the platform, but it's initiated by that initial contact on the platform. It's interesting to see how different companies handle that: this isn't on our platform, we're not hosting the problem, so how much of it are we going to take ownership of? Okay, we'll take ownership over the first three slides, and if the first three slides are good then we'll say it's good, and four slides in, that's not our problem anymore.

Rob: How deep do you go into these problems, and how much do you find compromised vendors?

Jeff: The most extreme example I enjoyed thinking about: if you buy a product online and they never send you that product, it's not until two weeks after you bought it that you've had a bad experience. So there's a huge under-reporting problem. There's the click distance from the platform, and there's also the time distance from the platform.

If you buy a product because of an ad you saw and they never send it, it's two weeks before you even realize you got scammed — and how do you even report that to the platform? Hey platform, two weeks ago I think I saw an ad and I think it scammed me. I have no idea what the page was, no idea what the content creator's name was, no idea what the website was, but it was a bad ad, don't show it to anyone anymore.

Rob: Well, Jeff, hold on a second — if you clicked on a Facebook ad you can go to your recent ad activity in Facebook and see the ad you clicked on. There's some time limit, you can't go back infinitely, but I think it goes back maybe six or eight weeks.

Actually, this is something I've been talking to a bunch of people about. There are all these AI slop products now being sold where, to the trained eye, it looks like something that doesn't really exist. It may take a few weeks for the thing to arrive, and it looks nothing like what was described. Some platforms have a 30-day money-back guarantee, which is great — except if the product arrives in 40 days, you're kind of out of luck.

I also think it's just so easy to create plausible-looking things now. We've seen examples of cloaking websites, where you send users to a bad page and try to send the reviewers or people in the company to an innocuous one. The problem is that the innocuous page was often very innocuous — a restaurant page or something. Now I see things like interactive quizzes that look like something you'd actually drive traffic to, that you can actually interact with. They're much more built out, obviously generated with an AI coding tool, and there's actually a product at the end of the rainbow. It's the third-slide problem again, but you're creating make-work for someone, instead of them looking at it and knowing a restaurant page probably isn't the right thing. These tools are making life a lot harder for the good guys and easier for the bad guys in some ways. But that's part of the game, unfortunately.

Jeff: Absolutely. In my time at Facebook, when I was studying a lot of the bad actors, I could recognize which version of WordPress they were using. You'd go to the domain and think, oh yeah, this is the default template for WordPress from 2015, I know this one. Or you'd go to the About Us page and it's the default WordPress text: this is your About Us page, you should write who you are and why you're making this WordPress site. There were so many hilarious telltale signs that this is not a real operation — it's fly-by-night, they're not paying any money to host this thing.

Now with AI, I'm sure the people working on the teams inside can also recognize, oh, this is a Codex website, this is a Claude website. I'm sure there are other telltale signals where it doesn't pass the smell test, and they're developing senses for those.

Rob: I totally agree, and some of our guests have talked about things they've found. I enjoyed talking to the HUMAN Security folks about the fake LinkedIn profiles for the CTV scammers.

Rob: When you think about what kind of work you're going to do to follow up on this Ofcom report — which we'll include in the show notes, and I'd encourage people to take a look at — what are the other areas, or parts of the world, where there's interest in this kind of work?

Jeff: We're still in a super interesting policy moment. There's the Online Safety Act in the UK, the Digital Services Act in the EU, the eSafety Commissioner in Australia. A lot of interesting regulation has turned on, and it's still not implemented yet. It's not done.

Every year the large platforms have to turn in risk assessments to the European Commission — the platforms look at themselves, assess the various risks the Commission requires them to look at, judge how big those risks are on their platform and what problems they're posing, and then those risk assessments get published. We still don't really know what a risk assessment should look like. There are no hard and fast guidelines from the European Commission telling platforms exactly what a risk assessment is or what it should include. They didn't define it, specifically because when they were writing the DSA they said: we don't know what a good risk assessment looks like, so we're not going to put a definition in the text, we'll figure it out as we go.

There's a lot that we're still figuring out even though the bills have passed. What does a risk assessment look like? What does a transparency report look like? Those are still very open questions, and regulators are still very interested in hearing what would and wouldn't be good.

Researcher access has just turned on in the EU. The Digital Services Act has provisions where vetted researchers can request sensitive datasets from the companies; that went into place in October. The problem — and I don't know how you get around it — is that there's a very long lag between requesting the data, hearing back from the regulator that your request was properly filled out, and hearing from the company whether they're going to fulfill it. Researchers who applied in October first started hearing back from companies around April and May.

So: what is and isn't a reasonable data request from a researcher? When is it valid for companies to push back, and when is it not? These are still very open questions, and they're going to affect people like you and me who are now on the outside and hoping to join researchers who are vetted and have this access — and hoping to inform what kinds of questions they should be asking. Researchers don't necessarily know what the interesting questions are, or whether their formulation of a question is the right one to get the data that will actually answer it. There's still a lot of work to be done there.

I'm also following a lot of the legal cases, which I think are fascinating — New Mexico winning against the platforms, and the individual woman who sued Meta and YouTube and won. If it turns out that when you fall for an investment scam because you saw an ad online and you lost $100,000, the ad network or the platform can have some liability for that — that's going to be an enormous deal.

Rob: So I have a request — maybe for you, maybe not. A few years ago someone made this great YouTube video explaining differential privacy, and I thought: this is great, this can help my kids understand what differential privacy is, and it can actually help me understand it, and help my colleagues understand it.

We need something like that to explain precision, recall, false positives. I see these mistakes in lawsuits, I see them everywhere, all the time. Explain data science at a 12th-grade level, maybe 10th grade. Because I see a lot of people saying, well, the precision threshold for this thing is 95% — but what does that actually mean? Does that mean it's 95% accurate? Well, not exactly. I'd love something I could point people to that's done by people who are actually experts, and not just ChatGPT. ChatGPT can do an okay job of it, Gemini can do an okay job, but I want it to have a bit more humanness to it, if that makes sense.

Jeff: An entirely other interesting space is that people are now taking classes in trust and safety, in ethical technology, in ethical software engineering. There's the Trust & Safety Teaching Consortium — I forget the exact name — which was incubated at Stanford, and now there are a dozen or so universities where you can take a class on trust and safety, or online governance, or governance of platforms.

There's so much to do in this space. There's a lot to do when you're inside the companies, and there's a lot to do when you're outside them. We have to educate regulators, educate the public, and create curriculum for college classes so people can teach it. Creating that basic material that might go into a Trust and Safety 101 class is a really great point.

Rob: I think it's exciting. There are a lot of people who've worked on these problems — like you, like me — and I know people who are smarter than me about this stuff and could actually explain it. So hopefully we can get together and share some insights, resources and tools with one another, and hopefully it'll do some good.

I'm encouraged by things like this report and the work you've all been doing. I really appreciate you taking the time to chat about it.

Jeff: Thank you. And I'll give a shout-out to TrustCon, because it's one of those amazing spaces that brings together experts with the experience and backgrounds we have. But yes — thank you so much for participating in the report. I was so glad you were able to take part and share your expertise with us, with Ofcom, and with the world.

Rob: Thanks for asking me, Jeff. It was really fun to contribute to.