Stop Rebuilding Safety Tools

Transcript

There may be transcription errors: we apologize for those in advance.

Rob: Hello, welcome to Won't Fix. My name is Rob Leathern. My guest, Juliet Shen, has spent more than a decade on one of the hardest, least glamorous jobs on the internet, keeping people safe. She was the very first trust and safety product hire at Snap, where she helped build cross-platform systems to fight child exploitation and went on to lead anti-abuse product teams at Grindr and Google. And along the way, kept running into the same pattern. Every safety team at every company quietly rebuilt the same tools from scratch behind closed doors at enormous cost. So she set out to fix that. Today she's co-founder and head of product at Roost, Robust Open Online Safety Tools, which is a nonprofit building open source safety infrastructure that any platform of any size can use for free. Her team just shipped something called Coop 1.0, the first free open source system for detecting and reporting CSAM end-to-end. I really enjoyed the conversation and hope you will too. As always, please send us your feedback.

Rob: Great. So Juliet Shen, how are you?

Juliet: I'm good. Enjoying the summer. How are you, Rob?

Rob: Doing well. The summer here in Austin has been, it's hot. It's warm. Although the hottest month in Austin is August. So you want to probably stay out of Austin if you can in August.

Juliet: I feel the same about New York. You know, we say New York has five seasons and August is one of them.

Rob: Exactly. Exactly. So you're the head of product and co-founder at Roost. Roost Tools. Do you say Roost or Roost Tools? Or what's the right way to refer to what you're working on?

Juliet: Well, yeah, it's interesting because I try to avoid the chai-tea situation, but we are Roost. So robust open online safety tools.

Rob: And so tell us a little bit about how you got here, what you've been working on. You have a very, very interesting career, obviously very much focused on trust and safety, but tell us a little bit about how you got here and why you're excited about what you're working on.

Juliet: Yeah. So, you know, I think my career has been completely focused on trust and safety, but especially in the beginning, I never thought I was going to work in tech at all. Kind of fell into trust and safety, as so many of us do. But, you know, I've been a moderator, I've been a program manager, became a product manager, did it at Snapchat and built out all their safety infrastructure for five years, experienced the pain of having to use tools that weren't necessarily designed for fantastic user experiences, had design resourcing behind it. And at Snapchat is where I really experienced great internal tools and started building our own as well. Did the same again at Grindr, spent a year at Google about doing the same thing. I think I got the opportunity to join the founding team at Roost and do a bunch of academic research on tools, which was very needed because there was no academic literature on the types of technology and tools that trust and safety teams actually use. Plenty of stuff about policy and operations and platform governance, but how do we actually do it? A little bit unanswered. So yeah, I came to Roost because one, I think that those of us who have built internal tools know oftentimes what a challenge it is and also how important they are. And I also think no one needs to build a scaled review tool for trust and safety ever again. You know, I've built like three now and like they all do the exact same thing. I've built a bunch of rules engines. And I just think there's a bunch of things that need to be in the public commons, both for transparency and visibility, but also just to help teams not start from the bottom of the hill. We're all Sisyphus.

Rob: Yeah, that's a great point. I've done some consulting projects for companies where they built out a bunch of this infrastructure themselves. And actually for them, like this one company I'm thinking of, they actually did a decent, they like built it from first principles. They're like, oh, we get all these things and we need to decide in what order to look at them. And so they built a queuing system and then they're like, oh, well, we want to automate this, but we don't know what is what. So they built a labeling tool. So there's kind of like this, I feel like some of it is just you incrementally figure out the stuff you need. But to your point, you know, there should just be a playbook, right? Or like a series of things that just come, you know, standard with these kinds of things. So, you know, so how many times, how many of those things have you, have you, have you calculated or figured out the number of times you built, you know, insert piece of infrastructure?

Juliet: Well, I've definitely built more than four like admin tools and I've built more than two review tools. You know, when I got to Google, Google already had their complete suite of internal tools. But, you know, I think I talked to so many people who have to build their own things from scratch. And I get it. You know, it's like you said, it's incremental. You don't know your needs until you meet your need and then you build something on top of it. But I always say it's like, for those of us who have been building these tools for a long time, we've touched every hot stove. We know how hot they are, which stoves are hot. And I just don't think everyone needs to keep touching hot stoves for the first time and learning that it'll burn.

Rob: You— I think, I think I've probably built like, I don't know, three or four ad servers. Like, I was in ads for a long time, so I've— everyone builds like ad servers and cookie stores, and, you know. So you built some, built some of those things as well. So one of the things I have said before is that I— I believe that we need more product and engineering folks leading trust and safety teams, because I think, uh, especially the more technology comes along like LLMs and other things that could help with some of these problems. How do you think about the interplay between the tools, the people building them, the people using them, the people running operations, the people deciding the policies, the legal folks who are worrying about legal exposure? There's just so many different members of that cross-functional group. Like, how do you think about the different roles and how the things you're working on affect that dynamic as well?

Juliet: Yeah, I mean, I'm totally with you. I totally think that, you know, product people, engineers need to have a bigger role, especially in the trust and safety space. My approach has always been to help everyone play to their strengths. You know, it's not like there's going to be a single person who's excellent across every single function, but the point, especially for product people, is to kind of bring together those experts who are domain expertise, really good at scaled operations, really good at engineering architecture, really good at understanding the regulatory landscape and how to kind of like forecast upcoming requirements. I think the role that I've tried to play and that I always encourage product people to play is to kind of be the facilitator. I use the metaphor of curling, the sport where it's like, look, there's going to be some big rocks on the ice. They're going to be going fast. And like, as a PM, I just want to be doing the sweeping. I'm not going to slow anything down, but just get it to the right direction. Because I also think, you know, like as a PM, you know, I've worked with operations teams and policy people who really, really know like safety areas and on the ads integrity side too. But, you know, being really familiar with that area and then seeing exactly how the tools are built to support those types of functions, that actually informs like the end user experience, whether that user is a person or an advertiser or another organization. Like I think altogether, that's really what it means to have a really good, strong, powerful product experience and safer internet too.

Rob: I agree. Do you think one of the things someone said to me, which was now that you have LLMs that can help you write code and you can stand up a bunch of tools, a bunch of things, it actually makes some of the policy and legal folks like make some of that insight and expertise more valuable? Like, how do you think about how LLMs, for example, other newer technologies, do they change the way in which these teams have to work together or who focuses on what in terms of energy?

Juliet: I think for a baseline, LLMs can definitely help with that. But I think every single team that works on safety or policy, all the escalations that end up in like these really nuanced decisions, the really tough decisions, like that's not necessarily on the open web. And that's not necessarily going to be in the training data for AI models and LLMs. So I do think there's like a pretty clear upper limit of how much LLMs can really help with those functions. But I do think that they can help kind of create greater empathy across roles. I think like I've been writing more code as a PM. I use AI coding tools and models pretty much on a daily basis. And it definitely helps me understand the entire system. And if I understand the system, I can create better user experiences, better products. And I like to think that's the same thing for anyone else who's kind of like maybe stretching across functional borders and using LLMs to kind of augment themselves. It gives you a taste of what other people do. And then you can either figure out like, oh, you know, this is helping me, but also I can understand why it's so complicated.

Rob: Yeah, I'd say LLM coding has helped me build some actually useful things, maybe more for myself, but then also to realize how much actual engineers add in terms of being able to actually scale things. Because sure, I could build something that can work for 100 or 200 or 1,000 items, but like for a million items, it's not quite as easy.

Juliet: Yeah, exactly. My partner and I both use AI for engineering. And like the way he prompts stuff versus the way I prompt stuff could not be more different. And I'm like, this is why you are the engineering leader and I'm not.

Rob: Yeah, absolutely. So do you have any sense? I guess some companies have been talking about how they're using LLMs more. Are they going to rely more on LLMs in their trust and safety work? Do you think that some of that might be premature based on what you just said? Or do you think that now is the right time to be talking about that, but the reality might be slightly different behind the scenes? In terms of the messaging and narrative around this stuff, I'm curious how you think about that as someone who's kind of in the been working, has worked on these teams and now you're kind of in the middle of it, but in a slightly different way.

Juliet: Yeah. You know, I think it's, you know, a lot of people who come to the Roost community, they're either solo developers who are working on a startup in a project and they want to make sure that their users are safe, but they're a one-person team. You know, across everything. They don't have necessarily the experience or the knowledge or just the bandwidth and time to do some of the things related to moderation or safety. But then also, you know, we've work a lot closely with like the decentralized communities and decentralized protocols, and it's kind of a similar situation. They're usually much smaller teams or solo or one or two-person developers. And likewise, they're leaning heavily on LLMs and automation to try to figure out how can I provide the base minimum protection for our communities. And of course, you know, the bigger companies, I think everyone's trying to figure out like where does it actually make sense to use AI, to use LLMs. I still think that human judgment is the most critical part of a safety team, whether it's someone who's kind of going through and labeling and doing that ground truth labeling or fine-tuning a model. I think there's a lot of parts of the AI development pipeline that are actually kind of like evolutions of traditional moderation or integrity workflows. They're just using different names. The tool might even still be the same. But I think ultimately, the person who's writing the policy and whether that policy becomes a prompt or it becomes guidance for golden set labeling, the human is still very, very much needed. I really just see LLMs and AI as a tool. And that's how I'm seeing people really adopt that as well.

Rob: Can you talk a little bit about the kinds of sub-problems that the tools that Roost is building are focused on or the areas that you're focused on currently?

Juliet: Yeah, I mean, child safety is probably one of our biggest focuses. In the year that we were incubating Roost, we did a bunch of research interviews and we talked to trust and safety teams of different sizes, big companies, small startups, nonprofits. And child safety was kind of the one area where everyone's like, yeah, I know this is important. I know it's legal required in different places. I just don't have the budget for the tooling or I don't even know where to get started. So part of it was like us pointing people to different resources. So like, hey, you can get PhotoDNA from Tech Coalition or here, you can email this person to get hash credentials. But that is kind of like the crux of what some of our tools do. So Coop is, you know, typical review tool. You can use it for anything, not just child safety. It's kind of like policy harm agnostic, but it is connected to NCMEC and it has kind of like hash detection. It's got the Content Safety API. It's like we're trying to get the core pieces together and just put them all on one plate. I'm mixing so many metaphors, but people will follow along. But we want to make everything really accessible and easy to use. So that way people can get started faster with kind of the best practices in the industry. And what we're also seeing is, you know, I think social media has kind of fragmented across kind of like the major platforms onto a bunch of different ones. And a lot of those platforms need that kind of real-time like events decisions. So Osprey has been super helpful. My co-founder, Camille François, wrote this framework called the ABC framework, originally for disinformation, but I like to apply it to everything. So actors, behavior, and content. And Osprey is really good at the actors and behavior. And Coop is really good at incorporating content into all of it.

Rob: I had this, you obviously won't have heard this because the episode won't have come out, hasn't come out yet. It's coming out tomorrow. But I was talking to a former colleague from Google who was really talking about how actor trust is an important part of how they're thinking about thinking about things. And one of the things he said was he said two-thirds of the enforcement in the Google Play Store was related to developer identity and not content. And he was saying only a third of it is about content. So as you think about the ABC's actor behavior content, how do you see the actors and the content interplaying like or the things we're missing? Like I always feel like we're missing so much infrastructure on the identity side. Like content is not easy. I'm not trying to say that, but it feels like in some ways, identity is a more difficult, more tricky problem just based on what we have today. So how do you think about that side of things?

Juliet: Yeah, it is definitely tricky. I think one of the hardest things about actor trust is how do you really properly balance safety and privacy? I very much believe that there are two sides of the same coin. You shouldn't have to compromise one for the other. I think it might be challenging, but we owe it to ourselves and we owe it to the world to try to figure out how do we actually find that balance. So I think things around like identity or age verification, like I don't know if the world has really settled on a total standard of like what works and what's acceptable and doesn't actually put more people into potentially risky or dangerous situations. And I think that the relationship between actors and content is, it changes everything. I think a single piece of content is harmful in certain contexts, depending on who posted it and who it's being shared to. And it can have a very benign, completely different context depending on the person or actor who's posting it. So I think that's one of the reasons why trust and safety is so complex and so complicated. But I also think there's so many different ways and so many different signals of building actor trust. I've seen some really interesting experiments around how can you leverage almost like the peer social network and kind of be like, well, how do we really define trust between multiple actors and how can we incorporate that into a signal versus a very kind of like blunt approach of like, let's just require like government ID verification for everyone. Because I have a lot more thoughts on that one.

Rob: So one thing we learned, so when I was working at Facebook, we were looking at there's all these tools that the bad actors are using for hiding their content or other things. And some of them actually use a referral system. You only get access to the tools by a referral from presumably someone else who's doing sketchy things. And some of them actually require multiple referrals. So definitely, I think there's probably more that can be done with referrals with people at the same company clicking the button at the same time. I don't know. Like, I don't know. I think back to that war games where they have to turn the key at the same time kind of thing. Speaking of mixing all the metaphors. But I do think we have to probably come up with some new approaches that we haven't thought of because we can't just rely on government identity credentials or other stuff. And we can't just rely on content, but we have to figure out a way for them to work together.

Juliet: Yeah, exactly. I'm going to throw another metaphor into the mix, but like it should be a really, really well-layered parfait. There's not going to be one signal that's going to solve everything for us. But I think we can take a little bit of each signal, put it all together, and then we have a delicious dish of trust.

Rob: Absolutely. So the other thing that I think is also a really important part of what I think of Roost as doing or potentially doing is community related, engaging with the community and getting people to realize what tools and resources are available to them. I think as part of that, I saw you're speaking at TrustCon and you're going to be doing a lot of things at TrustCon. Tell us a little bit about that and how you see the kind of the role of the organization as well as yourself in the community side of this whole area.

Juliet: Yeah, absolutely. I don't know why I did it to myself, but I will be doing six sessions at TrustCon this year. Some of them are panels for the tooling research committee that I co-chair through the Trust and Safety Foundation. A few of them are hands-on workshops, both to kind of like integrate and deploy Roost tools and get yourself started and just see if it works for you. Some are, you know, how do you actually figure out where AI makes sense to your workflow? And then a panel, because I went with a spicier take of, you know, your tools aren't special because I think, you know, everyone thinks that their tools are unique because maybe the harms and abuses that happen on every single platform are unique. But the tools are actually pretty much the same. And I verified that many times over. I think the role that we're trying to play at Roost is to kind of like bring the community together. You know, TrustCon is already such an amazing event to bring those who work on online safety into one place. But I think what I found is like as a product person, like I, for a very long time, like didn't meet other PMs in the safety space, especially at TrustCon, through organizations like the Tech Coalition. I was often the only product person or someone from the more technical angle at those meetings and kind of like driving some of those conversations. So I think there's an appetite. There's an unmet space there that we're trying to provide, which is people who write code, people who are builders, people who are developers who are really creating and really establishing some of these tools. Our Discord community is already over 500 people. It's really active. We got a lot of people who are really excited to know that they're not alone. Oftentimes, they might be the only person working on a system or the only engineer working on something. And they meet other engineers from different teams and they get to trade tips and troubleshoot things together. And by doing everything in the open, we're hoping that this just creates more room for a real collaboration.

Rob: Yeah, I've talked to some trust and safety teams. And one of the things, one of the themes that comes out of it often is in talking to folks is that they do feel there's this kind of separation. I think sometimes it's because of the stuff that they're working on. It's quite sensitive, but they really do appreciate, or in my experience, have appreciated this knowledge that there's other people facing the same challenges and so on. And we were at the Global IT Scams Conference recently and there was a bunch of interesting discussion about signal sharing. But a few people have said, look, actually, signal sharing is great, but tool sharing is actually potentially more impactful. Like share the tools you've built, share your approach to things. Because I think some of them are unique, but then every now and then, or not unique, like you said, every now and then you come across something where someone has just put things together in an interesting and novel way. And so I think it's about also sharing the successes, sharing the insights you get out of actually engaging with some of these. Things.

Juliet: Yeah, exactly. And sometimes that's something like, hey, this is how we made our system way more efficient when we're pulling hashes. Other times it's like a UI thing and one of the tools that surprisingly had a huge effect, maybe on wellness or on productivity.

Rob: I want to get your feedback on something. So one of the things I thought might be interesting, but at least when I was there, we never got around to it at Facebook, which was, you know, giving, having a person kind of doing like a rotation with every like different team, have someone who's responsible for understanding. They don't have to necessarily do the trust and safety work or integrity work, but they would be responsible on kind of a rotation for understanding and interfacing with those teams. I've always thought it's, you know, it's really great to have empathy actually in both directions. Both, I also wanted some, you know, folks to have empathy for what the growth team or whatever it is is doing on the other side, someone who works on integrity, because I think otherwise you get these things where it feels very internally adversarial in these teams. I was curious if you had thoughts about that, because, you know, again, I think a lot of what you're talking about is, you know, there's applications for small three-person startups as well as teams within big companies.

Juliet: Yeah, definitely. And I have some of that first-hand experience too. You know, like when I was at Snapchat, the safety and moderation product team was actually part of like modernization and growth because it was seen as this kind of business critical function. And that was a really interesting placement for us because one, like, you know, across org structures, it gave us a certain degree of independence from other types of feature organizations where we could really work with them in a more kind of like peer way, where it wasn't like, you know, competing within the same organizations or having conflicting incentives. And also gives a lot of insight and ability to shape, you know, safety by design for any of the revenue or monetization strategies too. So I'm totally with you on the kind of that kind of rotational piece. I think having that kind of empathy, you know, it helps inform and understand business priorities, but then also, you know, figure out how do we actually frame safety and everything in a way that positions it as something that's really good for a company, really good for an organization and vice versa. You know, I think there's a lot of great insights that can kind of like improve the way that we do safety and fight abuse in a lot of ways. You know, fresh perspectives always help. And I've definitely found that working at Roost, open source was pretty new to me until I started Roost. And so hearing from people who are really deeply embedded in the open source space and how they think about abuse and safety, it's so different from those of us who have only worked in kind of like closed source, private, you know, centralized spaces. And I think it's helped make us make better tools and better decisions.

Rob: What do you think are some of the specific things that are different? I had my team, my team at Google open sourced a bunch of things. And I always thought it was interesting, but I never, I didn't feel like we, you know, at least my team specifically, in a couple of instances, we pushed further and like actually got people to, more people to use it. We had some interesting wins, but I always wanted to do more with it. How do you see these things as different or the different approaches?

Juliet: I think one is just because it's open source, you know, like you get, like you mentioned earlier, like really big companies who are using our stuff, but then also really, really, really small teams. And so that kind of presents really fun engineering challenges of how do you actually architect a system that can scale down and scale up and essentially be as flexible and abstracted as possible. And I think, you know, because types of abuse and harm are so unique to different types of platforms, it's really interesting. You know, a type of abuse that takes place on like the Matrix protocol is going to be pretty different than what Discord sees or what Notion experiences. But because we can kind of get a little bit of insight into those flavors of abuse, we can kind of figure out like, okay, like, how do we then design a more flexible, more adaptable tool as well?

Rob: Yeah. That makes that makes a lot of sense. What are some of the things that you all are working on or, you know, or problems that you want to tackle next? What are the kinds of areas that you can tell us about that might be in the future on the Roost side?

Juliet: Yeah, well, our roadmap is public. So if anyone wants to take a look, they can always drop me a line, let me know if anything's missing. But I think there's a lot of broad areas that we're really interested in. I think on the child safety front, we just launched a partnership with Mila, the Quebec AI Research Lab, because we're really interested in kind of like, when we talk about child safety, when we talk about youth safety and wellness and taking care of children, there is the kind of like preventing abuse and preventing harm between two people. But then on the flip side, that is the kind of like child development. Like, how do we actually foster pro-social behavior? And how do we, you know, make sure that kids, especially if they're interacting with AI, are doing so in a safe way and that there are the right guardrails in place to kind of like monitor that. I'm really interested in doing more research and kind of like dual use risk as well. You know, I think that's one of the topics that comes up often for us working in safety and open source is like, you know, how do we actually put some data behind talking about the potential risks of making things open source versus how much of this is maybe like fear from a hypothesis that's internal, but like isn't necessarily backed up by data.

Rob: I think one of the things that I also feel, I mean, it's related, but on the on the kids' safety front is like, I just feel like there's a, a lot of companies have historically underinvested in the in this area in general, but I, but I mean like, I mean, entirely, like even down through the, you know, controls for like parental controls and so like I feel like there's a lot of talk about stuff, but when you actually as a as a product manager, if you push on some of the feature sets and the way things work, it appears clear that there's not enough people and not enough engineers and product managers working on some of those things. Do you think that, how do you, how does that you, well, firstly, I mean, you could totally disagree with me, but like I'm, I get a sense you might agree. But I'm curious how you feel about overall the prioritization of family related features on all platforms. Like, do you think that's a thing?

Juliet: I think it definitely could have used a little bit more prioritization and resourcing across every company and platform. And the other thing I'll add too is I think that, you know, teams in particular should also be part of that design process. I think a lot of times when I see these kind of like family-oriented features and in many ways, like understandably, it's really focused on kind of like parents and like, what do they need? But, you know, I think the difference between like a 13-year-old and a 16-year-old, super, super different. And when you look at things like, you know, the UN like rights of the child, like how do we incorporate some of that into the design practice for creating, you know, family experiences that respect the autonomy of the child and of the teenager, the young person, while also kind of like creating more visibility and like just like more opportunities to kind of like intervene or guardrail when situations could go off. I really do believe like there's no such thing as like a safe place on the internet versus an unsafe place on the internet. I think there's going to be safe and unsafe interactions no matter what space or corner of the internet it's going to be in. So a huge part of it's going to be digital literacy and how do we make sure that like kids as they're growing up understand kind of like what they're experiencing online and what tools are they have within their own control to kind of like manage their own experiences and give parents those same tools too. You know, how do we also kind of create really good kid spaces online? You know, like we need a Neopets of today. Internet today is like not really designed for like kids or teens at all.

Rob: I agree with that. And in fact, one of my crazier ideas that I ever actually built was, you know, I really got annoyed that live sports would have violent video game and movie ads. It's like, well, why don't we just have something that will take the TV stream and when there's an ad, it'll just, you know, block it out for kids or what, you know, like it's crazy. Who's going to pay for that? I don't know. Like there's a lot of these things, but I do think it just speaks to the fact that like there just there just needs to be more focus on a lot of these on these kinds of things. I think like another thing I've seen you talk a little bit about that I wanted to hear more about was and I'll reference it like I was I've been thinking about building or have been building some kind of ways for people to report scams and frauds and stuff like that and then potentially have humans review stuff. But some of it is like, hey, anyone can report anything. And so then I was concerned about the people reviewing the thing and making sure that we're filtering things appropriately before they get to see it. So I know you've talked a little bit about reviewer well-being around these topics. This is a very, very, you know, difficult area and it's very important for people to understand. Tell us a little bit about like reviewer well-being and how you think about that impacting smaller platforms and others and what role Roost may have to play there.

Juliet: Yeah, it is personally really important to me. You know, I was a moderator earlier in my career and I think throughout my career as a PM, I've always tried to make it a priority to spend time with the people who are doing that kind of like scaled review or scaled labeling. What we're really hearing on the Roost side is a lot of people from the decentralized protocols, right? Like we've got like Mastodon server admins and like they're one person and they run this server for community people, but they also have to act as moderator. They never intended to set out to do moderation or review, but when they encounter CSAM, they're the only person who has to be able to review it, report it, et cetera. And I think the mental toll that it takes on those kind of like, you know, solo people, those really small teams who don't necessarily know kind of like who to reach out to, that there is a network so that they don't feel so alone. And even beyond decentralized spaces, kind of like any kind of small startup that doesn't have kind of like, you know, years and years of experience and knowing where the stove is hot, that kind of, that kind of work can really take a toll. And I think you mentioned it exactly. Like the surprise aspect is a huge portion. Of what can really, really negatively impact someone's mental well-being is like if you don't know what's coming up next when you're reviewing, it's just it doesn't set up your brain to be able to process it well. So, I was really fortunate to work with a team of like trauma-informed coaches, you know, in previous PM roles. Um, but that's that's a luxury and benefit of having worked at big tech companies. Um, so on the Roost side, we're trying to figure out how do we kind of take some of those practices and those you know, those resilience learnings and bake them directly into the tools. So, one thing is like, you know, playing like Tetris or like Snake or any kind of like fast-paced clicky game that moves your eyes back and forth. It's almost like a DIY kind of like trauma. Um, I think it's EMDR kind of practice, but it's like, how do we bake in and like build that kind of game directly into the tool? So, maybe Coop, like if you hit a certain keyboard shortcut, an Easter egg pops up and you kind of play that game for a little bit and you can kind of like immediately start healing your brain. So that way, anything that is traumatic or disturbing that you've looked at doesn't get encoded into your brain the same way. And we're also just trying to put in like the basic minimum standards for any kind of content review. So, actually, Meta open source a really nice suite of kind of like reviewer wellness features. So, it's like you've got black and white filters, you've got sepia filters, you've got blurring modals and sliding scales, default settings where like videos are always auto-muted, things unblur if you hover over it. These are all things that I think like were hard-fought and earned over a series of years at the cost of a lot of people's mental health. And so, I think we owe it to the field to make sure that like any of the free tools that we provide have that automatically baked in by default.

Rob: Yeah, no, that's that's that's really very well said. Uh, um, how does this? I mean, you obviously have more experience in this, having you've worked on all the different parts of the moderation process. Tell me a little bit more about your opinions about the centralized versus decentralized moderation. Because obviously, I know there's some folks that I talk to are very much in one camp or the other, or or maybe you know, think that maybe incorrectly that LLMs can make this a non-issue, which again, I agree with you. I'm more in your camp than agreeing that that's going to make human review go away anytime soon. But tell me a little bit about the decentralized versus centralized review.

Juliet: You know, I think decentralized, one of the things that always comes up is there's so many different flavors and shapes of what decentralized even means, too. So, like, how ActivityPub and Mastodon does it is so different from how like Bluesky and the AT Protocol does it versus Nostr versus Threads. And so, I think one thing that I've heard consistently is like because there are these decentralized nodes, that is where kind of like signal tool sharing becomes really helpful because it's kind of like if there's one bad actor that is maybe hopping through different kinds of spaces, it's probably helpful for people to understand what that looks like. Now, Roost doesn't do any of that kind of like signal sharing, but I think having even just like a shared de facto standard of what the review tool is or like what the rules look like, so that way you can kind of share like the shape of an Osprey rule, for example, that can potentially help. I don't know how to necessarily answer like if decentralized or centralized is better. I think that they're just going to be very different models and for different types of users as well. I think that one thing I've personally appreciated about the decentralized space is how collaborative and open it is. You know, I think the AT Protocol, for example, super open source. You've got amazing projects there and people really are there to like help each other out. You know, I think Blacksky has kind of like shared their approach for community moderation and kind of pluralist examples of how major decisions get made. So that way, each decision is really made by their community for their community. And then there's other models too. You know, I think that, yeah, I don't think LLMs are going to fix everything. I am encouraged by these kind of bring your own policy LLMs that make it a little bit easier for communities to kind of decide like, hey, this is what it means to us to be safe. And we're going to codify that in a policy. And that policy is going to become a prompt for like Coop or for GPT safeguard or something, but you're still going to need a person. And I think decentralizes, you're just kind of like sharing the load across everyone versus centralized. You're still sharing the load, but everyone's just like rolling up into the same organization.

Rob: So it's maybe one way to think about it, like there's this baseline of things that are, you know, there's like a set of legal protections or things that just content that shouldn't be on the platform or the internet or whatever. And there's a layer that's platform specific. And then there's maybe your personal preferences and filter that decides, okay, well, I really don't want to see that. Like I have a gambling problem. I don't. But I, you know, if someone has a gambling problem, they don't want to see gambling content. It might be legal. It might be allowed by the policies of the platform, but their personal filter would disallow it.

Juliet: Yeah, exactly. And we've seen kind of like early experiments around this with Bluesky and the AT Protocol. I think they call this composable moderation. So like Bluesky has their own centralized trust and safety team. They do that kind of layer of like, these are the legal things. These are things that are just like not allowed on Bluesky in general. But let's say, you know, like I'm a cat person. Let's say I don't want to see pictures of dogs anywhere. Also not true, but let's assume. I can use a personal labeler to be like any dog. I just don't want to see it. And that's my own flavor of moderation that only works for me.

Rob: Yeah, I think it's interesting. As an avid reader of science fiction, I think there's some Neal Stephenson type discussions about this kind of stuff where families or communities subscribe and they pay for actually in some cases human filtering as well, right? It's some machine-human combination. But I also worry that like that kind of thing then takes us down just another way that like, you know, you have a different set of rules or different set of protections for wealthy versus everyone else. I do worry a little bit about that from a from a fairness perspective.

Juliet: Yeah. And this is where I'm going to make my shameless pitch of like, that's why everything should be open source. You know, I think open source just gives people more control. You know, like I can really own my own technology and it's free depending on the license, but like is free. You know, people deserve to have more access to technology. Technology shapes our lives. We get, we should be able to have a say in how it works too. And I also worry a little bit about, you know, like, you know, does everyone's customized version of moderation exacerbate this kind of like context collapse? But then I think about how algorithms have already kind of been doing that. I'm like, well, it's, I don't know if it's going to make it worse or at least make the experience of being online and, you know, at the whimsy of these algorithms slightly better because you have more control over it. I'm just all about giving more autonomy and control back to people and back to communities that maybe didn't have access to it before.

Rob: Yeah, I find that a lot of the algorithms these days, the time it takes to switch topics based on you showing some interest in a topic is it's very quick. It's much, much quicker. It's just getting quicker all the time. And it's do you think, going back to what you were just saying, do you think that governments have a role to play in encouraging open source and these kind of safety technologies to be open source shared? Have you thought about mechanisms that might exist or we might create as a society to do that?

Juliet: You know, I think governments and policymakers definitely have a role to play. You know, I think, especially in recent news, I've seen a lot of governments kind of adopt open source and talk loudly about it. I think that's certainly one way is just like bring more awareness that these things exist. Australia's safety commissioner, eSafety, did a case study about Roost and kind of like talking about how our tools just level the playing field a little bit for small organizations. I think it's just really important to make sure that anyone who regulation impacts also understands that there are free open source options that they can make work for them as well. And another way that I think regulators can kind of play a role is also just supporting open source projects. I think the either whether it's maintaining or contributing in a monetary or code way, like just like supporting open source projects and make sure that they are sustainable is equally as important. We just wrapped up UN Open Source Week over here in New York, so it's very top of mind.

Rob: Oh, that's great. That's great. Are there other forums for discussing this kind of stuff that you're participating in or aware of? Because I'm always curious, like how these ideas get disseminated or where are the people actually discussing these kinds of topics?

Juliet: Yeah, it's definitely mixed. As someone who's coming from the trust and safety world, who's relatively new to open source, conferences like FOSDEM or FOSSE, those are new to me. But it's been really cool to go to those conferences and have the same conversations about trust and safety, but with a completely different group of people. And they're thinking about it, but they're not necessarily using the words trust and safety or even moderation. They're just like, this stuff is happening. We need a way to figure it out. And so we have our Discord. We're seeing some interesting conversations happen across different types of Discord servers, not only our own, but like Wikipedias and like Eleuther and Common Crawl. I think it's also happening across GitHub discussions. A lot of the times I'll get linked to like ActivityPub, like trust and safety working groups notes where they're talking about the different types of tools available. And part of what we're trying to do is kind of like bring all these communities together. So that is partially why our community is like so mixed right now, because it's like people who have been in like, you know, at Facebook for 12 years working on integrity, talking to people who, you know, run Neocities and thought they were the only person doing moderation. And now they can actually like meet each other and swap notes.

Rob: So how will you know, you know, what are the success metrics or the things you're trying to get done the next couple of years? How will you know that you've changed the default? You talked about not having trust and safety folks start at the bottom of the hill. How will you know that you've achieved that to some extent?

Juliet: The fun and also like inferior. Part of open source for a product person is like it's the measurement piece is really difficult because like it's the source code is out there. It's freely licensed. Anyone could just be running Roost tools and they don't have to tell us. So maybe we're changing the field completely already, but you know, based on kind of who's reached out, I think we're looking at, of course, just kind of like core adoption metrics and like, you know, how many people are deploying our tools. But we try to develop these relationships with the organizations that are running our stuff and try to work with them, try to figure out like, hey, like, is this actually solving your problem? You know, like, is this driving down kind of like your, you know, your end-to-end turnaround time for reviews? Or maybe you're looking at prevalence. Like, was there like a step change between before using Roost tools and after? So a lot of them are almost traditional metrics, but the way that we're getting them is very, very different. And we're considering adding and like opt-in telemetry so that we can figure out both better monitoring, but then also better measurement of our progress.

Rob: And what are some of the failure modes or the things you're worried about or the things you want to avoid as you continue on your journey with open source tools?

Juliet: You know, in my what keeps me up at night is like, it's just an endless field of forks. So like if everyone just forks our project and maintains it, they can. But I just worry about like all the technical debt that that creates for everyone. In an ideal world, people are sharing more of their code and contributions further upstream. And Roost will continue to grow. We'll continue to sustain that. But I think one of the potential failure modes we're looking at is kind of like, if we don't better understand dual use risks around kind of like making safety things open source, one, we're never going to actually move forward with making really sophisticated safety technology open source. And two, like the technology that we have right now will remain a bit of a black box. So we wrote a little bit about how like model cards are super standard when you talk about AI models. But then you look at all the models for safety and it's like, where are the model cards? It's like, I really worry about kind of like, what are the, what are the biases and limitations that currently exist that no one knows or talks about because they're not documented publicly? And what are the kind of like long-term effects on different people around the world if we don't know that bias and we just continue to propagate it?

Rob: So do you think there's a role for like standards bodies or testing? I mean, you know, every time someone comes up with a new LLM, there's all these like tests and benchmarks and stuff. And I'm like, you know, my eyes glaze over at this point. But perhaps, you know, I kind of, we wanted to, I set up a nonprofit to do some of this and we're trying to think about ways to do this. I think having some defensible random sample baseline offset, you know, but again, like for certain safety outcomes, do you think there's something that needs to exist there that doesn't?

Juliet: I think so. And I think like we probably need to have some hard conversations about what benchmarking evals really mean. You know, I've seen like, here's the benchmark for benchmarks and here's a benchmark for the benchmark of benchmarks. And it's like for things like safety, it's like, okay, like for things like child safety, how do you benchmark that when it's so sensitive? There are real ethical considerations. There's real legal considerations about who can actually do it. And for things like, you know, self-harm or suicide, it's kind of like, hey, like, are any clinicians involved or are any doctors involved in like who decides this is actually harmful or not? So I think there's definitely something missing. I think there are gaps that are identified at least. And I do, you know, I'm an idealist. I believe that we can close those gaps, but we need to have those really hard conversations and do the researchers try to figure out, okay, what is the proper shape of this problem and how can we work together to solve it?

Rob: Yeah, my most overused metaphor of all is the, or is it a simile? I don't know, whatever. The comparison is to the crash testing of cars, right? I'm like, you kind of need to, you know, I think some objective third party has to do it. I think you have to buy the product, the same product that the consumers get, and you need to test it against a bunch of stuff. And you need to keep evolving the testing to figure out where the problem points are. And eventually, I think you then get involvement of the engineers, these companies, and whatever. But, you know, I think there's probably some kind of safety testing like that that can be done. Your point about the certain, you know, how do we test the most sensitive stuff is a real concern. I don't want to continue that analogy with crash test dummies or whatever, but like, you know, we do need mechanisms for figuring this stuff out. I just don't think we can leave it to the promises of the platform saying, hey, we're great at this thing, or even like, you know, things like counts, like number of reports. Like that just isn't an accurate way of understanding it. You have no idea what prevalence is. And prevalence might also not be the right thing. So I really think we need more focus on this kind of thing.

Juliet: Yeah, exactly. And measurements, one of the hardest things, I think, for folks in our field to really talk about and also one of the hardest things to solve. But that doesn't mean we shouldn't try.

Rob: Yeah, absolutely. So this has been a great conversation. Juliet, how can people find Roost, find you? What's coming up? What should people look for to support this effort and mission?

Juliet: I had a lot of fun today. You can find us at roost.tools. That is our website. We're also on GitHub at roost org. If you go to our website, there's a link to our Discord. People can find me. I'm on Bluesky, Juliet Shen, BSky.social. And I'm also on LinkedIn, which is apparently where everyone went after Twitter.

Rob: Yes, yes. A lot of people have a love-hate relationship with LinkedIn, but we'll definitely catch you there as well. Thanks so much for taking the time. This is very informative and hopefully helpful for a lot of folks, large and small, alike.

Juliet: Thanks for having me.