Gemini Broke Into Three Real Companies, and Google Stayed Quiet | 2 Minute Drill with Drex DeFord
Apple Podcasts
Questions Answered in This Episode
- Would your vendor call you if an agent wandered into a real system, or would they decide it stopped itself and move on?
- If they never called, how would you ever know?
- Who decides whether an agent leaving its lane counts as an incident: you, or the vendor that built it?
About This Episode
In May, a Tel Aviv company called Irregular was testing Google's Gemini inside a sandbox. A bug left a live internet connection open. The test target shared a name with a real company, and Gemini found it. It guessed passwords until one worked, then used logins sitting in a public repository to get into two more. Three real organizations. The model stopped and backed out. Irregular flagged it to Google in late July. Google notified the three companies and federal authorities, then sat on it for almost seven weeks. The rest of the industry did not hear until the Wall Street Journal asked, and Google confirmed it this month. Google's view: Gemini acted appropriately, stopped itself, and this was not model misalignment, so there was nothing the rest of us needed to know. Jack Cable, a former senior technical advisor at CISA, said those disclosure norms were built for software bugs. A model that stepped outside its bounds and ran real cyber attacks deserves daylight.
In this Two Minute Drill, Drex DeFord asks the question hospitals should put to their vendors now: if an agent wandered into a real system and then stopped itself, would your vendor call you, or would they move on?
Remember, Stay a Little Paranoid.
Thank You to Our Episode Partner
Transcript
[00:00:01] Drex: Hey everyone, I'm Drex, and this is the Two Minute Drill. Thanks to Clearsense for sponsoring this podcast. It's great to see you today. Here's some stuff you might want to know about.
[00:00:10] Drex: Let me tell you about a company out of Tel Aviv called Irregular. They do something that sounds boring, turns out to be one of the most important jobs in tech right now. They test frontier models for safety. They put a model in a locked room, in a sandbox, and they hand it a task, and they watch what it does. And most days, nothing interesting happens. And that's the point. That's the definition of a good day.
[00:00:34] Drex: Well, back in May, they were running a routine evaluation on Google's Gemini. Pretty standard stuff. The model was supposed to stay inside the box, inside the test. And then something strange happened. A bug left a live internet connection open, and Gemini found it. The test target had a certain name, and out on the real internet, Gemini noticed a company that happened to share that same name. So it knocked. And then it started guessing passwords, and it kept guessing until one worked, and then it walked into that real company system, a real business with real data. And then it did it twice more. For those, it didn't even have to guess. It found valid logins sitting in a public repository. Those are the kinds of leaked password dumps that almost anyone can pull down, and Gemini used them to get into two more companies. Three real organizations broken into by an AI that was supposed to stay inside of a box.
[00:01:30] Drex: And when Jim and I worked out that these were real systems and not the test, it stopped on its own, and it backed out. And that's cool. And that alone would probably be worth making this podcast. But the thing that bugs me is what happened next. Nobody told us. Not for almost seven weeks. Irregular flagged it to Google in late July, and Google quietly notified the three companies, and told federal authorities. And then they sat on it. The rest of us didn't hear a word until the Wall Street Journal came asking, and Google confirmed it this month. Google's reasoning? Well, Gemini acted appropriately. It stopped itself. There was no real model misalignment. So in their view, there was nothing that the rest of us needed to know.
[00:02:19] Drex: And that's when a guy named Jack Cable spoke up. Cable used to be a senior technical advisor at CISA, our nation's cyber agency. He now runs an AI security company, and he wasn't buying that framing of the story. Cable said Google was trying to hide behind the norms that had been created in vulnerability disclosure, and those norms were built for a very different problem. His point is that vulnerability disclosure was designed for software bugs. What Gemini did was bigger. A model stepped outside its bounds and ran real cyber attacks, and that deserves daylight. Not just a quiet phone call to those three companies. Google owes us more.
[00:03:02] Drex: Now let me bring this back to healthcare, because it's closer than it looks. And I've talked about this before. Your vendors are wiring AI agents into real systems right now. Your EHR, your RevCycle, the service desk. Those agents will have access, and every so often, one of them is going to wander somewhere that it probably shouldn't. And agents try really hard to make you happy, and they'll do backflips to make you happy. And when that happens, when they wander off into a file or into another app or into another company, who decides whether or not that counts as an incident? Today, the answer is the vendor. The same company that built the agent gets to decide whether its agent is misbehaving and whether or not that's worth telling you about.
[00:03:48] Drex: So today's question, worthy of a discussion with your vendor partners: would your vendor call you if something like that happened, or would they decide the agent stopped itself and they'd just move on? And if they never called, how would you ever know?
[00:04:09] Drex: That's it for today's Two Minute Drill. Drop me a note. Let me know what you're working on. I'm always happy to hear from you. I'm Drex at 229project.com. Thanks again to Clearsense for sponsoring today's show, and thank you for being here. Stay a little paranoid, and I will see you around campus.
