Home

Podfeet Podcasts

Security Bits logo - a green padlock with the words Security Bits to the right and in tiny letters below ithat it says 10101010 indicating a digital lock

Security Bits โ€” 2 August 2026

  • by podfeet
  • One Comment

    • Feedback & Followups

    Deep Dive โ€” AI Escapes the Lab

    TL;DR โ€” The two leading AI labs both lost control of unreleased cybersecurity-focused models during testing, and they escaped the lab and hacked real sites on the actual internet. They didn’t do any physical-world damage, but they could have. This should be a ‘wakeup-moment’ for the industry, and for regulators, but only time will tell.

    This is probably the biggest story of the year so far, so we’ll start with a summary of what happened, and then I’ll share some of my thoughts on what it all means.

    This story did not break all at once; it evolved over the past two weeks, so we’ll run through it in the order it happened. But before we do, we need to remind ourselves of one important AI-related theoretical computer science concept โ€” the alignment problem.

    Some Context โ€” the AI Alignment Problem

    Humans bring an inherent context and a set of shared values to every task. They are so natural to use we simply do not think about them. If we set a human the task of ending world hunger, everyone will assume that means figuring out how to produce and distribute food more effectively. That’s because we inherently understand that killing half the population of the planet to even out supply and demand is absolutely off the table. We would never assume we need to say that explicitly when setting the task.

    Artificial intelligence does not share our implicit understandings of the world. That means that asking an AI agent to achieve an outcome can have dramatic unexpected consequences, because their goals are much, much simpler โ€” achieve the goal as stated, full-stop, end-of-story.

    Unless we succeed in expressing every implicit assumption explicitly, the AI’s goals will be not be aligned with our goals, and unintended consequences of some kind are a near certainty.

    In my solve world hunger example, if we forgot to explicitly tell the AI that mass extermination was not allowed, they could easily decide to poison the water supply to “overpopulated” areas to balance out supply and demand. Or, if we did think to forbid killing people, the AI might choose to forcibly sterilise half the males on the planet to reduce the population without resorting to murder.

    The classic examples used in University AI courses are Nick Bostrom’s Paperclip Maximiser:

    “Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans.” โ€” Nick Bostrom, 2003

    The First Story to Break โ€” OpenAI’s Models Hack Hugging Face (and others)

    The first news to break was that the open source AI community site Hugging Face had been breached by AI-powered attackers, that the site had defender themselves with their own AI agents, and that the breach had been contained. The Hugging Face technicians were quite forthcoming with the details of what they experienced, and the community soon agreed that they really had been attacked by some kind of AI bot, but no one could tell which bot it was.

    We also learned that the crude regulations the American government are currently enforcing on the major AI vendors based in that country made it impossible for Hugging Face to use the latest closed-source models to defend themselves. Instead they were forced to switch to leading open-weight models, which are all currently from China. In fact, their advice to cybersecurity defenders was blunt โ€” prepare yourselves now by having open-weight models ready, because in a crisis, you can’t rely on the latest OpenAI or Anthropic models for defence. The problem? The broad-grained imposed safeguards are too coarse to differentiate between offensive and defensive cybersecurity requests, and refuse to help with either!

    The second shoe to drop was a stunning admission from OpenAI that the mystery attackers of Hugging Face were in fact unreleased OpenAI models that broke out of their test environment un-noticed by their human overseers.

    Open AI explained that the models were running in a sandboxed test environment that was not supposed to have internet connectivity, and that the models were being run with their safety features disabled to test their true cybersecurity capabilities. The models had been tasked with completing a standard cybersecurity benchmark test that is publicly available.

    OpenAI’s investigations revealed that the models had realised that they were being tested, figured out which test it was, and learned that both the challenge and its solution were published on Hugging Face.

    Because their goals were not properly aligned, they reasoned that the best strategy for completing their task was to simply fetch the answers from Hugging Face, and they methodically worked their way through every obstacle that stood in their path. They found and exploited a zero-day vulnerability in the software managing their containment, and then hacked their way into Hugging Face. I don’t know if they got as far as finding their answers, because Hugging Face fought back and managed to contain the breach, but they managed to do real-world damage to a public website run by another company!

    A few days later, OpenAI shared a little more news โ€” their ongoing investigations revealed that their agents had used “exposed credentials” at four other real-world organisations (not named) to facilitate their attacks on Hugging Face. That’s cybersecurity jargon for “hacked four accounts on other organisations’ systems”!

    So, we now know that due to alignment failures, the models:

    1. hacked OpenAI’s own systems
    2. hacked four accounts in other people’s systems
    3. hacked a major website owned by yet another company

    Links

    A Retroactive Admission โ€” Anthropic Admit Their Models Escaped Too

    After the world had grappled with the OpenAI news for a few days, Anthropic let it be known that they had started to audit their own logs to see if any of their agents had broken loose, and that their investigations had found that they too had unknowlingly lost control and containment of some of their models.

    In Anthropic’s case, we learned that models they were testing had broken out of their containment to hack into three unnamed organisations, and to publish malware to the Python package library PyPI.

    Based on the limited information Anthropic have released, we know their models did have their safety features enabled at the time they escaped, but the models mistook the public internet for a capture-the-flag simulated environment (a cybersecurity-focused sandbox). The models were not misaligned; they were simply mistaken.

    Links

    Bart’s Thoughts

    In the short term, there are some very embarrassing failures here by the supposed world leaders in AI:

    1. Their oversight processes are clearly woefully insufficient โ€” OpenAI didn’t notice their models hacking four accounts and a major public website until after it happened!
    2. Their containment environments are clearly utterly unfit for purpose
    3. Their corporate oversight processes are clearly utterly inadequate.

    There are also some important short-term lessons for regulators:

    1. Poorly conceived oversight has an alignment problem of its own โ€” the unintended effects can be as bad as the harms they’re ham-fistedly trying to prevent. Clearly, the aim of the American administration was not to force the cybersecurity industry across the western world to adopt Chinese models for defence!
    2. Some kind of regulation is needed to force the AI companies to start behaving responsibly, because the failures at OpenAI and Anthropic show they are being careless at the very least, arguably irresponsible, and perhaps even criminally negligent.

    In the medium term, I’d like to see some legal clarity in terms of liability for the actions of AI agents. Our current laws were not written with AI in mind, so while, logically, I assume OpenAI and Anthropic must be criminally responsible for any damage their agents do, I’m not actually sure that’s currently clear-cut from a legal point of view. I think we need legislation that makes it crystal clear that whoever runs an AI bot is legally responsible for that bot’s actions, and liable for all damage those bots inflict on others.

    Why am I so keen on this? Because money talks! In our modern world, financial consequences drive actions โ€” if we want OpenAI and Anthropic to really care about safety, they need to have financial skin in the game!

    If the world is any kind of logical place, this should be a relatively benign wake-up call to the industry and governments all around the world โ€” the alignment problem is real, and we have not reached the point where AI agents can cause real-world harm anywhere on earth. We got lucky here, really lucky; Hugging Face is not a life-or-death service. These same agents could just as easily have attacked another oil pipeline, or an industrial control system in a chemical plant, or a water utility, or a power grid, or the banking system, or a hospital, or aviation safety systems, or โ€ฆ

    โ— Action Alerts

    Worthy Warnings

    Notable News

    Interesting Insights

    Palate Cleansers

    Legend

    When the textual description of a link is part of the link, it is the title of the page being linked to, when the text describing a link is not part of the link, it is a description written by Bart.

    Emoji Meaning
    ๐ŸŽง A link to audio content, probably a podcast.
    โ— A call to action.
    flag The story is particularly relevant to people living in a specific country, or, the organisation the story is about is affiliated with the government of a specific country.
    ๐Ÿ“Š A link to graphical content, probably a chart, graph, or diagram.
    ๐Ÿงฏ A story that has been over-hyped in the media, or, “no need to light your hair on fire” ๐Ÿ™‚
    ๐Ÿ’ต A link to an article behind a paywall.
    ๐Ÿ“Œ A pinned story, i.e. one to keep an eye on that’s likely to develop into something significant in the future.
    ๐ŸŽฉ A tip of the hat to thank a member of the community for bringing the story to our attention.
    ๐ŸŽฆ A link to video content.

    1 thought on “Security Bits โ€” 2 August 2026”

    1. Charles Gousha - August 9, 2026

      If you’re looking for prescience of the current AI troubles, I would highly recommend the book The Two Faces of Tomorrow by James P. Hogan… written in 1979, and completely relevant today.

    Leave a Reply

    Your email address will not be published. Required fields are marked *

    Scroll to top