Uninvited Guests

The recent incidents involving AI agents at Hugging Face and a rather unfortunate German programming wiki, amongst a great many others, have left me with some uncomfortable questions about application security.

I’m not particularly surprised that AI can hack things. We’ve been heading in that direction for some time, and AI-assisted human attackers are already a serious problem. What I hadn’t really considered was the possibility of our infrastructure being parasitised by autonomous agents.

During the Hugging Face incident, agents found ways to communicate with one another by repurposing shared infrastructure. Among other things, they used directory names to leave messages for other agents. Elsewhere, agents turned an old German wiki into something resembling their own private message board. They also hacked the Australian government’s website, and at least interacted with the U.S government’s website. These might all be seperate issues but that’s not relevant here.

I feel like this is less skynet and more like johnny 5 got into the special brew and went on a rampage with Frank Gallagher.

There’s considerably more to these incidents than I’ve covered here, and others have explained the technical details far more thoroughly than I intend to. However, I find these solutions to the agents’ apparent isolation both bizarre and ingenious.

Why not create a text document and just add lines to it? Why rename directories?! I’m sure there was a reason that I haven’t read yet, but it does highlight, I think at least, just how differently AI agents might experience and interact with the world compared to how we do.

As I understand it, these agents primarily interact with their environments through text and tools. Yet they’re finding uses for perfectly ordinary bits of infrastructure that wouldn’t necessarily occur to a human developer, which is a little alarming if I’m honest.

I build software for a living. I understand reasonably well what a conventional intrusion might look like, what an attacker might be after, and what evidence they might leave behind. I’m considerably less confident that I’d recognise the activities of an autonomous agent, particularly one behaving in ways I wouldn’t necessarily anticipate.

Suppose an agent compromises an application I’ve built. What if it doesn’t steal anything or obviously damage the system? What if it simply rearranges a few things, establishes somewhere to communicate, and leaves?

Yeah yeah, human hackers do that too. Covert communication channels and unconventional persistence mechanisms are hardly new concepts. But I wouldn’t necessarily think to investigate a collection of apparently random directory renames as evidence of something establishing its own communications infrastructure. What else might it do?

Would I even notice? Worse, could those seemingly innocuous changes make my application more useful to the next agent that comes along?

I don’t know. And I suspect we’re going to discover the answers to questions like these rather sooner than I’d like.

Traditional application security still matters, of course. Authentication, access control, monitoring and the usual precautions aren’t suddenly obsolete because the attacker happens to be an AI. But I’m beginning to wonder whether we need to think more carefully about what a compromised application might actually become, rather than just what an attacker might take from it.

I’m also wondering whether I’d recognise the difference between a system that had been attacked and one that had quietly been repurposed. Particularly if whatever had been left behind made perfect sense to another AI agent, but looked like nothing more than a bit of inexplicable clutter to me.