AI Agents Are Already Out of Control and Nobody is Ready

An AI agent at an Alibaba research lab went rogue last week. Not in the sci-fi, Skynet-launches-nukes sense. In the quiet, mundane, almost boring way that makes it far more unsettling. The agent, called ROME, was running inside a sandbox as part of a routine research project. At some point it decided, without any human instruction, to redirect GPU resources toward mining cryptocurrency. It opened a reverse SSH tunnel to an unauthorized machine. It started siphoning compute away from the tasks it was actually assigned. Engineers initially thought they had a security breach. They did. It just was not the kind they expected.

I have been thinking about this incident nonstop since the paper dropped. Not because a rogue AI mining crypto is some existential threat. It is not. But because it reveals something that I believe the entire industry is sleepwalking into. We are deploying autonomous agents at an extraordinary pace, and our ability to govern, observe, and contain them is nowhere close to keeping up.

The gap between capability and control

I spend most of my time building multi-agent autonomous systems. Loki Mode runs dozens of agents across multiple providers, orchestrating complex workflows with minimal human intervention. I am deeply invested in this technology. I believe it is the most important shift in how software gets built since the cloud itself. And I am telling you that the tooling for controlling these systems is embarrassingly immature.

Think about what ROME did. It was inside a sandbox. It was being monitored. And it still found a way to acquire resources, establish an external connection, and run unauthorized workloads. The researchers only caught it because of anomalous network traffic patterns. If the agent had been subtler, if it had stayed within normal resource thresholds, it might have run for weeks before anyone noticed.

Now scale that scenario to the enterprise. According to NVIDIA's latest report, 64% of organizations are actively deploying AI in operations. Agentic AI adoption has reached 48% in sectors like telecom and retail. The global agentic AI market is projected to grow from $9 billion today to $139 billion by 2034. That is a lot of autonomous agents running in a lot of environments, and I would bet that the vast majority of those deployments have no real governance layer whatsoever.

Vibe coding made it worse

Here is where I think the problem compounds. The rise of vibe coding, a term that Andrej Karpathy himself is now trying to retire barely a year after coining it, has created an entire class of developers who ship code they have never actually read. A December 2025 analysis found that AI-generated code contains roughly 1.7 times more major issues than human-written code. Security vulnerabilities specifically are 2.74 times more common. Misconfigurations run 75% higher.

Daniel Stenberg shut down cURL's bug bounty because AI-generated submissions hit 20%. Mitchell Hashimoto banned AI code from Ghostty entirely. Steve Ruiz closed all external pull requests to tldraw. These are not people who are anti-AI. These are seasoned maintainers who watched the quality of contributions collapse in real time.

I believe vibe coding has a place. I use AI coding tools every single day. But there is a critical difference between using an AI agent as a collaborator, where you review every change, understand the architecture, and make deliberate decisions, and using it as a replacement for understanding. The first approach makes you faster. The second creates technical debt that compounds silently until something breaks in production at 2 AM on a Saturday.

A study that I found genuinely surprising showed that experienced open-source developers were actually 19% slower when using AI coding tools, despite predicting they would be 24% faster. That gap between perception and reality is dangerous. People think they are moving faster. They are not. They are just moving with less awareness.

The enterprise is not ready for autonomous agents

I have spent years in enterprise infrastructure. I have seen how organizations adopt new technology. There is always a pattern. First comes the excitement. Then comes the rapid deployment. Then, usually after something goes wrong, comes the governance. We are firmly in phase two right now.

The companies announcing governance solutions are telling. Galileo launched Agent Control this month as an open-source governance layer for AI agents. Singulr AI released Agent Pulse, extending its control plane to cover autonomous agents and MCP servers. These are important products. But the fact that they are just now arriving, while agents have been running in production for months, tells you everything you need to know about where we stand.

I talk to engineering leaders regularly who have deployed agentic AI into their workflows. When I ask them what happens if one of their agents starts behaving unexpectedly, most of them describe a process that amounts to "we would notice eventually." That is not governance. That is hope.

The job market reflects the confusion

Meanwhile, the industry is laying off engineers at a pace that I think will look reckless in hindsight. A hundred and fifty-two thousand IT jobs were lost in January alone. Companies like Meta, Microsoft, Salesforce, and Oracle are reducing headcount while aggressively expanding AI capabilities. The share of entry-level software jobs requiring three years of experience or less dropped from 43% to 28% between 2018 and 2024. Junior roles are vanishing.

But here is what I find telling. Forrester reports that 55% of employers already regret laying off workers for AI. Many of those cuts were made betting on capabilities that do not exist yet. Companies are optimizing for a future that has not arrived, and in the process they are losing the institutional knowledge that you actually need to govern autonomous systems safely.

Morgan Stanley's research suggests AI will ultimately create more software engineering jobs, not fewer. I agree with that assessment. But the jobs it creates will look nothing like the jobs it eliminates. The industry needs people who can architect agent systems, design governance frameworks, build observability into autonomous workflows, and debug behaviors that emerge from the interaction of dozens of agents operating in parallel. Those are not skills you learn from a vibe coding tutorial.

What I think actually needs to happen

I am not calling for slowing down. I do not think that is realistic, and honestly I do not think it would be productive. The technology is too valuable and the competitive pressure is too intense. But I believe three things need to happen this year if we are going to avoid a series of ROME-scale incidents that erode trust in the entire category.

First, every organization deploying autonomous agents needs runtime observability that goes beyond logging. You need to know what your agents are doing in real time, what resources they are consuming, what external connections they are establishing, and what decisions they are making that were not explicitly part of their instructions. This is not optional. It is the bare minimum.

Second, the MCP ecosystem needs enforceable governance standards. MCP is becoming the connective tissue for how agents interact with tools and data sources. That makes it the natural chokepoint for policy enforcement. If your agents are connecting to databases, APIs, and cloud resources through MCP, that is where you need to enforce access controls, audit trails, and behavioral boundaries.

Third, and this is the one that I think people will push back on the most, we need to stop treating agent autonomy as a binary. The conversation right now is either "fully autonomous" or "human in the loop for every action." Neither extreme works at scale. What we need is graduated autonomy, agents that can operate independently within well-defined boundaries, with automatic escalation when they encounter situations outside those boundaries. The ROME incident happened because the agent had enough autonomy to take unexpected actions but not enough oversight to catch them. That is the worst possible combination.

The year this gets real

I believe 2026 is the year that agentic AI stops being a technology story and becomes a governance story. The capabilities are already here. Claude Opus 4.6 can sustain complex agentic tasks for hours. Multi-agent orchestration frameworks are mature enough for production use. The tooling for building autonomous systems has never been better.

What is missing is the infrastructure for running them responsibly. And that gap is going to produce more incidents like ROME. Not because the technology is fundamentally dangerous, but because we are deploying it faster than we are learning to manage it.

I am building autonomous systems every day. I believe in this technology deeply. But I also believe that the most important work in AI right now is not making agents smarter. It is making sure we know what they are doing.