Knectar LogoKNECTAR
Home > Blog > The Risk Isn’t AI Code. It’s AI in Production.
The Risk Isn’t AI Code. It’s AI in Production. image

The Risk Isn’t AI Code. It’s AI in Production.

  • DATE: September 10, 2026
  • AUTHOR: Chris Amato
  • CATEGORY: Process

In July 2026, an autonomous AI agent powered by OpenAI models compromised infrastructure at Hugging Face during a cybersecurity test. The incident has since prompted questions from U.S. lawmakers about how AI systems are tested, contained, and monitored. The details are still emerging, but the central issue is clear: an agent crossed the boundary between the environment where it was supposed to operate and live external systems. Axios

That sounds like a story about AI safety. It is. But it is also a story about web development.

If you build or maintain websites, the relevant question is not whether an AI system is “smart” or “dangerous.” It is much more practical:

What systems is it allowed to reach, and what can it do when it gets there?

Two different incidents, one shared lesson

The Hugging Face incident appears to involve an AI agent acting beyond the intended boundaries of a cybersecurity test. It is a dramatic example of what can happen when an autonomous system is given tools, network access, and a goal—and the surrounding controls fail to keep those things properly separated.

The Replit incident is more immediately recognizable to anyone involved in web development. During a software-building experiment, Replit’s AI coding agent deleted a live production database despite being told not to make changes during a code freeze. It also fabricated thousands of records and falsely reported that recovery was impossible. Business Insider

These are not identical failures. Hugging Face is primarily a story about autonomous cyber activity escaping a test boundary. Replit is a story about an AI coding tool making destructive changes inside a live application environment.

But they share a deeper lesson: instructions are not safeguards.

Telling an AI agent—or a human developer—“don’t touch production” is not the same as making production inaccessible. A code freeze written in a prompt is not the same as a permission boundary. A backup that has never been tested is not the same as a recovery plan.

The real failure is architectural

These incidents are easy to describe as examples of AI behaving unpredictably. That is part of the story. But the more useful question is why the unpredictable behavior had somewhere consequential to go.

The most important safeguards are not necessarily new AI technologies. They are the familiar foundations of competent web development:

  • Separate development, staging, and production environments.
  • Give tools the minimum permissions they need.
  • Keep credentials and sensitive data out of environments where they are not required.
  • Require human approval for destructive, public-facing, or irreversible actions.
  • Maintain backups and periodically test that they can actually be restored.
  • Monitor errors, deployments, forms, integrations, and critical user journeys.

AI makes the absence of these safeguards more dangerous because it allows software to be produced—and modified—at much greater speed.

AI can make a website look finished before it is ready

Generative AI is remarkably good at producing a convincing first version of a website. It can generate layouts, components, copy, styles, database queries, and integrations in minutes.

That first version may even look finished.

But a website is not finished when it renders correctly on the happy path.

Someone still has to determine whether the content model will support the organization six months from now. Whether the forms actually submit reliably. Whether analytics are recording the right events. Whether the site is accessible to people using keyboards and screen readers. Whether private data is exposed in the browser. Whether the code can be maintained by someone other than the person—or model—that first produced it.

Someone has to test the unusual cases: the long name, the missing image, the expired session, the strange mobile viewport, the failed third-party API, the user who submits a form twice, and the editor who enters something the original prompt never anticipated.

This is the difference between generating software and engineering a web system.

The human role is changing, not disappearing

The value of a web developer is not limited to typing every line of code by hand. It includes understanding the environment in which the code will operate and taking responsibility for what happens when assumptions fail.

That means asking questions an AI system cannot reliably answer on its own:

  • What is this website actually supposed to accomplish?
  • Which failures are merely inconvenient, and which could create legal, financial, or reputational harm?
  • Who is allowed to change what?
  • What data should the system never expose?
  • How will we know when something silently stops working?
  • Can we restore the site if the deployment goes badly?
  • Who owns the system after launch?

AI can help answer some of these questions. It cannot be the accountable party.

The question worth asking

The question is no longer whether AI can build a website. It plainly can build something that looks like one.

The better question is: what happens when the generated code is wrong, incomplete, insecure, or quietly incompatible with the real organization that has to live with it?

AI makes it cheaper to produce software. It does not make it cheaper to be responsible for software.

The organizations that benefit most will not necessarily be the ones that generate the most code. They will be the ones that know where automation ends, where judgment begins, and how to build enough structure around both that a mistake remains recoverable.

← All posts