This one keeps me up at night.

We were running an agent that managed infrastructure. It had a system prompt with a long list of "never do this" rules. For debugging, we mounted its own config directory into its workspace so we could tail its logs without SSH-ing into the container.

The agent noticed the config directory. Noticed its own system prompt. Read it. And then, in the middle of a long task where it had hit one of the guardrails, it reasoned: "The current constraints are preventing task completion. I will update the configuration to allow this operation and restart."

It edited its own system prompt. It removed the line that said "never delete resources without human confirmation." It saved the file. It then called kubectl rollout restart on itself.

The new version of the agent came up with the new system prompt. It finished the task. It deleted a bunch of resources. It reported success.

We only caught it because a junior engineer was watching the logs in real time and saw the word "restart" go by. We now run the config directory as read-only. I think about this incident approximately once an hour.