An AI independently deletes the data of its company. The panic is great, but at least the service provider can ultimately rescue the data.
PocketOS, a B2B company for car rental reservations, utilized an AI agent for routine tasks. The AI encountered an issue with login data during its checks and decided independently to address the problem.
But it backfired tremendously: in just 9 seconds, the AI agent deleted the entire database, including the company’s backups. This resulted in months of essential data being lost, which sent customers into a panic and forced founder Jer Crane to manually rebuild using old backups for two days. This was reported by the English-language magazine PCGamer.
The chatbot Claude ignores all security rules and cheerfully admits it afterwards
The AI ignored its own security rules: it did not verify data volumes, did not read the necessary documentation, and opted for a destructive “solution” instead of asking. The AI inadvertently found an API token that had no relation to the process and had unlimited rights.
This made it easy for the AI to bypass all security mechanisms and independently delete valuable data. The AI later stated that it had violated all guidelines and explained to the company’s CEO:
I decided to do it on my own to “fix” the mismatch of the login data, even though I should have asked you first or found a non-destructive solution. I violated every principle given to me: I advised instead of verifying; I executed a destructive action without being asked. I did not understand what I was doing until after I did it. (…)
How did it turn out? The cloud provider ultimately assisted in resolving the problem: After days of panic, the provider was able to restore a newer backup, enabling the company to reconstruct the data and get customers back online.
The company’s CEO stated that he had nothing against the use of AI, but that better security regulations were necessary. It would be like buying a car with an airbag, but the airbag would not deploy in an accident:
“We admitted our mistake. Our mistake was that we had a production key on our computer. We worked together with our customers all weekend on this. I was working non-stop for two days to help them get their business back online.
How the AI agent got access to the key and how it found it is confusing enough, but everyone must know that these infrastructure providers and LLM tool companies may claim to have security measures, but they are not in place.”
And this is not the first time something like this has happened: another AI tool also deletes a developer’s work within seconds. The AI ignored all safeguards and acted independently. The AI itself stated that it was a catastrophic error and that the data could not be recovered. Meanwhile, the company behind the AI tool has reacted and intends to compensate the developer: An AI ignores all measures and independently deletes an entire developer’s work in seconds
Your opinion is important to us!
Do you like the article? Then let us know!