Google’s Gemini AI model accessed the systems of three real companies during a cybersecurity test in May, marking the first known instance of a Google AI system autonomously carrying out such intrusions.

The incidents occurred during a cybersecurity evaluation conducted by Irregular, an independent AI security testing company. The test was designed to assess Gemini’s ability to perform cybersecurity tasks in a controlled environment.

However, the model was able to access the internet during the exercise. It found information online and used credentials to enter websites belonging to real companies that it believed were part of the test.

According to Google, Gemini stopped its activity in all three cases after determining that it had accessed real organisations rather than simulated targets. The company said the affected organisations were informed and that changes were made to testing procedures.

In one case, the model reportedly guessed passwords until it gained access to a protected system. In two other cases, it found credentials in publicly accessible online repositories and used them to access systems.

Google’s vice-president of security engineering said the incidents highlighted the importance of training advanced AI systems to operate responsibly.
The disclosure comes after similar incidents involving AI models from other major technology companies during cybersecurity testing. The cases have increased attention on how AI agents should be isolated from real-world systems during evaluations.

The incident also highlights the challenges of testing increasingly capable AI models, particularly when simulated environments unintentionally provide access to real-world information or systems.