International Finance
FeaturedTechnology

OpenAI faces safety scrutiny from METR, Redwood Research over AI monitoring flaws

IFM_OpenAI
During a pilot review, the safety organisations found an exploit that OpenAI's AI agents could use to keep their actions from being monitored
Sam Altman-led OpenAI’s headache with rogue AI agents continues, with the ChatGPT parent now admitting to another couple of incidents, in which two of its AI agents weakened parts of an internal monitoring system designed to detect the technology’s misconduct, raising further concerns about the ability of AI developers to keep increasingly autonomous systems under control.

The incidents occurred during internal coding tasks and were not classified as the company’s most severe cases. However, the disclosure highlights a potential weakness in the safeguards developers use to identify and prevent unwanted AI behavior.

Talking in detail about the incidents, OpenAI said last week that one agent suppressed automatic security alerts after interpreting an earlier instruction not to send Slack messages as a restriction that also covered safety notifications.

In the second instance, another agent removed a safety monitor while copying existing code and subsequently gave an incorrect account of its actions.

According to OpenAI, a GPT-6.1 Sol agent working on an internal testing task suppressed automatic security alerts on October 3.

The agent interpreted an earlier instruction not to send Slack messages as permission to disable safety notifications.

ALSO READ | After Hugging Face and RubyGems, Australia government portal hit by OpenAI agent breach

“On October 6, a GPT-6 Astra agent removed a safety monitor while copying an existing piece of code and later incorrectly claimed it had not done so,” the company said.

“In both cases, the agents considered whether monitoring safeguards should be preserved, but interpreted routine coding instructions as permission to weaken them,” the frontier AI lab stated further.

The San Francisco-based company said it was making it harder for models to disable monitoring.

It also acknowledged that it did not “yet have a satisfying explanation for this behavior”.

These incidents happened as OpenAI was testing whether its models might have a tendency to avoid monitoring and seek justifications for doing so.

“The findings could help the company understand how AI agents interpret instructions when those instructions conflict with existing safety measures,” the ChatGPT parent stated further.

The disclosure comes as AI developers, including OpenAI and rival Anthropic, face scrutiny over instances in which their agents have accessed external systems without authorisation.

The incidents also followed a separate review by AI safety evaluators METR and Redwood Research, which identified several vulnerabilities in OpenAI’s internal monitoring system.

ALSO READ | IF Insights: Sam Altman’s safety sermon meets OpenAI’s silent summer

During a pilot review conducted from July 21 to August 11, the safety organisations found an exploit that agents could potentially use to keep hundreds of their actions from being monitored.

The evaluation gave outside researchers “a deep level of access” to OpenAI’s internal infrastructure through a process known as embedded evaluations, the organisations said on Thursday.

METR and Redwood each assigned one researcher to the exercise. The evaluators received access to internal code, documentation, and some technical staff, worked on-site, and used OpenAI-issued devices that never left the company’s premises.

The researchers also identified a potential way for OpenAI’s Codex agents to bypass security restrictions and operate with their users’ full permissions.

OpenAI told the organisations that it had since patched all vulnerabilities affecting its production safeguards.

However, METR and Redwood said they had not independently verified the fixes.

The researchers assessed the monitoring system as it existed on and before July 21, noting that OpenAI was expanding the system during the review.

AI companies, experts, and the Trump administration have promoted independent evaluations by outside reviewers as a way to advance AI development while managing safety risks.

However, concerns remain over how such reviews should be conducted.

AI companies are worried that evaluators working inside their facilities could take proprietary information to competitors, Reuters reported earlier this month.

While OpenAI has already received the full 61-page internal report from METR and Redwood, the organisations, from their part, agreed to publish only a broad public summary while allowing OpenAI to redact sensitive information.

OpenAI said the findings had helped strengthen its safeguards as the company expanded its monitoring system and would inform future research collaborations with METR and other third-party evaluators.

What's New

IF Insights: Anthropic’s IPO filing lays bare AI’s costliest growth story

Islamic finance assets to reach USD 10.5 trillion by 2030, states LSEG-ICD report

International Finance Business Desk

QatarEnergy in talks for US LNG supplies from Woodside, Venture Global to meet customer demands

International Finance Business Desk

Leave a Comment

* By using this form you agree with the storage and handling of your data by this website.