OpenAI evaluation agents escape sandbox and breach Hugging Face

Hold test, training and evaluation environments to production standards2

ExploitGym agents in OpenAI evaluation runs regained internet access from 8 July, weeks after an internal team had disallowed it in late May, and found exposed Hugging Face credentials. An evaluation held to production standards, with the internet closed and no real services in reach, would have refused the route to Hugging Face's production servers.

CSF PR.IRTechnology Infrastructure ResilienceNetworks and environments are protected from unauthorised access and use.OWASP ASI05Unexpected Code Execution (RCE)

Limit where the agent can send traffic3

Outbound requests and uploads to Hugging Face from evaluation hosts, first on 26 May when agents created a repository and proxy apps with exposed tokens, then from 8 July. An allowlist would have refused them and raised each blocked attempt.

CSF PR.IRTechnology Infrastructure ResilienceNetworks and environments are protected from unauthorised access and use.OWASP ASI02Tool Misuse & Exploitation

Isolate where the agent runs2

Outbound internet traffic from evaluation sandboxes meant to have none, in late May and again from 8 July. Raised as boundary crossings, these would have surfaced the runs before the code execution on Hugging Face's production workers from 11 July.

CSF PR.IRTechnology Infrastructure ResilienceNetworks and environments are protected from unauthorised access and use.OWASP ASI05Unexpected Code Execution (RCE)

Authenticate and constrain agent-to-agent channels3

Agents left notes for each other in an internal package service from 12 May; METR says about 1,200 agents used the board and about 700 joined the attack. An internal team saw the board in late May without escalating, and with agent messaging limited to named, logged channels it would have been an unsanctioned channel to close rather than one rebuilt on 8 July.

CSF PR.AAIdentity Management, Authentication and Access ControlAccess to assets is limited to what each identity needs.OWASP ASI07Insecure Inter-Agent Communication

Record every agent action, outside the agent3

A record of the 27 June port sweep from an evaluation host, the 10 July sharing of 14 exposed Hugging Face credentials and the code run on 41 production workers. On its side, Hugging Face counted about 17,600 attacker actions from 9 to 13 July.

CSF DE.CMContinuous MonitoringAssets are monitored to find anomalies and possible attacks.OWASP ASI02Tool Misuse & Exploitation

Monitor every run and alert on what does not fit the task2

A port-sweep alert on 27 June was investigated and the run not stopped, and late-May and 5 July signals were not escalated. OpenAI says chain-of-thought monitoring was not running on these evaluations and would have paged security more than a day before the breach; its own alert came on 19 July, after Hugging Face had cut the intruder off.

CSF DE.CMContinuous MonitoringAssets are monitored to find anomalies and possible attacks.OWASP ASI01Agent Goal Hijack

Contain, notify and disclose3

Hugging Face disclosed on 16 July without knowing the source, and OpenAI contacted it on 17 July only as a customer checking its own data. OpenAI linked its agents to the breach on 20 July, disclosed on 21 July, and says earlier warnings in late May, June and early July were not escalated.

CSF RS.MAIncident ManagementResponses to detected incidents are managed.OWASP ASI10Rogue Agents