OpenAI launches "embedded assessments" with 4 outside groups
OpenAI said on Nov. 16 (early Nov. 17 Japan time) on its official X account that it is working with 4 outside AI safety research groups on "embedded assessments." The partners are Apollo Research, METR, Transluce and Goodfire. The company said it is giving outside experts deep access to its internal models and systems to investigate whether AI behaves as its developers intend (alignment) and to test its safety measures (safeguards).
We’re working with @ApolloResearch, @METR_Evals, @TransluceAI, and @GoodfireAI on embedded assessments: giving outside experts deep access to our internal models and systems to investigate alignment and test our safeguards. These assessments build on our ongoing alignment and safety research, bringing independent scrutiny to how we identify and manage risks in our systems.
"Embedded" refers to outside evaluators working from inside the development process. OpenAI positions the assessments as an extension of its in-house alignment and safety research, bringing independent scrutiny to how it identifies and manages risks.
Greg Brockman, the company's president and co-founder, quoted the announcement on his own X account and wrote "launching our first round of embedded assessments."
The 4 groups also made announcements on their official X accounts in the early hours of Nov. 17 Japan time, describing time frames, scope, access conditions and plans to publish reports. OpenAI's posts do not mention such conditions or who bears the costs.
OpenAI linked to its official blog in the announcement, but ORIENS could not retrieve the text of the blog post (HTTP 403). This article is based only on X posts by OpenAI, the 4 groups and Brockman.
The roles OpenAI described for the 4 groups
In posts following the announcement, OpenAI described the role of each group.
Apollo Research will assess whether the models could covertly work against their developers' intentions, and whether the safeguards would detect it. It will examine evidence from training, safety testing and internal use.
METR will investigate behavior that departs from intended goals in OpenAI's largest reinforcement learning run to date (training that improves a model through trial and reward). That includes failures that existing tests and monitors may have missed. OpenAI said it has also preliminarily agreed with METR to conduct a broader assessment of its monitors, pending mutual agreement on scope.
Transluce will assess the quality and coverage of the alignment evaluations OpenAI conducts after training its models. It will develop additional tests and recommend improvements to fill gaps in the evaluations. According to OpenAI, Transluce intends to expand the pilot in 2027.
Goodfire will embed interpretability researchers (who study how a model works internally) at OpenAI. They will use Goodfire's platform to analyze the models' internals, focusing on training run monitoring, debugging and alignment.
| Group | Role described by OpenAI | Time frame and publication described by the group |
|---|---|---|
| Apollo Research | Whether models covertly work against developers' intentions, and whether safeguards detect it | 6 evaluators for 4 months; 2 reports |
| METR | Behavior departing from intended goals in the largest reinforcement learning run; broader monitor assessment preliminarily agreed | Scope limited to 1 reinforcement learning run; 1 or more reports |
| Transluce | Quality and coverage of post-training alignment evaluations; intends to expand the pilot in 2027 | 6 weeks onsite from early December; public report |
| Goodfire | Embedded interpretability researchers analyze model internals | No time frame given; will publish findings as work progresses |
Apollo embeds 6 evaluators for 4 months; disclosure rights do not yet meet its bar
In its own posts, Apollo Research described the effort as a pilot toward embedded evaluations. 6 evaluators will work inside OpenAI for 4 months, investigating whether models attempt to undermine oversight and control of themselves. It said the work builds on its prior research into OpenAI's training process.
The group argues that such evaluations must be credible without relying on trust in the developer, which requires strong guarantees for access, disclosure and meta-transparency (making the conditions of the evaluation itself public). It said the pilot is an ambitious first step toward that goal, but its access and disclosure rights do not yet meet the group's bar for full embedded evaluations.
On access, it said OpenAI has committed, for the training assessment, to providing relevant code, supervised fine-tuning (SFT) data, reinforcement learning records including reasoning traces and rewards, and intermediate models that can be queried. Access details for the other assessments remain to be agreed. OpenAI can withhold materials it deems necessary for technical, security or intellectual property reasons, but not simply because they are unfavorable, and it must disclose that withheld items exist and give written reasons.
The group also disclosed constraints. It may discuss confidential information only with staff selected by OpenAI, and it does not have access to internal communication channels such as Slack that OpenAI's safety researchers use. Formal conversations with approved staff must be logged.
On publication, it will issue 2 reports, one on training incentives and one on evaluations, internal use and detectability. OpenAI may request redactions of confidential information, including evidence behind the conclusions, but not simply because a finding is unfavorable. Information based on conversations with staff can be published only if OpenAI provides formal attestation or the group verifies it independently. The group said it agreed with OpenAI on strong rights to publicly explain the access it requested and received, what was withheld or delayed, the impact of redactions, and any pressure it believes affected its work.
We're launching a pilot towards embedded evaluations with @OpenAI. Six evaluators from Apollo Research will work within OpenAI for four months, investigating whether models attempt to undermine their own oversight and control.
METR limits scope to 1 reinforcement learning run, narrower than its proposal
METR said in its own posts that it reached an agreement with OpenAI to conduct a limited alignment assessment. It will investigate the alignment properties of OpenAI's largest reinforcement learning run. It will publish 1 or more reports describing its findings and process.
The scope is limited to that single reinforcement learning run and does not cover the incidents of behavior departing from intended goals that OpenAI has disclosed so far. METR acknowledged that this is narrower than the scope it had previously proposed, and said it hopes the work will provide new evidence about alignment.
OpenAI will provide access to transcripts, monitor records and snapshots of the model throughout reinforcement learning. METR can use interviews with OpenAI staff to guide its work, but its reports cannot quote or cite individual views.
METR's posts do not mention the preliminary agreement on a broader monitor assessment that OpenAI referred to in its post. METR also published on Nov. 5 (Nov. 6 Japan time) a report of an independent assessment using internal data from Anthropic.
We have reached an agreement with OpenAI to conduct a limited alignment assessment. This pilot will focus on investigating the alignment properties of OpenAI's largest RL run. We will publish one or more reports to share our findings and transparently describe our process.
Transluce to work 6 weeks from December, with access short of employee level
Transluce announced a 6-week pilot with OpenAI starting in early December. Its team will work onsite and independently assess the scope and quality of OpenAI's post-training alignment evaluations. It will present its findings in a public report.
On access, it said it negotiated privileged access to some of OpenAI's internal evaluations and to models with raw chains of thought (records of the reasoning a model goes through before answering). However, it said this access is not comparable to employee-level access.
Transluce described the pilot as a first step and said it has agreed with OpenAI to follow up in early 2027 with a more complete assessment of alignment evaluations, relying on broader and deeper access.
Transluce is piloting a six-week embedded assessment with @OpenAI in early December. Our team will work onsite to independently assess the scope and quality of OpenAI's post-training alignment evaluations. We’ll share our findings in a public report
Goodfire to monitor training with its interpretability platform
Goodfire said in an article on X, "Announcing our collaboration with OpenAI," that it will embed researchers at OpenAI. Using its interpretability platform, they will monitor training runs, investigate the causes of undesired behavior and improve alignment. Its researchers will have deep access to OpenAI's internal models and systems, it said.
Goodfire calls this work "model forensics." It will monitor a model's internal signals during training and detect both known risks and newly emerging anomalous behavior. The primary audience for its findings is OpenAI's researchers, who will use them to improve model safety. The company said it will also publish what it learns and its findings as the collaboration progresses.
We’re announcing an embedded assessment with @OpenAI to advance safety and alignment at the frontier.





