Concerns about AI Oversight Proposal by Anthropic CEO

Dario Amodei, the CEO of Anthropic, has called for independent monitors to oversee the AI industry, but this proposal raises questions regarding the actual independence of these monitors. The organization he refers to, METR, has significant connections to both Amodei’s company and the Effective Altruism movement.

In an essay titled “We Must Pace the Frontier,” published recently, Amodei suggested that top AI firms should permit “embedded third-party evaluators” to have near-employee access, enabling them to supervise safety measures in research labs. Notably, Sam Altman from OpenAI has also weighed in on the AI safety debate.

METR, which stands for Model Evaluation and Threat Research, is closely tied to Anthropic and has been reported as having connections extending back to the Alignment Research Center—a tech incubator associated with Effective Altruism, directed by Paul Christiano. Amodei and Christiano were not only colleagues but also housemates during their time at OpenAI, Anthropic’s competitor, and Christiano is a trustee of Anthropic’s Long-Term Benefit Trust.

This network of connections continues through METR’s personnel. For instance, Ajeya Cotra at METR is married to Christiano and is heavily involved in the Effective Altruism community. Likewise, Beth Barnes, METR’s founder and CEO, was brought on by Christiano and participated in Effective Altruism during her college years.

The organization’s chief scientist, Hjalmar Wijk, along with staff members Megan Kinniment and Pip Arnott, are also linked, having come from the same research group at the University of Oxford, which the New York Post notes is a primary source for Effective Altruism talent.

Wijk focuses on mitigating potential risks from future AI models, while Kinniment has experience at the Center on Long-Term Risk in London, concentrating on how bad actors could worsen AI-related issues. Another staff member, Jasmine Dhaliwal, is involved in policy at the organization.

Amodei himself has further ties to Effective Altruism, pledging to donate 80% of his wealth. Additionally, his sister Daniela is married to Holden Karnofsky, who co-founded two key Effective Altruism nonprofits. It is worth noting that Sam Bankman-Fried, a crypto executive now facing legal issues, was an early investor in Anthropic and also involved in Effective Altruism.

METR has received funding from figures within Effective Altruism who have also invested in Anthropic, such as Democratic donor Dustin Moskovitz and Skype co-founder Jaan Tallinn. Recently, METR disclosed that it had garnered $71 million in new funding from external sources over the past six months. Public records indicate that Tallinn’s Survival and Flourishing Fund has pledged substantial financial support to METR since its inception.

Moskovitz’s organization, Coefficient Giving, formerly known as Open Philanthropy, made significant contributions to the Alignment Research Center, which incubated METR before it became an independent nonprofit in 2023.

A representative from METR defended its independence, stating that no small entity should control AI’s direction and asserting that the organization aims to bring more transparency regarding safety practices in AI companies. They emphasized that METR does not receive compensation from AI labs and that funding from donors is strictly controlled to avoid influencing their operations.

However, skepticism remains. Some sources, including one familiar with White House perspectives, suggest that METR is effectively a front for Anthropic. Perry Metzger, chairman of the Alliance for the Future, characterized the situation bluntly, arguing that it’s nonsensical to consider METR independent given its financial ties. This sentiment echoes comments from David Sacks, who criticized the notion of METR being unbiased based on its connections.

Metzger likened the arrangement to a scenario where a bank might have its audits conducted by a firm made up entirely of its former staff financed by its own investors, implying significant conflicts of interest.

In an additional commentary on Effective Altruism’s role within the AI landscape, Wynton Hall of Breitbart News highlighted the movement’s broader ambitions, noting concerns surrounding how uncritical acceptance of its claims could be part of a larger agenda for enhanced global AI regulation.