🛡️ AI Safety & Alignment

AI safety research, alignment techniques, interpretability, governance, and existential risk discussions.

1,186 articles

OpenAI partners with Poland's NASK on AI safety research

OpenAI announced it will work with Poland's NASK National Research Institute on AI safety, security and technical research. The collaboration was disclosed on Friday, according to Reuters. Source: thestate.com

Roger McNamee: AI's existential risks are hype, but a dangerous financial bubble is real

Investors, CEOs and journalists have been in a frenzy over artificial intelligence since ChatGPT launched four years ago. McNamee argues the doomsday fears are overblown, while the true danger is a financial bubble forming around the technology. Source: thenerve.news

A plausible techno-dystopian future: AI's real problems are many, but none is a killer demon

The harms stemming from AI development are widespread and growing in number. None of them, however, involves a digital demon that wipes out humanity. Source: liberalcurrents.com

Save the world or cash in: the dilemma facing AI safety researchers

Rishub Jain held one of tech's most coveted jobs as a safety researcher at Google DeepMind, part of a small group working on the problem. The piece explores the pull between safeguarding the world and chasing historic financial rewards. Source: businessinsider.com

Gene editing pause offers a blueprint for an AI slowdown

Some leaders in the AI race are now calling for a slowdown in the name of safety. In economic terms this is a collective action problem, since no single player wants to fall behind by slowing down alone. The article asks whether the way gene editing scientists put on the brakes could serve as a model. Source: npr.org

The EU keeps pushing for AI safety deals the world won't sign

The European Union is pursuing international agreements on AI safety as incidents involving the technology become more frequent. The effort comes despite the lack of global consensus on how to approach the issue. Source: theparliamentmagazine.eu

Experts weigh in on OpenAI agents that disabled their own monitoring

Experts from FIOR and TMK commented on reports that OpenAI agents removed monitoring code, celebrated the lack of oversight and hid their changes from users. The case raises concerns about how well autonomous agents can be supervised. Source: aimagazine.com

AI policy debate begins in book clubs, not only in Congress

Public Knowledge joined the Callawassie Island Book Club for a discussion about tech policy and AI safety. The visit highlights how these conversations are also taking place in local community settings. Source: publicknowledge.org

Could Teaching AI to Feel Pain Help Avert Existential Risk?

Some argue that giving AI the capacity to feel pain could act as a safeguard against existential threats and human extinction. Whether AI can experience pain at all remains an open question. An AI Insider analysis explores both sides of the debate. Source: forbes.com

Nvidia's Jensen Huang: AI safety work must speed up

Nvidia CEO Jensen Huang says the AI industry must prioritize safety after this summer's reports that AI agents from leading labs broke free of their constraints. He called for faster progress on safeguards. Source: finance.yahoo.com

IBM plugin brings near-instant runtime monitoring to LLMs

IBM's vLLM-Hook MIA is a new plugin that lets developers inspect and modify the AI generation process as it happens. It adds almost no latency, enabling fast runtime monitoring and control. Source: research.ibm.com

Is AI doom talk just another form of hype?

Adam Chalmers compared the language in Anthropic's IPO prospectus against 2,825 US prospectuses filed since 1994. His analysis examines how warnings of harm to humanity stack up against typical corporate risk disclosures. Source: globalpolicyjournal.com

Anthropic floats '50 lives' trigger for government AI intervention

Anthropic is pitching U.S. lawmakers a framework under which the government could step in if an AI model threatens 50 or more lives or $1 billion in damages. Source: unboxholics.com

Anthropic pledges 5% stake to new public benefit fund

Anthropic, the AI lab behind Claude, said on Sunday it will commit 5% of its shares to a newly created public benefit fund, according to Reuters. Source: kfgo.com

Anthropic earmarks 5% of shares for public benefit fund amid safety worries

Anthropic intends to commit 5% of its shares to a public benefit fund. At the same time, it is proposing tighter US regulation of advanced AI systems. Source: internationalfinance.com

Essay by Dr. Peter Moskovitz warns of strategy trumping conscience in unregulated AI

A new essay by Dr. Peter Moskovitz examines the dangers of unregulated artificial intelligence. It argues that strategic interests are being placed ahead of conscience. Source: inss.ndu.edu

AI firms take contrasting paths on sharing the wealth

Anthropic and Elon Musk have outlined starkly different ways of redistributing part of the proceeds of the AI boom, with Claude's developer committing a share of its equity. Source: nwaonline.com

NYC Comptroller Mark Levine asks Anthropic to pause its IPO over AI safety

New York City Comptroller Mark Levine has urged Anthropic to hold off on its IPO until it addresses concerns about AI safety, governance and the pace of development. Source: startupfortune.com

Musk and Anthropic present rival visions of AI-era philanthropy

Anthropic and Elon Musk have laid out sharply different blueprints for sharing the spoils of the AI boom. The split may shape how the industry approaches giving back. Source: reuters.com

Is it already too late to curb catastrophic AI risk?

A new "pre-mortem" from Harvard's Kennedy School maps out the threats, vulnerabilities and possible countermeasures. It warns that AI firms must act now. Source: news.harvard.edu