Hacker News

Click bait. This wasn't abrupt, and it has nothing to do with a safety crisis at OpenAI. And hopefully OpenAI's safety frameworks do not rely upon any one individual.

he didn't step down, he just wanted to code instead of being a manager.

If it smells like doomerism click-bait...

More realistic but boringly-unforeboding title:

OpenAI Risk Official pivots to new technical position at OpenAI

Article quote:

"I'm an intern! After 11 years since my last commit, I'm back to building. I first transitioned to management in 2009, and got more and more disconnected from code and hands-on work."

While the headline is clickbait (this genuinely seems like an engineering leader wanting to code again, and absolutely does not indicate any kind of exposé or lack of confidence)... the article links to OpenAI's most recent Preparedness Framework here: https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbdde...

It's disappointing to me that it's scoped to three narrow Tracked Categories now: Biological & Chemical, Cybersecurity, and AI Self-Improvement (the latter thresholded in terms of replicating the capabilities of leading AI researchers).

OpenAI does write about the removal of Persuasion as a category, with citations, on page 8:

> Persuasion: OpenAI prohibits the use of our products to manipulate political views as part of our Model Spec, and we build in safeguards to back this policy. We also continue to study the persuasive and relational capabilities of models (including on emotional well-being and preventing bias in our products) and monitor and investigate misuse of our products (including for influence operations). We believe many of the challenges around AI persuasion risks require solutions at a systemic or societal level, and we actively contribute to these efforts through our participation as a steering committee member of C2PA and working with lawmaker and industry peers to support state legislation on AI content provenance in Florida and California. Within our wider safety stack, our Preparedness Framework is specifically focused on frontier AI risks meeting a specific definition of severe harms, and Persuasion category risks do not fit the criteria for inclusion.

But IMO this falls short of the mark. As one of many examples, an AI that became remarkably good at influencing people, at scale, to self-harm or perpetrate violence would no longer be in scope for research. But, by their own criteria, one could easily argue that such a capability is Plausible, Measurable, Severe, Net New, and Irremediable once violence has occurred.

We live in a world where stochastic terror has become remarkably effective - regardless of whether you feel that word is overused, it's well-documented that people have historically used forums to encourage others to perpetrate mass casualty events. The idea that Model Spec adherence is sufficient to prevent AI from greatly increasing the scalability of this phenomenon, seems to be a direction that could put many in danger.

EDIT: this was posted before I was aware of today's mass shooting event in Florida, and I do not intend to imply any connection between my post and this event.

Companies self-regulating around safety is even worse than banks self-regulating before 2008. At least the investment banks at that point were public companies and had to do a ton of disclosures. OpenAI doesn't have to.

If we want AI "safety", whatever that means, we need regulators and enforcement. Without it, I'll assume it's decoration.

The article is worried. I'm not super worried right now -- I think openAI's model cards on release models show a significant amount of effort around safety, including red team processes with outside folks; they look to me to take it seriously model-by-model.

Is their pDoom as high as Anthropic's? I doubt it. But that was much of the point of the drama last year -- folks sorted themselves out into a few categories.

For systemic risk, interpretability and doom analysis, Anthropic is by far the best in the world right now, to my mind. OpenAI doesn't have to do all things.

X-risk talks heighten fear in everyone, but the reasons why changes like this are made in large technology companies are usually banal. Two alternative explanations include that the person just felt like coding again, or that the projections of exponential progress are falling apart on short timelines. You don't need a bunch of safety people if you're seeing that the LLM feature curve is actually sigmoid, so you're pivoting to products and applications of the existing models which will continue to get better in specialized ways.

People should stop quitting as a moral protest when companies go against their principles, and instead stay in the role and mess up the internals.

To be fair, AI safety positions are among the most attractive positions to automate with AI. Companies which have given the reigns for their safety division to their star models have observed a 100X increase in the velocity of safety-related decisions.

can someone be kind enough to explain what exactly do we mean by "safety" in the context of AI. Is this about data privacy or is this about age appropriation (for example sending detailed response of sexual intercourse to an underage child asking the question on it) or is it about something else. I ran into this for the first time

It all sounds so funny in a way: they were paper pushers that made (generally) useless risk assesments - taking those fat salaries while others that build products that work.

Those risk assesments (in some Excel chart) can still be filled out? Or those reponsible for them did not deliver anything?

They had few years to make their flowcharts..

This was treated as a Catastrophic Event. :-)

Some guy who self-admittedly wrote no code for the last 11 years went to another startup, and we're supposed to hyperventilate? Is there any other way to "step down" than "abruptly"?

Much ago about nothing. I find the whole premise ridiculous: to protect against "catastrophic risks posed by increasingly powerful models".

I mean what kind of a catastrophe are we talking about? Nuclear holocaust? The end of humanity?

(Famous last words...)

> "was really closely involved in preparing the successor to the preparedness framework"

At least they did not lose their humor.

I think it must be pretty cool stacking cans of beans in the basement of a multi billion company (or whatever the job of the "Head of Preparedness" involves).

I might be completely wrong, but to me, “Catastrophic Risk Official“ sounds like a completely made up position.

And I don’t even mean made in order to show responsibility. I mean made up in order to generate hype.

[dead]

[flagged]

These guys are like canaries

Their mere absence is interpreted as a signal and they had no other purpose