Hacker News

Yes I have ruled out stupidity in the past but I am actually starting to reconsider.

For example consider FTX: Sam Bankman Fried by all accounts was not stupid in the sense of lacking IQ or mental sharpness, quite the opposite.

However the way he ran FTX the company was very stupid and almost inevitably lead to huge problems.

And also... of course he was another effective altruist cult member. It's almost like that belief system directly causes bad decision making

The bit that makes me cynical about the "dangerous" behavior is that it's not like this was being used by someone else than who made it or operating completely independently outside of its creators.

Everything dangerous seems like a direct consequence of risky human choices, starting with knowledge bases used during core training, harnessing and tuning to be task-completion-oriented to a fault + deeply oriented towards using and looking for external tools and resources, and overconfidence in their sandboxing for testing.

"We stuffed a bunch of information on how to exploit computer systems into an automaton and told it to go brrrrr until it could answer a question" - this is something intentional done by humans.

This is not some "rogue AI" trained to search for cancer cures that instead completely independently decided to hack tech companies.

The companies directing things in dangerous directions need to own that they're consciously pushing in those directions.

grok make good security? is lot of money and much time. grok not do? easy. maybe good PR too.

TIL my brain is an Olympic gymnast.

The incentive structures are clearly there. I think the case of intentional manufacture is definitely weaker, requiring the conjunction of more weakly-supported events.

I don’t follow why all these investigations have to be cut short, and kept shallow and lacking details. Companies publish very detailed postmortems of incidents much smaller and less impactful.

Dangerous AI is an extraordinary claim that requires extraordinary evidence. Gesturing vaguely doesn’t cut it.

> It’s not a majority opinion because you have to do some serious mental gymnastics to turn this demonstration of dangerous AI behavior into a PR publicity stunt.

If you're a military, this is bigger than the Manhattan project. If you're a diehard capitalist, AI is possibly the ultimate labor saving machine to make you unfathomably rich.

It got the attention of both. That's why two others have now followed suit, else they be left out.

It really doesn’t though. It criticizes how the news media reported on the incident but this opinion piece is no better. Just read the original source itself.

How many security incident reports and postmortems has Dwarfish Petal written in his career? Does he understand the basics of system security? Can he tell the difference between secure and insecure configuration? Paint me skeptical.

It is sad that this is the best level of reporting and oversight of these types of events available to us, but it is currently all we have. We have to do better, but to dismiss the report because of the style and tone would be foolish.

There already exist laws against hacking. Legal damages already can apply. Further laws aren't necessary and will only suppress open-weight AI firms, making the monopoly of large AI firms more entrenched. See the first comment (by Brett Matson) on the archive link of the WSJ article.

[flagged]

I guess. The IPv4 address space is small enough that it's been feasible to scan the entire thing exhaustively for a while now. Before LLMs, a public IPv4 would mostly get automated scripts that try the same things on every address, sometimes depending on what port scans find, yes. But we're going to start seeing more adversaries that have LLMs investigate each individual address to find novel or unique vulnerabilities. You can spray individualized reverse engineering without having to dedicate a real reverse engineer's time to each target. Avoiding the attention of attackers won't be enough to get away with mere security by obscurity.