Hacker News

Now, ask yourself if you want AI taking decisions around your life... it will pull numbers... someone will have a "goal" of 1000 "hits" and there you go, 2-3 months in jail working to check what went wrong...

But don't worry, the other P AI system will ensure that the first one doesn't do mistakes...

Important context is that the context wasn’t even in a connected system. Unclear if the human analysts would have even caught this, or if they would have known to check this unconnected system.

> Some US intelligence analysts took notice, the officials said. Bloomberg reported in June that one analyst detected changes to the site as early as 2019, and logged remarks about the modifications in a system that wasn’t connected to the primary military intelligence database, which informs targeting

"In the final days of February, the Trump administration demanded an overwhelming aerial assault. More than 1,000 Iranian targets were hit within the first 24 hours alone. That order compressed the time frame for finalizing targets, the officials said, instilling an urgency that — combined with gaps in underlying information about the site, cuts in civilian-protection personnel and an overreliance on artificial-intelligence technology — set the stage for the deadly error."

-------------------

Maven is an untested tool for rapidly selecting targets under time pressure during a war. It's reason for existing is speed. If you're in a shooting war, the ability to select good military targets faster than your enemy, but not necessarily with zero error, is an advantage.

Planning a first strike is a different situation. Time pressure should not be a significant factor. Humans should have been doing their due diligence on every single target. If an untested tool for rapid mid-battle target selection was relied upon completely because of a hasty executive order, then the responsibility falls entirely on the executive.

Seems like a scenario where two completely avoidable wrongs....killed 120 children. AI being one of the wrongs.

"A computer can never be held accountable / Therefore a computer must must never make a management decision" https://simonwillison.net/2025/Feb/3/a-computer-can-never-be...

Except laundering our biases & discriminations, the unaccountability seems to be the purpose of a huge huge swarth of AI usage. To eliminate accountability. That sure seems like A Feature Not A Bug to the worlds worst people.

https://www.propublica.org/article/hegseth-civilian-harm-mit...

[dead]

https://www.politico.com/news/2026/03/10/pentagon-iran-schoo...

> The Pentagon chief last year slashed offices that didn’t contribute to his goal of “lethality,” including the group that assists in limiting risk to civilians, known as the Civilian Protection Center of Excellence. Around 200 employees who worked on the issue, including at that office, have been reduced by about 90 percent, according to two current and former officials and a person familiar with the effort. The team that handles civilian casualties at Central Command, which oversees the Middle East, has dropped from 10 to one.

Negligent homicide in pursuit of their DEI goals, basically.

The headline says "contributed to missile strike", not caused it.

can -> can't

Chatbots in general can be quite opaque in how they operate, the biggest issue with agent products is communicating it's capabilities and limitations to users. If you have a bank app it's pretty clear whether or not it can send wire transfers, but a if that's a chatbot, users will have to think to ask it (and even then it could hallucinate)

No, this is what happens when software developers don't understand how users actually use their products.

This was a totally expected outcome, similar to how CV cameras arrest the wrong minorities in neighboring states 100s of miles away because it told the officer it was them; or how health insurance companies use ML algorithms to deny Americans care.

It's going to happen in other sectors of life and it will continue killing innocent civilians.

Small children often develop coping mechanisms for their lack of agency. It may be part of these people’s worldview since early childhood that anyone who opposes them should magically die.

To me it's extremely obvious that this is the desired outcome - no accountability for anyone with this setup.

The new world is complex and AI technology has outpaced the regulation cycle, so it's inevitable that we come to a place where you don't have accountability and can never have it because 0) the courts can't handle the case, they don't have the frameworks for it, so it takes longer time 1) you can overload the courts with slop, 2) by the time there is a decision the world has moved on and technology has moved on

Really good time to do various crime.

Even ignoring the risk of civilian casualties (which to be clear should be weighed heavily) it’s just bad ROI. Like each missile probably costs more than the extra targeting work they should have done.

This is actually a much bigger close call than the hugging face incident. A hallucination almost triggered a possible precursor to WWIII.

Wait... I thought they said they didn't kill children and that was Iranian propaganda.

> That didn't happen.

> And if it did, it wasn't that bad.

> And if it was, it was AI's fault, not mine.

I'll add a check that all individuals within are genuinely adults to avoid this.

C'mon, a Palantir AI isn't gonna talk like Claude.

It's gonna say something like "they were all military-aged males". https://en.wikipedia.org/wiki/Military-age_male

Israel likes to argue that Hamas embeds fighters with first responders, which even if so, they're still responding to injured. But Israel considers this not even collateral damage but justified.

Mind you Israel is currently funding a lobbyist organization in the UK to flag Médecins Sans Frontières (Doctors Without Borders) as a terrorist group (because many years ago a couple of people who had affiliations with other anti-Israel groups were found to be also affiliated with MSF).

Hegseth should be in prison

https://www.propublica.org/article/hegseth-civilian-harm-mit...

Jail? I'm going to bet that absolutely nobody higher up will face any consequences for this. The outcome will be "we will review our processes".

[flagged]

>It's easy to blame the military and the higher-ups

My brother in Cthulu, they're the ones making the life or death decision that killed people, who else should be to blame?

>But more relevant to HN, this couldn't have happened without technology professionals working for Palantir.

If you wanna go the historical tech-tree scapegoat route then none of this would have happened if DARPA wouldn't have invented the internet, or if Thomas Edison wouldn't have invented the lightbulb, or if the fish haven't ventured onto land and evolve into us land mammal apes who invented sharpened sticks as murder objects.

Thank you for sharing the talk.

> I encourage others who work in the public sector, whether directly or through a contractor, to read and internalize Don't Get Distracted by Caleb Hearth.

I know a guy who works for well-known hardware company. He isn't a software dev, but close enough, works with Verilog on hardware design and verification. One day he tells me how he is working on video processing module and he doesn't know if that module will be part of surveillance system, missile guidance or nanny cam. It's possible that even middle management doesn't know more than him, and very likely the module will find uses in multiple products. He is a small cogs in a large machine, maybe his work will save a baby's life, maybe it will kill a person, he cannot know.

Of course it could have happened. You don't even need targeting for this to happen. You can drop bombs at random and kill whoever is unlucky enough to be in the wrong place. The US has a long history of doing it (Japan, Vietnam, ...).

So yeah, the military and whoever decided that a war was a good idea are to blame.

People working at Palantir are as much responsible for the killing of these 123 children than seatbelt manufacturers are responsible for the people who died because their product failed. That is, because they didn't do their job correctly, people died, definitely worth feeling bad about. However, should people stop working in seatbelt manufacturing because of it? Absolutely not, we need seatbelts, because without them, things would be worse.

Palantir, or whoever does target identification, is like the seatbelt of the reckless drivers of the militaries, who, for the lack of precise information, would just carpet bomb the place. It is not a hypothetical, it is how the US has worked before they had precision ammunition.

This is essentially a trolley problem. The war is the trolley, people who work for Palantir are the ones pulling levers. And if there is no one to pull the lever, the trolley is gong on a track with a lot of people on it.

The govt is investing in the military whether you like it or not, there is a set amount of GDP going into the sector regardless of your moral choice.

Would you rather not have someone make good software that does not misidentify schools at sites? Or should we outsource this to worse engineers who don't care about morals and don't find this metric meaningful?

If I write healthcare software, and a doctor orders the wrong drug that kills the patient, am I responsible?

Let's not forget the shareholders who invested in this company knowing what they do. That includes some of my colleagues. I told them that whoever created and named this company is clearly a psycho.

They never have. Their money insulates them from the consequences of their actions.

There is no justice. And Iran will probably blow up some innocent civillians in response the opportunity arrives, instead of targeting the actual culprits.

Responsibility can fall on multiple parties, including both the tool user and the tool builder.

This is not AI killing poeole, this is US government killing poeole.

AI didn’t kill innocent people here, the Department of Defense and its decision-makers did.

I think your source is undermining your claim. That Wikipedia article strongly suggests that the whole event was fabricated by Russia.

Says the Russian Emergency Situations Ministry, which then immediately refused to allow the UN access to verify the claim.

No. That's too easy on the higher ups, they never pull triggers.

Completely disagree. When we’re talking about something like air, missile, or artillery strikes, the operator of the weapon system may literally never even see the target, on a screen or otherwise. They may simply be entering targeting information provided to them by an operations center. These decisions are social and group decisions, with distributed responsibility, which is why it’s so important to have institutions like the Civilian Protection Center of Excellence which historically has identified and helped implement the policies which prevent these kinds of disasters, but which unfortunately the current administration decided to shut down without a replacement.

Sometimes, the difference between a successful strike on a lawful combatant and disaster is one analyst pointing out that the building the enemy is hiding in is next to a square that’s used for a market on Tuesdays and wouldn’t you know it, it’s Tuesday. That’s not something the person “pulling the trigger,” can always be expected to know, which is why every ops center needs to be designed to elevate those voices, when the time comes

The one that pulls the trigger is often not the one doing any targeting, especially on a strategic strike weapon like a tomahawk. They are receiving a target data package with preprogrammed instructions. Their role is far more... mechanical. They're not seeing the targets at all.

I would argue `s/pulls the trigger/gives the go ahead/` is intuitively more appropriate. The sailor on the boat that fired off the tomahawk (iirc) was likely not in a position to make an in-the-moment determination

> unreleased Pentagon probe

From TFA.

He's rich and connected, of course not.

It was a school. No one on either side is denying this.

> Stories like this always have all sides muddying waters.

Which makes the administration's near complete silence on the strike rather suspicious, no?

https://www.propublica.org/article/hegseth-civilian-harm-mit...

Yes it is really the case here.

> The terrorists

What fucking terrorists? We're talking about a country bombing another country's school. A country with a population of 93 million. Are you trying to insinuate that all 93 million Iranians are terrorists?

Maybe you meant the American terrorists?

> Stories like this always have all sides muddying waters.

All sides agree that the US bombed a school full of children, actually.

"stupid and energetic, the worst kind of general" - Indy Neidell

"maximum lethality, not tepid legality."

> they knew it was a school full of kids.

I would like to know how you know this