URL already posted: https://health.aws.amazon.com/health/status
I've got an estimated bill for $1.7 BILLION over this month. Normal usage is < $5.
Obvs have created an urgent AWS support ticket. Anyone else seeing something like this?
Update: Reddit link: https://www.reddit.com/r/aws/comments/1uyuaw7/help_my_bill_s...
yuchen205 hours ago
I got 3 consecutive emails warning that my budget crossed its $18 threshold. Opened it up: cost was 78 million. Thought it was a phishing attempt, logged into my actual account, and... still 78 million. EMOTIONAL DAMAGE.
root-parentan hour ago
Wanna bet the description of this job post will be updated by the end of the day?
"Software Development Engineer II, AWS Invoicing"
https://www.amazon.jobs/de/jobs/10428480/software-developmen...
"...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinforcing areas: intelligent launch readiness — where autonomous AI agents analyze, generate, and validate the information needed to go live in a new market; cloud-native service orchestration — where configuration-driven microservices replace per-launch bespoke engineering with centralized, reusable capabilities so that expanding into a new country becomes a zero-code configuration change rather than a development cycle; and continuous validation..."
ibejoeban hour ago
Wow:
In this role you will:
- Design and build agentic AI systems that analyze, generate, and validate...
- Build agentic architectures that compose specialized AI agents dynamically...
- Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage...
cliglot18 minutes ago
I just find it funny how people claim that LLMs will put money in the hands of domain experts. There’s not a single damn bullet about the fucking domain lol.
root-parentan hour ago
The irony is, the only purely deterministic thing, will be token consumption...
jdiff39 minutes ago
I severely doubt the world ever gets to such a point that the entire world melts into AI hallucination. And token consumption depends on so many other things, it's not all that deterministic either.
serf15 minutes ago
(token usage) is trending towards predictability for a lot of reasons. it's not deterministic but it's getting easier to reason about usage.
blitzar9 minutes ago
> 194,400.00 USD annually
Fuck it, im in.
sebmellenan hour ago
That job description feels so far beyond parody that I could scarcely believe it until opening the link! What a world.
root-parentan hour ago
It gets worst:
"Senior Software Development Manager, AWS Global Bill Generation" https://www.amazon.jobs/de/jobs/10471948/senior-software-dev...
"We're transforming from monthly batch processing and manual war rooms to continuous billing, autonomous agents, and self-healing infrastructure. We believe operational burden is a technical problem, not a staffing problem"
This looks clearly...a staffing problem...
wbl25 minutes ago
If you can make the software cover the toil you save the staff for the tough cases.
ghurtado32 minutes ago
> This looks clearly...a staffing problem..
I think that big tech recently decided that I got 99 problems but staffing ain't one
I guess Nothing is a staffing problem when you make a rule that firing people is always the solution.
[deleted]43 minutes agocollapsed
iam-TJ38 minutes ago
[dead]
jayanmn19 minutes ago
Enterprise account . We got - 3trillion and change
idiotsecant4 minutes ago
Quick do your IPO before the books update
chii14 minutes ago
-$3 trillion! That's the highest earning investment that has ever existed!
rclevengan hour ago
I did too, those awstrack.me URL's look super suspicious and I hadn't seen this alert trigger before so didn't know what to expect.
At first I was sure it was a phishing attempt. Then went to the console (not using those links) Saw there was an outage where the console was wrong (no mention of email alerts) Then I thought I was hacked - what a perfect cover up for someone to evade detection when the console was wrong. Looked at some logs, realized the incident text was just not exhaustive on the impact. Went back to my cup of coffee.
Note to self- should have looked here first.
01284a7e11 minutes ago
Yes, I am taking legal action, no doubt.
SegfaultSeagull2 hours ago
Time to get a second job buddy.
wglass23 minutes ago
It's crazy enough this will be fixed soon.
Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to convince AWS their system was wrong, which took another big chunk of time. Meanwhile the discrepancy continued to accumulate.
After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS. I've never seen a calculation error on their part since.
aerhardtan hour ago
One can almost smell the vibes.
This is peanuts compared to a major cybersecurity catastrophe that’s surely in the making.
To give credit to the technology and the people using it - and I’m not being facetious - it’s actually incredible that at the current levels of usage the unprecedented catastrophic event has not yet happened.
Nicook37 minutes ago
some things never change. Pre AI I was always shocked that such large and complex systems actually run as well as they do. Especially after getting to see how the sausage is made/works.
unethical_ban25 minutes ago
It was the mid 2010s when I sensed a lot of SaaS becoming popular. Just host your ticketing systems, your IT management planes, your security management consoles, your SOC, all off-premises.
I wonder if businesses are thinking of ever swinging back to locally hosted, with the increased hostility of the Internet re: AI, vulnerabilities, DoS, and so on.
The_Blade38 minutes ago
Always messing up some mundane detail!
wpasc17 minutes ago
THIS IS NOT A MUNDANE DETAIL MICHAEL
kolanos21 minutes ago
$1.7-billion isn't a mundane detail Michael!
wpasc16 minutes ago
you beat me before I refreshed the page. what would you say... you do here?
RIMR41 minutes ago
Oh, that's the really fun part. The unprecedented catastrophic event is already happening. Several of them, in fact.
By the time we notice, it'll be too late.
Imustaskforhelp37 minutes ago
its like slowly boiling the frog
IAmGraydon8 minutes ago
Clearing LLMs out of our business infrastructure is going to be a massive undertaking. Though I have a tech background, I work in commercial real estate. We are recently seeing new levels of idiocy from the employees, including real estate brokers with zero tech knowledge "coding" solutions to find sites for clients and blindly trusting the output (which I came to find out was complete bullshit), as well as some who have literally stopped communicating with any of their own language - meaning every interaction they have with anyone not in person is made by an LLM. It's a massive threat to our brand and has got to stop. I can't imagine what companies with thousands or tens of thousands of employees who have really been riding the LLM train are going to have to deal with. This thing is more of a virus that exploits human laziness than actual useful tech.
lukaslueg6 hours ago
Apparently what used to be `GB of storage consumed` is confused with `Bytes of storage consumed`, leading to a cool off by 2*30 error.
> You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!
VulgarExigencyan hour ago
The user is asking me to calculate how much money they should charge their customer. The values they've given me are 0.45, 1.67, and 2.50. This is 2.50 + 1.67 + 0.45 = 4.62, but it could be any other number. Perhaps we should be concatenating the numbers instead. Wait! The . could also mean multiplication. 0 . 45 . 1. 67 . 2 . 50 = 3015000. But wouldn't multiplying by 0 zero it out? That can't be right, we wouldn't be charging anything. So 3015000 must be correct.
You should charge your customer 3015000 thousand dollars.
idiotsecant2 minutes ago
Would be funny if it wasn't so close to true
ghurtado29 minutes ago
> You're right to question my calculation.
Literally impossible to tell whether this is parody or an actual response any longer.
I challenge anyone to write something so stupid that an LLM couldn't possibly respond with it. I don't believe such limit exists.
leugiman hour ago
Oh great so 2*30=60 he only owes 28.3$ million... hehe
I guess you wanted to say 2^30 which makes 1.5$
hansvman hour ago
My hunch is the HN formatter swallowed the double asterisk typical of python exponents.
While we're being pedantic, 2^30 is 28 in normal programming languages ;)
stefan_an hour ago
Vibecoded the billing system, raised revenue 9000%. Great for that promo package.
raverbashingan hour ago
AI slop. Or just a distracted dev
root-parentan hour ago
>> Or just a distracted dev
And a distracted tester? And a distracted pipeline of regression tests?
No, the truth is way worst...
silon42an hour ago
I'd love to see the spike in their projected earnings internal dashboard :)
27183an hour ago
Either way it shows their QA and testing procedures are incompetent. It's just not acceptable for a utility like AWS to move fast and break shit. Should make you question whether it's safe or advisable to use any of their services.
It probably shouldn't be legal for banks, hospitals, governments, or any other critical infrastructure to be hosted on AWS if they do things like this.
daft_pink4 minutes ago
Maybe it’s one of those absurd situations where canceling a service doesn’t actually stop the charges. Instead, they quietly begin billing you for some random add-on that was bundled with the original service. You never knew it existed, never knew it had to be canceled separately, and now you’re paying full price for a completely pointless ghost service because the only thing it was tied to has already been canceled.
It sounds ridiculous, but something very similar happened to me with Amazon WorkSpaces. During the WorkSpaces setup, an AWS Active Directory (Directory Service) instance was provisioned as part of the deployment. When I later canceled WorkSpaces, I had no idea the Directory Service had to be deleted separately. I kept getting billed for it, and it ultimately cost more per month than the WorkSpace itself had.
bobbiechenan hour ago
AWS saw Anthropic billing a guy for $16 million on zero usage and thought, why stop at the millions?
https://www.techtimes.com/articles/320266/20260712/anthropic...
ruddct5 hours ago
If you owe the bank $100, that's your problem. If you owe the bank $1.7 billion, that's the bank's problem.
fatnoahan hour ago
I saw this in action on a smaller scale. In a past job, my wife organized events for a decent sized company. After an event, she'd typically have a $300k+ balance on her corporate Amex. When she went on maternity leave, the person filling in for her job neglected to actually pay the bills, so when she returned there were quite a few emails and voicemails from Amex regarding the over $500k balance.
The messages started as polite and eventually started to get more desperate in tone. At no point were they threatening or adversarial.
Imustaskforhelp38 minutes ago
I think that this might reflect more on Amex to be honest.
Amex realises that threatening would hurt their business trust more than anything. During the great depression, Amex accepted checks from other banks which were falling and paying through their own wallet as a matter of integrity. Amex has always been built around this idea of trust and prestige.
They make most of money from what I have heard on the transaction fees which are more than others (3% compared to 1%). They might get desperate but I am sure that they are one of the last guys who would wanna threaten you if you are paying some large bills for them (as compared to normal credit card companies which might even hire people to extract your loans in some messy situations)
So perhaps be so rich that the credit card company understands it as well and treats ya differently :-D
xp8422 minutes ago
Interesting. And hard to square with my perception of banks as completely mercenary and ruthless. I had a decade-long personal boycott (I know, LOL) of Amex after they, because, with otherwise perfect credit, I forgot about a $30 department-store card bill and got a 30-day-late mark on my report, Amex got spooked and abruptly closed both my never-late accounts with them (which were at or close to 0 balances). This was around 2008 though, so perhaps this was a genius algorithm designed to try and detect the very first whiff of consumer defaults, so they assumed that $30 was the first domino to fall of my personal financial ruin that could lead to me charging my accounts to the max and then going bankrupt.
(I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
danlittan hour ago
This joke only works if you actually impose a cost on AWS of 1.7 billion. If they just serve you a bill for no reason, it's still your problem.
[deleted]16 minutes agocollapsed
xp8429 minutes ago
Next question we'll find out is what if you owe the bank $1.7 trillion?
sajithdilshanan hour ago
Not if you’re Elon Musk
michelban hour ago
Elon Musk is everyone's problem
[deleted]an hour agocollapsed
focusgroup0an hour ago
[dead]
browningstreetan hour ago
I realized recently that Whole Foods no longer automatically and reliably detects your Chase Amazon Prime credit card when paying. So they don’t give you the discounted pricing automatically. I wonder how many customers are checking out the way they always do and are paying full price when, for years and decades, this worked fine.
The customer service people I talked to in the grocery store said this changed sometime in the last year. My guess is that it’s an unintended side effect of removing the pay-by-palm feature.
This is obviously unrelated but I joked about what else Amazon wasn’t reliably calculating….
hedoraan hour ago
Class action lawsuit time!
Either that or 1000’s of small claims court cases.
Even with arbitration, the overhead of dealing with that would be crippling. Hopefully someone over there decides to do the right thing, and auto-refund.
xp8430 minutes ago
Relevant to this, I've recently noticed a trend of mass tort cases being opened up in the past couple years, and they seem to do very well. The way these seem to work is attorneys identify a company who has clearly ripped people off, and what I presume is a repeatable way to guarantee a win (thus translating to a guaranteed settlement offer). Then they advertise for eligible clients, sign those clients individually to contingency agreements, and run the playbook. A couple months ago after signing up for one of these, I received a check for about $350 (after the agreed-upon 40% attorney fee), from Ticketmaster, and I had another one related to AT&T. It took about 10 minutes more effort from me than a typical class-action settlement, because I had to e-sign those representation papers.
So really, there's a third option now, that's much easier than class action, even when class actions don't get certified.
ofjcihen29 minutes ago
There are a hundred small things like this that seem to be popping up in what used to be simple and reliable systems and as much as I know they aren’t ALL because of vibe coding I can’t help but wonder how much is.
browningstreet18 minutes ago
Weirder is what happened a day later. I got an email that said my Chase Amazon Prime credit card was being re-associated with my Amazon.com account.
I never reported this nor took it up with either Amazon or Chase directly. There was a refund of my Whole Foods purchase (they needed to void my purchase and re-ring everything to give me the discounts.. I asked them to refund my purchase and I’d do without my Whole Foods purchase entirely).
Looking back I think at least 3 recent visits were charged to me at full price because of all this. Hard not to think of enshittification and whether Amazon Prime is even worth it, alas.. I live in a fairly rural area at the moment and need delivery.
mawadev2 minutes ago
This is just the cloud area, what if Amazon starts vibe charging regular customers because of some bug? Accounts that are directly linked with regular people's payment methods?
szge8 minutes ago
I wonder what's going on; they still don't have a potential solution after 7 hours and they have multiple teams on it. Never seen anything quite like this
lelandfe8 minutes ago
This just hit global news: https://www.theguardian.com/technology/2026/jul/17/amazon-we...
> Amazon Web Services customers receive bills for up to $1.5tn after global glitch
rboyd4 hours ago
Ask for some leniency. Let your account rep know about your budget difficulties and ask if you can make good faith payments of a few billion per month until you get back on your feet.
tedgghan hour ago
I got a 20K bill once and it was actually drafted from my bank account. It took me a couple of months and involving the office of the AG of my state to get the issue resolved and get my money back. Since then I never touched any AWS product, moved my small stuff to Azure. It’s been years since AWS have these issues with billing, you can find the stories online, students billed 60K for a compromised account launching servers to mine crypto which AWS somehow was unable to flag and block, and let run for months.
drew870mitchellan hour ago
AWS is basically a utility. I think it's inevitable that their carelessness around billing will end up with them being regulated like one.
srdjanr39 minutes ago
I wouldn't expect their detection of hacked accounts to be 100% correct. Sure, it might be obvious when a human takes a look, but humans can't proactively look at every account's usage.
ButlerianJihad5 minutes ago
For a while I had a portion of my "homelab" on AWS. I was an educator in a classroom where the students were learning cloud stuff, and the instructor was encouraging the students to stand-up cloud environments for learning, so I figured that I would do the same.
I used AWS' free tier, of course, and I enjoyed the initial setup in EC2, and I did a LAMP-stack MediaWiki installation. It wasn't too difficult, but two things sent me away forever.
1. It was impossible, or at least highly labor-intensive, in this modern era to adequately secure an ordinary Linux system running Internet-facing services. I put fail2ban and I filtered a lot of ports, and still spammers attacked me on Layer 7.
2. It was impossible, actually impossible, to limit or cap my cloud expenses in any billing cycle. Sure, run free-tier all I want. Sure, come in within the limits almost every month. But if I configured one thing wrong, or one thing went runaway, I'd have a sizable bill that I couldn't dispute. And even worse, those "runaways" weren't necessarily things in my sphere of control, but could be triggered by basically anyone coming in and using my VPC resources, especially egress network traffic.
So I closed out my cloud account, and I developed a lot of sympathy for businesses and corps that now are forced to run "in the cloud" rather than on-prem or their own machine rooms, but now they have no way to control expenses.
dawnerdan hour ago
That’s why you always use a spend limited card with variable cost providers.
myself2487 minutes ago
Or just own your own hardware. Spend a few bucks at Microcenter, build a machine, and there's simply no mechanism by which they could decide later that you should actually pay 100x more, and then magically suck it out of your bank account.
None of this can happen unless you first cede control.
wewewedxfgdf5 hours ago
I once got a credit card statement that said estimated time to repay ....... more than 100,000 years. It was discouraging but I did pay it off. And sooner than estimated.
TedDoesntTalk2 hours ago
Were you still alive after paying it off?
ambicapter2 hours ago
No, but they have the internet in the afterlife, apparently.
_joelan hour ago
They do, but the latency is terrible
Bluesteinan hour ago
> 100,000 years
100K years. Now that's load-bearing ...
27183an hour ago
That's good for the credit card company, they can project stable revenue 100k years into the future.
bobson38111 minutes ago
A guy on the sysadmin subreddit managed to 8x the global GDP https://old.reddit.com/r/sysadmin/comments/1uz2fv2/aws_says_...
sscaryterry5 hours ago
Vibe coding billing systems is a top-notch idea :)
ainiriandan hour ago
Hey what do you think about vibe coding weapon systems? Do you want to be my cofounder?
sscaryterryan hour ago
Sure! What could possibly go wrong?
chairmansteve2 minutes ago
Drones are already vibe targeting in Ukraine/Russia.
roskoalexey3 hours ago
They sent 3 warnings to my email, ok, I understand bugs happen (probably vibe-coded). But they didn't even send any notification that it's a bug. Going to leave AWS after that.
xp8412 minutes ago
Somehow I highly doubt anyone will leave AWS over this unless their use of AWS is way more low-complexity than the average account.
People make similar pronouncements after every us-east-1 outage makes the news, but I feel like AWS would be going out of business by now if people followed through.
It reminds me of airlines, where after a particularly grueling irregular ops experience, a few dozen people file off the plane swearing "Never again, <airline name>!" but really, we all must know deep down that the airlines are all subject to the same external inciting factors, internal profit motivations, and human imperfection, and thus all pretty equally likely to cause us a bad day or ruined trip. The effort spent to avoid one isn't really worth it.
anzovec2 hours ago
same
pqvst6 hours ago
Probably the closest I've ever been to getting a heart attack. Normally <$1 per month, and now suddenly $284,006,266,443.74. Whatever the bug is on their end, this is unforgivable.
everforwardan hour ago
Yeah, this one is bad because it’s off by so much I’m shocked it wasn’t caught by tests, alerts about unusual changes in the billing system, or even accounting. Like surely the P&L reports look all kinds of wrong right now, they have to be showing like 6M% profit margins and revenue measured in quadrillions.
I’m also a little surprised this didn’t trip a circuit breaker. For something as non-real-time as billing, I’m surprised they don’t have an automated kill switch that pauses the billing system and fires a page if variance in bills spikes. Naively some kind of “if the standard deviation of customer bills for this year changes by more than 50%, pause the billing system”. At that number of customers, those numbers should be pretty stable beyond internal billing changes they could normalize for.
TrickyRick7 minutes ago
If I were to guess this bug is in the "display" part of the system which is probably distinct from the "actually take money from the customer" part of the system. One can imagine they have gates on the "actually take money" part, especially for a large bill like ours which was ~$300b or about 2.5x AWS' 2025 revenue... In one month. Surely if we had actually accumulated that bill they would be the ones with the problems when we can't pay it.
krawat34 hours ago
Same here. I got an email with a bill of $233 million and an estimated $433 million until the end of the month. I panicked and nuked my entire setup (which wasn't used that much, anyway, the alert threshold was $1) - I really wonder how many people did the same.
It's been 2 hours and I still haven't fully calmed down.
zengineer6 hours ago
Same - just had some malicious bots running through my platform last week and really thought they found a security hole after all. Even though the amount sounded ridicoulus, I got quite nervous and a very bad feeling when I logged-in AWS and saw that price.
saghman hour ago
The should pass a law saying they should have to pay you the amount over the correct bill as compensation; I bet they'll stop making mistakes like this pretty quickly after that
gomid3 hours ago
Same. Cold sweat for about 20 minutes. Even though I saw the service health notification, I still spent the last hour trying to find where my storage spiked. In any case, I'll be tearing down plenty of stale infra after this!
bradhean hour ago
Current month $13,648,114,178,401.01 188,253,226,212%
Forecasted month end $18,729,381,032,152.4
Apparently my company owes the combined GDP of France, Germany, and UK to AWs.
xp8418 minutes ago
"Have you considered using Reserved Instances? You could save up to 2 trillion dollars next month. Book a call with your AWS rep."
glenstein4 hours ago
Probably the safest bet is to pay your bill in full to stay in good standing and then get refunded the difference when they revise it down.
NordStreamYacht4 hours ago
With interest, of course.
philipallstar6 hours ago
Maybe they're using too many humans and not enough AI in their software development. That must be it.
the_real_cher4 hours ago
The code base is not gigantic enough they need AI to generate massively more lines of code.
rwmj2 hours ago
But they're going to try anyway.
marcosdumayan hour ago
My guess is the GP swallowed a comma.
paulddraper2 hours ago
Well AWS never had bugs before.
egeozcanan hour ago
They need the customers to pay more so they can fix the bugs. It's self-correcting.
mrtksn5 hours ago
Wow, those price increases due to the RAM and storage shortages AI caused are brutal.
jumperabg5 hours ago
Most likely they also forgot to include "make no mistakes" instructions to their in-house LLM that deploys to production.
HugoTeaan hour ago
Rookie mistake
galoisscobi8 minutes ago
I just deleted my aws account. I don't need these vibes in my life.
qriosan hour ago
As someone who usually works with data analysis, the distribution of the numbers strikes me as odd. Almost all of them have one number that appears four times, and one or two that appear three times. And overall, there are an unusually small number (0–9) of digits that appear at all.
Maybe it's not just vibe-coded, maybe the numbers themselves are being hallucinated by an LLM.
berkesan hour ago
> Almost all of them have one number that appears four times, and one or two that appear three times
To me that looked suspiciously like string-handling in a weakly typed language.
Like when you do `"100" + 1` in JavaScript, or `int("100" * 2)` in Python.
I've seen my share of such bugs in PHP, Python, Ruby, JavaScript. In production. Obviously not as simple as the examples, but subtle, like when a library update changed `someFancyLocalStorage.getOrDefault("lastOrder", 100)` by always casting the value to the type of the default (released as patch release). Or where typedEnvGet() should typecast "numbers", but keeps it a string when theres whitespace `AMOUNT_PER_CALL=100\n`. Or where a number passes through a deep stack of middleware and 99.9% of the times remains an int but in rare race conditions becomes a string. etc.
No evidence that's the case here. But from my experience, the repeating and strange formats of numbers hint strongly in that direction.
galonk5 minutes ago
Pedantic as hell but `"100" * 2` in Python (= `"100100"` for those who don't know) isn't really typing, it's operator overloading. Any language with that could implement the same questionable design decision.
everforwardan hour ago
Someone said the numbers are all off by 2^30 because they screwed up and are charging the per GB price for each byte.
It’s probably an artifact of them all being currency multiples of 2^30
ardacinar36 minutes ago
Well, for my case, I was paying $0 (Exactly, I managed to hunt down and delete every last resource in my account a few months ago). It was displaying $430 million for me. I don't think that is 0*2^30.
pcarmichael5 hours ago
https://health.aws.amazon.com/health/status
"Operational issue - AWS Billing Console (Global) Service - AWS Billing Console Severity Impacted - Inaccurate Estimated Billing Data"
Polizeiposaune18 minutes ago
Update as of 7:53am PDT:
"The rollback of a recent change did not resolve the issue and we are continuing to investigate multiple mitigation paths. Estimated bill updates remain paused."
masafej5365 minutes ago
>Estimated bill updates remain paused
Wait what if someones actually getting usage spiked
pfshort4 hours ago
117 billion us dollars. Eat that GDP of Kuwait! But yes I have never scrambled so hard to try to get on the phone with someone at AWS in my life. Terrifying 10 minutes until I found that banner on the support page. It should be front and center on the dash, not hidden away. And in yellow.
dgrin913 hours ago
Mine was 10 trillion today. At first I thought it was a lot, but then I realized its still smaller than the US national debt, so it cant be that bad.
tanseydavid13 minutes ago
For anything below a Trillion, you should just take it out petty-cash. </sarc>
My sympathies -- I know I would be overcome with panic in such a situation.
dv_dt5 hours ago
Cynically I wonder if this has an outcome as an unintentional (or intentional) anchoring exercise for future cost increases
ardacinar2 hours ago
I hope they're not planning for that large of a cost increase.
nblgbg27 minutes ago
My guess is that it's because of some vibe-coding stuff! We are using LLMs to write code, validate code and test the code ! What can go wrong ?
wewewedxfgdf5 hours ago
Cloud pricing has gotten ridiculous.
Host your own people. Host your own.
warumdarum5 hours ago
The old hypsters have to subsidize the new hypsters.
btown18 minutes ago
If AWS was a predatory mobile gacha game, we'd get 300 apology gems as credit to our accounts for this mixup, to help us in our rolls for the next 3-letter acronym they release.
Do the right thing for the players, Matt!
cryo32an hour ago
How do we know if our bills were ever right if this made it into production?
ahokaan hour ago
That's the neat part, you don't!
Hamukoan hour ago
Well, they publish unit prices for everything, so you could just get to counting. Whenever I've had to do cost estimates, you estimate how much AWS resources you need and then times that by the unit price.
dang2 hours ago
One user posted a screenshot: https://prnt.sc/UqjcYD3RSQrS
sebmellen32 minutes ago
Wow, $139 B.
glaslong11 minutes ago
Seems like a scam. Call your CC company and issue a chargeback :p
scrapcodean hour ago
Tale as old as time. When I was coming up it took a $20-40/m investment to get a "dedicated" server that you could start tinkering around on. When you couldn't afford that, you bricked the family PC trying to figure out how to configure your own LAMP stack.
Nowadays you just have to risk accidentally billing your parents CC the tune of multi-generation wealth to get that real-world experience.
danny_codesan hour ago
Hetzner has hard usage cutoffs
iamrik96 hours ago
I feel much better after seeing the $B estimates here; I only have an estimate of $34M so far
Folks can track it directly on AWS Health: https://health.aws.amazon.com/health/status
conspan hour ago
Maybe you went over 9,223,372,036,854,775,807 twice and came back to positive.
bfjvibybd6cuvu6an hour ago
It's ok, I owe them 1.22 trillion.
paulddraper2 hours ago
Peanuts
whatever116 minutes ago
Is it even possible to audit the cloud pricing? They just give us a number and we pay.
sokoloff6 minutes ago
On AWS, you can enable CUR (cost and usage reporting) and get detailed, line-item billing figures that you can audit.
And naturally, companies like Cloudability [now Apptio] and others have sprung up to do parts of this for you [at a fee, of course...]
https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur...
I'm sure other cloud vendors have similar functionality (because they need this on the back end to do their own billing anyway).
TekMol6 hours ago
It was over $500k in the email I got. Not a fun experience. My hands were trembling.
Makes you wonder - what if there really would be an incident where some massive amount of traffic got routed to your infrastructure by some heavyweight player? Say Wikipedia accidentally switches their IP to your CloudFront? Would you really be on the hook for $500k?
Hamuko5 hours ago
Well, even if AWS tried to charge my credit card on file for $500k, it would definitely not go through. Then they’d probably either forgive your bill or just ban you, since I imagine the threshold for taking people to court is fairly high.
marksk5 hours ago
logged in this morning to find a bill of $595 Billion... heart rate went through the roof... then I noticed the open issue, phew! nice one guys... you got me there...
But with AWS costs rising anyway (not by that much but OK), I'm probably not the only one to start reconsidering their cloud strategy. I think this might have just pushed me over the edge.
sshine5 hours ago
Even though it's just a bug, being charged $595B on a platform that is known to cost spike, reminds us that we're not in control of the platform, or our company's expenses.
dirkk05 hours ago
same here, I am still in shock. took me 10 minutes to find the 'operational issue' message in the dashboard. longest 10 minutes of my life.
charles_f4 hours ago
Can you not set spending limits in AWS?
inigyou4 hours ago
No you can't. Spending limits imply realtime billing backend flows and they also imply deleting all your data so that you don't pay for storage.
benterix2 hours ago
I heard this false justification already in 2007, in spite of many customers asking for it.
Incidentaly, smaller competitors solved this issue decades ago, while the big cloud decided it is more convenient never to implement it.
handoflixue3 hours ago
Realtime billing seems entirely within the abilities of AWS.
"Limits except for Storage" seems even easier - I don't think I've ever heard of a storage-based billing story, although I'm sure one or two exist
everforwardan hour ago
Storage-based billing is huge, unless you mean something other than “places that make you pay for storage separately”.
Also many places I’ve worked, storage is a huge part of the spend but that depends a lot on what you do. e-commerce doesn’t use a ton of it, but if you handle user-generated content or do any kind of training (LLM, computer vision, etc) then you can very much end up in a place where storage becomes a top line number for infra spend.
GitHub pre-Copilot was probably like that. They host a shitload of data, most of which is just at rest the majority of the time. Storage and networking are probably the majority of their infra costs.
Planktonne2 hours ago
They could do it; they don't want to.
minitoar2 hours ago
What is a storage-based billing story?
kgwgkan hour ago
Once upon a time in a cloud kingdom far, far away a big, beautiful bill was issued based on storage causing much disconcertion. Etc.
prmoustache41 minutes ago
Storage could switch to read only.
That would mean an outage but that is still better than going bankrupt and teach you a thing or two about monitoring.
perching_aix29 minutes ago
Not only can you not set limits, even the alarms are not real time. So it is entirely possible to get on the hook for terrifying amounts of money and not know until it's all too late.
boristsr4 hours ago
No, alerts but not limits.
reformd4 hours ago
he did, 140 billion :D
masafej5364 hours ago
If you owe AWS 140B dollars its their problem ;)
nottorp4 hours ago
Looks like they set up a LLM to estimate billing?
kayo_2021103023 minutes ago
What an `effin disaster. The alert almost gave me a heart attack.
luciana1uan hour ago
somewhere a junior dev at AWS just learned their billing dashboard has been off by a factor of a billion and is currently having the worst shower of their career
port30004 hours ago
They have to pay for that AI Capex buildout somehow
nixgeekan hour ago
Wow. As a side effect, this outage is handing Corey Quinn material for the next 4 years of AWS shitposting. No longer is NAT Gateway the prime target.
Sheepzez5 hours ago
Yes, I've got an estimated bill of $4bn. Probably related to the ongoing "Inaccurate Estimated Billing Data" incident?
traceroute66an hour ago
bdangubic9 minutes ago
I just invested ALL my money into AMZN cause next earnings report will be FIRE :)
hedora37 minutes ago
Does the affiliate program still work for AWS? When do I get my referral fee?
hedora41 minutes ago
And to think the federal government claims inflation is in the single digits this year!
sankalpmukim4 hours ago
AWS pushed the wishful thinking internal calculator to production.
rtkwe44 minutes ago
Aw man I was hoping to punk my manager but our cost estimates are unaffected.
paulbjensen4 hours ago
AWS revenue for 2025 was $128.7 billion, so I'd say probably a bug.
archerx4 hours ago
Double your yearly revenue with this simple trick…
yonatan80704 hours ago
Vendor-locked customers _hate_ him!
mjmasn2 hours ago
It's a good job it was off by such a large amount, or I might have panicked instead of writing it off as a phishing attempt. I had an email saying my $7.50 budget had been exceeded with an actual cost of $3bn.
rclevengan hour ago
My first thought was "Oh hell, who left the NAT Gateway on?"
im-broke5 hours ago
Help, what is this number - US$87,967,679,887,258.36
sshine5 hours ago
That's 87 trillion, 967 billion, 679 million, and so on.
[deleted]2 hours agocollapsed
mlitwiniuk6 hours ago
I was actually in the toilet when I got an email I owe them $36,869,876,146.51. I literally just shit myself.
mlitwiniuk2 hours ago
Ok, back to $0.17 :D
andystanton6 hours ago
Mine was about the same and evoked a similar response.
Hamuko6 hours ago
I got one for 8 billion while I was eating lunch. Thankfully I managed to not vomit.
[deleted]2 hours agocollapsed
andystanton6 hours ago
A couple of relevant links: - AWS Status Page: https://health.aws.amazon.com/health/status - Reddit Thread: https://www.reddit.com/r/aws/comments/1uyuaw7/help_my_bill_s...
abkolan3 hours ago
Will wait for the RCA, the update says that they will resort to last known estimate as of 15 July. I’m guessing that would imply that the bug is at a lower level, write or an ingestion path.
anon495847 minutes ago
Imagine the chaos if, as people sometimes suggest should happen, AWS shut down running instances in accounts that exceeded a billing threshold..
durronan hour ago
$44 trillion over here, at least our bill was so outrageously high that I just laughed
compounding_it4 hours ago
Are you sure it’s a bug ?
The crypto network you hosted should pay for itself in 10-20 years just like LLMs. Don’t worry. Consider Bank of America until then if you are good on credit score.
bryanrasmussen2 hours ago
hmm, if these estimates of Amazon profit for the next quarter are correct Bezos is set to become a trillionaire! Take that Musk!!
ohnoooooooooo24 minutes ago
do you see cost ever day for the month of July or just the last day? I also have billions of dollars in cost explorer
anibal-sanchezan hour ago
The new data centers are more expensive:
ACTUAL Amount: $1,046,294,123,330.95
anzovec4 hours ago
In my 30s, I almost had a heart attack too. I got a notification saying that my cost budget had been increased to one million dollars...
roskoalexey3 hours ago
Total forecasted cost for current month $477,000,039,440.24
Insane
foo-bar-baz5295 hours ago
Hope they’re using 64 bits to store these prices
sva_5 hours ago
float will have to do it.
csunbird6 hours ago
Just got a budget alert that I owe $286,486,223.88 on a hobby aws account, almost got a heart attack.
shobhitguptaan hour ago
Have even seen a $9.2 trillion for a friend.
jimbokunan hour ago
This is a strong argument to either self host or work really hard to be cloud agnostic.
steveBK1235 hours ago
Golden era of software productivity they say
zcemycl6 hours ago
Aws has created more unicorns than any accelerators.
throwaway_57535 hours ago
Should have used Fable.
lsdafjasd5 hours ago
I have $13,034.40, while not having used AWS for the last 8 months. Not as much but still crapped my pants
meraku6 hours ago
Same here. Usually $0.15 per month, current bill is $15.4 billion.
Hamuko6 hours ago
I went from 0.03€ to $8B.
sshine5 hours ago
Not only did your cost spike, it changed currency and went from postfix to prefix!
I understand people complaining about large bills, but this is over the top!
josefdlange5 hours ago
Well, no coffee needed this morning.
$103,515,940,301.79
ElevenLathean hour ago
Our alert was for exceeding $300...by several hundred billion dollars.
axus2 hours ago
This is just Anthropic reaching out to their customers for help with their AWS bill.
cifvts5 hours ago
Yes, an incident is ongoing https://health.aws.amazon.com/health/status
abkolan3 hours ago
The panic was real. We read about keys getting stolen all the time. Was about to nuke my set up too.
princetman6 hours ago
Mine is showing $241,946,798,744.75. I know it will be reverted, but for a brief minute there I suspected someone compromised my account and triggered rust rewrite of everything using thousands of agents via Bedrock :)
Phew.
akerl_6 hours ago
https://health.aws.amazon.com/health/status
Looks like this is a bug w/ S3
hypferan hour ago
To be exactly that guy:
This cannot happen if you do not do this renting at variable rates.
A thing you own doesn't suddenly bill you trillions of dollars in error. It doesn't hyperscale either, but neither do you.
ryanschaefer2 hours ago
The market *hates* this one weird trick to juice earnings
Avicebronan hour ago
Nothing like generational debt to kick off a Friday morning
rootsu2 hours ago
Our org account's bill is showing up as > 100 trillion.
sebmellen31 minutes ago
You've got to grab a screenshot of that.
aweiland4 hours ago
Glad I saw this. Mine said I racked up $400B yesterday. My usual spend is $15.
tamimio29 minutes ago
Results of vibe coding and vibe configurations.
marioptan hour ago
VibeBilling, love it
AegirLeet6 hours ago
Maybe this is a new strategy to scare people into finally locking down their old, unused AWS accounts. It sure worked for me!
reactordev4 hours ago
“Due to a rounding error” or a buffer overflow, you now owe INT_MAX to BaldGuyCloudService.
Yeah, this most certainly is bad code wrapping around a value. AWS will post a notice soon if they haven’t already.
ninjin-carh6 hours ago
I got 109 billion - am I the winner?
princetman6 hours ago
Sorry mate, $241,946,798,744.75 for Glacier here.
nprateemop6 hours ago
Depends. Did you also get a free heart attack?
kubelsmieci4 hours ago
This is real risk. Someone could really have a serious health problem.
djantje4 hours ago
I also like the percentual change, that is a lot of comma's.
[deleted]18 minutes agocollapsed
xyz77862 hours ago
$250 billion. Nearly died right then and there
roosgit5 hours ago
Amazon, the first quadrillion-dollar company.
fathermarz4 hours ago
Just got mine. $534,366,582,647.75
jagged-chisel2 hours ago
Shocking! That seventy five cents is suspicious.
atmosx4 hours ago
Looks like you are the biggest shareholder. Well, going by the popular saying: “You own AWS now”.
bknight19834 hours ago
I'm disappointed I only got a bill for $28M, need to work harder on burning money. Seriously though I thought my life flashed before me
danousna4 hours ago
Yeah, small timers, I only got $4,4T. How will I finance this?
rodeduivel4 hours ago
MMT!
marcosdumayan hour ago
Unfortunately, it's only Amazon that can issue bills backed by that debt, not the GP.
MichaelNolan4 hours ago
$28m actually seems worse. If I wake to a $100b bill, that’s obviously a mistake. If I wake up to a bill in the millions then my first thought would be “oh no what did I do wrong, this will ruin my life”
reaperduceran hour ago
Obvs have created an urgent AWS support ticket.
I think I would have just waited to see what happened when AWS tried to hit my credit card for $1,700,000,000.
When do you ever get that opportunity?
kvcm4 hours ago
I had Hermes managing mine, and it made a partial prepayment to help smooth out the bump in my account balance. Unfortunately Billing Support say my $17.4B refund may take up to 10 calendar days to be processed.
rickette4 hours ago
Some guy named Claude screwed up.
cmollis5 hours ago
yeah.. i just to a daily cost alert.. it was only 23 trillion dollars this month. i thought, hmm seems kind of high this month.
huntoa36 minutes ago
invoicemaxxing
gomid4 hours ago
Curious if it's just s3 costs or other services as well?
jatin_oo712 hours ago
for me it was s3 cost only
mrcwinnan hour ago
So long as customers are good for it, AWS is about to crush earnings!
thisisauserid4 hours ago
FinSlops.
rvz6 hours ago
I expect such incidents like this to continue. So please keep vibe coding.
realizer6 hours ago
$627,487,837,871.49
I might be a winner.
Executor3 hours ago
This generation is too entitled! He should some learn responsibility by paying the full amount; otherwise Amazon should delete his services/data. Consequences!
kinkuraj5 hours ago
Yes I received an 2.8m USD budget alert.
balintpeter6 hours ago
Yea, same here. $420M+ bill, when we have <10$ per month usually.
xbar2 hours ago
Rife.
tcp_handshaker2 hours ago
If its less than 2 billion is likely to be real :-) I would relax only if its in the trillions ...
nprateemop4 hours ago
I guess on the plus side I'm $1.7B better off so I can retire...
znpy3 hours ago
Is AWS in their "move fast and break things" era ?
jagged-chisel2 hours ago
Lumber along and smash stuff
jatin_oo712 hours ago
storage, compute cost is increasing AWS be like lets increase prices
hokkos5 hours ago
Same, i am now a slave to Jeff Bezos to the end of my life.
cyanydeez5 hours ago
AWS has become the uber employer: before AWS, you just had regular employers steeling employee wages bit by bit by forcing work, skipping breaks, etc.
All hail the new generations of our uberployers.
1-6an hour ago
Fast and loose with billing data. Welcome to the new Amazon.
hoppp4 hours ago
How much is that in kidneys?
atmosx4 hours ago
A lot.
jatin_oo712 hours ago
aws becoming first quadrillion dollars company
jameskilton3 hours ago
My personal photo backup S3 account, with a budget limit of $10, now going to cost me ....
$1,299,988,247,332.56!
That was a fun set of emails to wake up to, figured they had to be phishing for how outrageous of a number it was. But nope! Fun little incident they've got going over there.
ratelimitstevean hour ago
a billion here, a billion there, sooner or later it adds up
[deleted]2 hours agocollapsed
tgv4 hours ago
Mine was a mere $49B. Fucking idiots.
atmosx4 hours ago
Cheap!
mapt4 hours ago
AMZN Q2 numbers are in, and it turns out they're going to Goldman Sachs the AI bubble.
1234letshaveatw2 hours ago
brb, off to buy some AMZN
ares6235 hours ago
this counts towards ARR right? would be stupid not to
pelagicAustral4 hours ago
Imagine it not being a bug...
Sebb7674 hours ago
As the famous saying goes: If you owe the bank a million dollars, you have a problem. If you owe the bank a billion dollars, the bank has a problem.
speedgoose4 hours ago
Time to become a shepherd in some remote mountains.
RGamma4 hours ago
Surprise hyperinflation. Check the breadshelves!
lovich4 hours ago
You really should get your spending under control. Unfortunately unless you become one of the real people class through a large lottery, it sounds like you owe the rest of your life to AWS until you can pay off your debts for being so careless.
rucury6 hours ago
Uhh class action incoming? $34,909,930,575.09 over here.
akerl_5 hours ago
What would your damages be? They’re not actually going to charge your credit card for 34 billion.
rucury5 hours ago
I mean, emotional damages are a thing right?
akerl_5 hours ago
Not really in the way the media would have you believe.
Like “I was scared for a couple minutes on a Friday morning until I saw the vendor status page” is orders of magnitude away from the bar here.
Neikiusan hour ago
I wonder how many people died of heart attack when they saw this.
Hamuko5 hours ago
I hope they send out some free credits at least. I imagine quite a few people got a real fucking scare today. They haven't even sent out any corrections yet.
fian4 hours ago
This is probably going to push me to completely close a couple of AWS accounts I setup when doing training courses so I could get certified (mandatory requirement from my work).
I'm not currently running anything and have no plans to at the moment. I've always had a mild dread that I'll suddenly get a bill for more than $0.00.
If AWS can goof in a way that causes obviously massive bills (like today), what's to say they can't goof in more subtle ways and start charging small additional amounts that many people may not notice and just pay it.
cyanydeez5 hours ago
someones been dognfooding the AI too muxh
r0ckarong5 hours ago
Pff rookie numbers, mine was 375 billion.
tlovage6 hours ago
I got estimated costs of $56.something billions. Usually ~$100/month. My heart rate currently still sits at around 160 bpm. Motherfuckers.
aisloper41 minutes ago
I blame A.I. usage
kylecazar4 hours ago
You didn't have savings opportunities enabled
port30004 hours ago
Rookie error
bdangubic4 hours ago
eh your typical off-by-7 (zeros) programmer mistake
endless_smash5 hours ago
[dead]
throwaway4387116 minutes ago
Clearly they weren't tokenmaxxing hard enough or weren't using the latest models /s.
What an absolute joke. All just so that line goes up. As if their fees weren't high enough vs. alternatives (especially egress). And I'm sure the pro-AI crowd will keep saying we're luddites for not loving this clearly revolutionary and disruptive tech.
jimwilsonan hour ago
[dead]
lostnfound87784 hours ago
[dead]
GuestFAUniverse5 hours ago
Don't worry. With so much debt banks start to treat you with respect. /S
Honestly, I would worry more about estimated billing that seems plausible in general, but is way to high for you personally. These ridiculous amounts? Not so much.
Hamuko5 hours ago
I got freaked out by the mere fact that I got a billing alert, since getting one would require my monthly spend to have suddenly exploded.
blitzar4 hours ago
In unrelated news I just hit my target for S3 revenue (projections). Promotion meeting locked in for tomorrow (fastest in the companies history), looking forward to being a L2 Amazon employee.
rf1544 minutes ago
Of course, this is only considered an error if the account is unable to pay. /s