Hacker News

Ok but we're in a thread about Ubers cloud bills, which are probably well into the 9 figures annually. It definitely gets talked about in board meetings.

Global public cloud spend is hundreds of billions of dollars a year. I wouldn't be surprised if it's AWS's marketing team that came up with the talking point about how much more expensive developer time is.

Edit: put this another way- wherever you work, you might know what parts of the architecture need some performance work but do you know what parts of the architecture cost the most money?

A couple of years ago, I optimize some shit and reduced the annual billing of 150k€/y, for a 3 days of work

I might say, "hardware" is expensive compared to (my) salary :)

It might take an engineer with no prior RDBMS knowledge a quarter to be able to optimize a DB for their use case, but then it’s effectively free. You found the optimal parameters to use for writer nodes? Great, roll that out to the fleet.

It's neither. I remember them moving to the cloud but apparently they moved to Google/Oracle (the latter making this article particularly interesting btw). As per the relevant press release : "It’s understood that Uber will close down its own on-premises data centers and move the entirety of its information technology workloads to Oracle and Google Cloud."

It doesn't get much better if you translate that paragraph from corpo speak to normal language: "We did the upgrade step by step. We did each step in batches. After we already decided how we were going to upgrade the clusters but before actually doing it we asked the teams responsible for keeping the clusters running for their opinion. This helped create an environment where we work together and helped monitoring the process"

I'm sure there are people who write like that. LLMs have to get it from somewhere. But that part especially is mostly empty phrases, and the meaning that is there isn't all that flattering

Relevant pg thread on twitter: https://x.com/paulg/status/1777030573220933716

If the meatbag was a salesperson though… very believable! ;)

> Human writing is short and to the point (either because they're lazy or want to save the reader's time)

Good human writing is short and to the point. (Technical writing at least.) But this is not a result of laziness — it’s actually more difficult.

“If I had more time, I would have written a shorter letter.” - Blaise Pascal, and probably others [0]

In any case I find these LLM “gotcha” comments incredibly tedious.

[0] https://quoteinvestigator.com/2012/04/28/shorter-letter/?amp...

>It's simple. Human writing is short and to the point (either because they're lazy or want to save the reader's time), yet still manages to capture your attention. AI writing tends to be too elaborate and lacks a sense of "self".

Corporate (and SEO) writing has always been overly verbose and tried to sound fancy. In fact, this probably is where LLMs learned that style. There's no reliable heuristic to tell human- and AI-writing apart.

There's a lot of worry about people being fooled by AI fakes, but I'm also worried about false positives, people seeing "AI" everywhere. In fact, this is already happening in the art communities, with accusations flying left and right.

People are too confident in their heuristics. "You are using whole sentences? Bot!" I fear this will make people simplify their writing style to avoid the accussations, which won't really accomplish anything, because AIs already can be prompted to avoid the default word-salad style.

I miss the time before LLMs...

> Not to mention any sane tech writers that value their time wouldn't write this much.

This is a big part of why the tech is so damn corrosive, even in well-meaning use, let alone its lopsided benefits for bad actors.

Even on the “small” and more-private side of life, it’s tempting to use it to e.g. spit out a polished narrative version of your bullet-point summary of your players’ last RPG session, but then do you go cut it back down to something reasonable? No, by that point it’s about as much work as just writing it yourself in the first place. So the somewhat-too-long version stands.

The result is that the temptation to generate writing that wasn’t even worth someone’s time to write—which used to act as a fairly effective filter, even if it could be overcome by money—is enormous. So less and less writing is worth the reader’s time.

As with free long distance calls, sometimes removing friction is mostly bad.

My hypothesis is that long form content generated by LLMs tend to sound like blogspam and press releases because those are exactly the kinds of things they were trained on. Most content generated by humans for public consumption is ANYTHING but succinct.

Their style is much more direct if you just ask them a question or to summarize something. (Although whether the answer is accurate or not is another matter.)

This. Thank you for verbalizing what I struggled to.

MySQL stores table data in a b+ tree where updates modify the data directly in place as transactions are committed, and overwritten data is moved to a secondary undo log to support consistent reads. MySQL indexes store primary keys and queries rely on tree traversal to find the row in the b+ tree, but it can also contain references to rows in the undo log.

PostgreSQL tables are known as heaps, which consist of slotted pages where new data is written to the first page with sufficient free space. Since it's not a b-tree and you can't resolve a row with just a primary key without a table scan, Postgres uses the physical location of the row called a tuple ID (TID, or item pointer) that contains the page and position (slot) of the row within that page. So the TID (10, 3) tells Postgres the row is in block 10 slot 3 which can be fetched directly from the page buffer or disk without having to do a tree traversal.

When PostgreSQL updates a row, it doesn’t modify the original data directly. Instead, it:

  1) Writes a new version of the row to a new page
  2) Marks the old row as outdated by updating its tuple header and relevant page metadata
  3) Updates the visibility map to indicate that the page contains outdated rows
  4) Adjusts indexes to point to the new TID of the updated row

Old rows continue to accumulate in the heap until the VACUUM process permanently deletes them, but this process can impact normal operations and cause issues.

Overall this means Postgres does more disk I/O for the same work as MySQL. The upside is Postgres doesn't have to worry about page splits, so things like bulk inserts can be much more efficient.

Worth reading up on Postgres' MVCC model for concurrency.[0]

Short version is that VACUUM is needed to clean up dead tuples and reclaim disk space. For most cases with smaller amounts of data, auto-vacuum works totally fine. But I've had issues with tables with 100m+ rows that are frequently updated where auto-vacuum falls behind and stops working completely. These necessitated a full data dump + restore (because we didn't want to double our storage capacity to do a full vacuum). We fixed this by sharding the table and tweaking auto-vacuum to run more frequently, but this isn't stuff you have to worry about in MySQL.

Honestly if you're a small shop without database/postgres experts and MySQL performance is adequate for you, I wouldn't switch. Newer versions of MySQL have fixed the egregious issues, like silent data truncation on INSERT by default, and it's easier to maintain, in my experience.

[0] https://www.postgresql.org/docs/current/mvcc-intro.html

When postgres deletes a row, it marks a disk space unused. If you delete a lot of rows, there will be plenty of unused space, but that space will not be released to OS. To release that space to OS, you need to run vacuum full or perform backup/truncate/restore.

In normal circumstances it's not needed. The unused space is reused for further inserts and if your database maintains steady rate of inserts/deletes, this won't be an issue.

However I experienced situation when database starts to grow uncontrollably and it was happening for weeks (when actual workload didn't change). I don't know what causes that behaviour. The solution was to run `vacuum full`.

For an in-depth read on the differences in MVCC implementations, this post is pure gold: https://www.cs.cmu.edu/~pavlo/blog/2023/04/the-part-of-postg...

VACUUM and VACUUM FULL (and/or with ANALYZE) can lock tables for a very long time, especially when the table is large. Incantation may also require 2x the space for the table being operated on. In short: it's slow.

Maybe just vacuum much more aggressively? Also there have been a lot of changes to the vacuuming and auto vacuuming process these last few years, you can pretty much forget about it.

Postgres 17 tremendously improves vacuum performance

You’re not renaming tables when you’re at scale.

Depends on your use case. I had a huge table get corrupted after a drop-swap rename during some DDL. 8.0.14 was the version, IIRC.

For simple use cases or with tried and true patch sets I'm sure it can be a work horse.

You can re-enable the old group by behavior, which we did.

More specifically, mysql_native_password is disabled by default in 8.4, but can be re-enabled if needed: https://www.skeema.io/blog/2024/05/14/mysql84-surprises/#aut...

Deprecation warnings are in 8.0. It's disabled in 8.4.

If you are up-to-date with all your libraries it all should go well, but if some project is stuck on some old code, mostly old mysql libraries, one might get surprises when doing the switch away.

If there are large amounts of Indian English in an LLM's training data, it stands to reason the LLM output will be very similar to Indian English, no?

I recently saw a tweet where someone pointed out that "today morning" was an Indian phrase.

I had to really think hard why it is incorrect / not common elsewhere. Had to see comments to learn -- someone explained that a native English speaker would instead say "this morning" and not "today morning".

As a Indian ESL speaker -- "today morning" sounded (and still sounds) perfectly fine to me -- since my brain grew up with indian languages where this literal phrase (equivalent of "TODAY morning") is not only very common, but also the normal/correct way to convey the idea, and if we instead try to say "THIS morning" it would feel pretty contrived.

Not exactly a stylistic difference but there are real differences in the dialects. Here's example from many moons ago: "Even I think that's a bad idea." That was an Indian colleague. It took me weeks to figure out that he was using "even" in place of "also."

In a like vein when Australians say "goodeye" they usually aren't talking about your vision.

> What are some stylistic patterns that are different when something is written by a US author v/s Indian?

Largely as @brk above you already mentioned, tendency to use formal and obscure words alongside a specific tone. I'll also re-iterate what @brk said, hard to fully describe, more of a "you know it when you see it".

If I had to pick some specific examples from the blog post, the following phrase is a good example:

We systematically advanced through each tier, commencing from tier 5 and descending to tier 0.

There are 101 ways you could write that in US English, but I reckon 99% of the US population would be unlikely to pick the above unless they were writing an academic paper or something.

This one is also quite Indian English in many respects:

Our automated alerts and monitoring system actively oversees the process to ensure a seamless transition and promptly alerts of any issues that may arise.

Similarly, we have stylistic elements such as the over-breaking of paragraphs to the extent it becomes a series of statements. For example:

Upgrading to MySQL 8.0 brought not only new features, but also some unexpected tweaks in query execution plans for certain clusters. This resulted in increased latencies and resource consumption, potentially impacting user experience. This happened for the cluster which powers all the dashboards running at Uber. To address this issue, we collaborated with Percona, identified a patch fix, and successfully implemented it for the affected clusters. The resolution ensured the restoration of optimized query performance and resource efficiency in alignment with the upgraded MySQL version.

A relatively short paragraph, but five phrases. Your average US English writer would likely word it differently resulting in it being trimmed down to two or three phrases.

As I said in my original post though, none of it is bad English, its just a different style.

In big companies they never just "hire a guy or two"; the bus factor would be atrocious in this case, not to mention vacations and such. The minimum unit of hiring is one team, about five people five or take. So the difference they are looking at is either hiring a team of Postgres people or forcing everyone on one existing team to learn Postgres deeply. From this perspective, standardizing on infrastructure makes more sense now.

Tbf, either can be made to be fast, or slow. MySQL used to have a huge advantage for range queries due to its clustering index (assuming you have designed a schema to exploit that capability), but as time has gone on that gap has narrowed [0] (despite the title, it's not just about inserts). My own benchmarks for strictly read-only queries have also borne that out.

The biggest difference, IMO, is if you _aren't_ aware of the clustering index, and so design your schema in a way that is suboptimal. For example, given a table with orders or something similar, with a PK of `(user_id, created_at)`, a query like `SELECT... FROM... WHERE user_id = ? ORDER BY created_at DESC LIMIT 1` can be quite fast in MySQL. With a monotonic integer as the PK, and a secondary index on those two columns, it can still be quite fast, depending on insert order. If however you invert the PK to `(created_at, user_id)`, while MySQL 8+ will still be able to use the index, it's not nearly as efficient – in my tests, I saw query speed go from 0.1 msec --> 10 msec.

In contrast, while there is a small difference in Postgres with all of those differences, it's just not that big of a difference, since it stores tuples in a heap. Again, in my tests, query speed went from 0.1 msec --> 0.7 msec. This is a point query, of course; with range queries (`WHERE created_at < ...`) Postgres suffered more, jumping up to ~25 - 50 msec.

[0]: http://smalldatum.blogspot.com/2024/09/mysql-and-postgres-vs...

There'd be less of that if there weren't a feeling that MySQL is the high-grade free hit that Oracle uses to get you hooked on their low-grade crap.

> pretty sure safe travels is critical to maximum value to their shareholders

Well, it is... sort of.

Obviously you can't have Uber be a guaranteed way to be robbed by a highwayman, but when you've cleared out most taxis in a given city, you can start to dictate the terms by which customers accept your service.

And if that means including language in your ToS that shove your customers into a binding arbitration agreement [0] that effectively shield you from the risks of hiring incompetent or malicious drivers, well... that's what that means.

[0]https://www.npr.org/2024/10/02/nx-s1-5136615/uber-car-crash-...

Honest question, how do you (or amyone) propose to vet drivers? They require drivers license and car insurance registration, anything like a CDL would make being a driver prohibitively expensive. Their rating system already works as a good signal the few times Ive used uber.