tapoxia minute ago
I don't really understand this architecture, but I thought Bluesky was distributed like Mastodon? How can it have an outage?
threecheese3 hours ago
> What I had missed is that we deployed a new internal service last week that sent less than three GetPostRecord requests per second, but it did sometimes send batches of 15-20 thousand URIs at a time. Typically, we'd probably be doing between 1-50 post lookups per request.
That’ll do it.
98codes2 hours ago
Ahh, the three relevant numbers in development: 0, 1, and infinity.
bombcar2 hours ago
Zero, one, many, many thousands.
htx80nerd12 minutes ago
less than ideal if I had to be frank.
goekjcloan hour ago
> The timing of these log spikes lined up with drops in user-facing traffic, which makes sense. Our data plane heavily uses memcached to keep load off our main Scylla database, and if we're exhausting ports, that's a huge problem.
I expect this is common.
electrondood19 minutes ago
Great write up... curious about the RCA. Thanks!
rvzan hour ago
Thank you for the post mortem on this outage.
templar_snow2 hours ago
[flagged]
lavelaan hour ago
Why?
jmclnxan hour ago
Lite Blue on a dark Blue background. That is a new one, I have seen grey text on lite grey, but blue on blue ?
The article does work in lynx, at least I can read it.
jonstaab40 minutes ago
nostr never goes down
pfraze37 minutes ago
All support to other decentralizers but nothing never goes down.
jonstaab13 minutes ago
1000x redundancy makes it vanishingly unlikely. Although I know we're due for a pole shift so all bets are off I suppose.