You must log in or register to comment.
And the world is a slightly better place, momentarily.
Huh, not for their users I’d say.
Fun fact, my mobile carrier also was/is toasted LMAO.
Nah, for the users too.
Especially for the users.
Nature is healing
I dont get it
The nerdy looking dude in the gif touched grass. “Wasted” is from a video game (GTA?) screen shown each time you died. So from this we can infer that our terminally-online protagonist, Hiroshi, did not survive the attempt to disconnect, even if only briefly.
deleted by creator
I called him Hiroshi because he vaguely reminds me of the guy from Heroes (Masi Oka.)
Doesn’t help that it rendered as a whopping 3 pixels for me
Ah now I get it, they mistakenly replied with it to “nature is healing” as a completely nonsequitur response instead of as a top level comment. Happens to the best of us.
Lemmy server operators can now say they have better 24h uptime than Meta! lol
Just updated my résumé.
I am feeling a lot of personal satisfaction that I had no idea this was happening and had to read about it on Lemmy.
Same
Yes. Would have been chef’s kiss to not even read about it here.
Hello to everyone that didn’t flock to X 👋 Welcome to your new home :)
Pouring one out for the SREs at Meta
Someone is having a really bad day today. I wonder if your phone dies when you get a certain number of pages or push notifications
Fun story. I had a flip phone years ago and you could have multiple recipients to a single text. And if the text was multiple pages, it would split into several texts. And you could resend already sent texts.
So one time I put in my girlfriend’s phone number in all 20 recipient slots. I then filled the text to the max size, though I don’t remember how many it split into. I then resent it over and over. This all took like 2 or 3 minutes.
Her phone was sending notifications over and over for the entire rest of the day. I’d guess at least 8 hours, probably more.
Fucking hell I used to love doing that! Man that brought back some memories. Would do it to my co-worker and just piss myself laughing.
The classic txt bomb. I use to do this if I had unused txts at the end of the month
No but it’s unusable. I had a weird bug on one of my phones that sent an SMS over as fast as it could as long as the phone was on. I wrote the initial SMS, the contents were something like “hey, wanna hang?”, and the poor guy on the other side was blasted for several hours of literally constant notifications.
Luckily my plan at the time had unlimited free SMS.
Looking at the downmeter shot someone posted above, it’s half the SREs in the country. Not sure what the root cause will be, but damn that’s a lot of money down the tubes. I would not want to be the person who cost Meta and Google their precious thirty 9’s of availability lol.
Which country?
Sokovia
Nah, what was that muddy country from Dilbert?
Elbonia!
The country where all of those services are maintained and hosted in… Just colloquial shorthand, not trying to be exclusionary.
Ok thanks for the clarification!
The country where all of those services are maintained and hosted in…
For Meta, Google, etc. that’s a number of countries all over the world.
That’s fair. Yall, I was really not trying to be shitty. It was just shorthand I used, thinking of their HQs. No ill intent intended and I apologize for any harm it caused.
Krakozhia
It’s likely there’s a root cause, like a fiber cut or some other major infrastructure issue. But, Down Detector doesn’t really put a scale on their graphics, so it could be that it’s a huge issue at Meta and a minor issue that’s just noticeable for everyone else. In that case, Meta could be the root cause.
If everyone is mailing themselves their passwords, shutting their phones on and off, restarting their browsers, etc. because Meta wasn’t working, it could have knock-on effects for everyone else. Could also be that because Meta is part of the major ad duopoly, the issue affected their ad system, which affected everyone interacting with a Meta ad, which is basically everyone.
I’ve been an SRE for a few large corps, so I’ve definitely played this game. I’m with you that it was likely just the FB identity or ad provider causing most of these issues. So glad I’m out of that role now and back to DevOps, where I’m no longer on call.
Yeah. And when the outage is due to something external, it’s not too stressful. As long as you don’t have absolutely insane bosses, they’ll understand that it’s out of your control. So, you wait around for the external system to be fixed, then check that your stuff came back up fine, and go about your day.
I personally liked being on call when the on-call compensation was reasonable. Like, on-call for 2 12-hour shifts over the weekend? 2 8-hour days off. If you were good at maintaining your systems you had quiet on-call shifts most of the time, and you’d quickly earn lots of days off.
Yeah I’d be less worried about internal pressures (which should be minimal at a halfway decently run org) and more about the externals. I don’t think you would actually end up dealing with anything, but I’d know those reliant huge corps are pissed.
Man, your on-call situation sounds rad! I was salaried and just traded off on-call shifts with my team members, no extra time off. Luckily though, our systems were pretty quiet so it hardly ever amounted to much.
I think you want people to want to be on call (or at least be willing to be on call). There’s no way I’d ever take a job where I was on-call and not compensated for being on-call. On-call is work. Even if nothing happens during your shift, you have to be ready to respond. You can’t get drunk or get high. You can’t go for a hike. You can’t take a flight. If you’re going to be so limited in what you’re allowed to do, you deserve to be compensated for your time.
But, since you’re being compensated, it’s also reasonable that you expect to have to respond to something. If your shifts are always completely quiet, either you or the devs aren’t adding enough new features, or you’re not supervising enough services. You should have an error budget, and be using that error budget. Plus, if you don’t respond to pages often enough, you get rusty, so when there is an event you’re not as ready to handle it.
Second half is the closest answer in this thread.
Hopefully they won’t need to cut their way into the data center this time.
What’s a SRE?
Site reliability engineer
Google terminology leaking its way into mainstream
Meta has SRE. Not just a Google thing, even if Google invented it.
Lots of places have SRE now, thanks to Google. Like I said, google thing leaked to mainstream
And nothing of value was lost
Good! Stay down!
I hope they secretly ran out of money because of FTC fines, and that they have no more credit to pay to run servers.
Tim Pool is blaming the Russians.
If they’re actually behind it:
I’d be interested in how it happened. Maybe someone can do it again
Lookup downdetector, almost all are flatlines.
Sap is also doing silly rn. Im gonna take a smoke break lol
Someone is having a very bad day
downdetector
Looks like it may have been AWS or something. All kinds of services were down a moment ago. Guess thats what happends when everything is on major cloud services.
Google have their own data centres (and cloud) so it may be something more in the connectivity area.
Maybe, I would expect redundancy. But ultimately I have no clue. I just remember the last time AWS went down. It seemed that a majority of the sites that I used daily were down all in one go.
Sometimes redundancy doesn’t help when it comes to network traffic routing. That system is based heavily on trust and an incorrect route being published can cause recursive loops and such that get propagated very quickly to everyone.
There was a case like this a few years back where a bad route got published by a small ISP, claiming they could handle traffic to a certain set of destinations, but then immediately trying to send that traffic back out again (because they couldn’t actually route to that destination), which bounced right back to them because of the bad route. It was propagated based on implicit trust and took down huge chunks of the Internet for a while
So could this be done maliciously? I’m just wondering about the Super Tuesday timing.
Yes, BGP Route Hijacking can be done maliciously although things like BGPSec can make it harder to pull off.
It affected the full 8 billion people in the world, not just the few hundred million on the US.
You’re talking about Border Gateway Protocol, BGP, route hijacking and it’s occasionally been a real headache over the years. Advertising routes used to be a more manual process so typos and incorrect entries, like what you’re talking about, we’re reasonably common. It was, and still can be, done maliciously too.
https://www.techtarget.com/searchsecurity/tip/How-does-BGP-hijacking-work-and-what-are-the-risks
Yup! BGP is an absolute mess and it is kind of a disgrace that it’s still the lynchpin of the internet
Yeah, they definitely host an unhealthy amount of the internet.
Infrastructure seems likely, but probably not AWS because it affected Google and Facebook so strongly. If it were AWS you’d see Amazon getting badly affected and AWS itself, followed by everyone who relies on AWS for infrastructure.
deleted by creator
Google is up for me
For me too, but noticeably slower than normal.
[email protected] took over and they’re bouncing the signal off the moon
Bandwidth issues, changed over to carrier pigeons (though now dealing with latency issues).
I’m not getting latency issues must be a problem with your setup
That’s my experience with all these services as well
Ah, wonder if problems contributed to Google Meet hiccups today. Very small, momentary.
Yeah probably furious hahaha
Something is causing this up
It was the houthis targeting red sea cables, check the news
I don’t think any major news sources confirm your theory.
BBC isn’t a major news source? Remember when they say countries do not confirm, it’s politically motivated. What governments choose to share is up to them and it does not confirm what their intelligence agency actually thinks.
Yeah there’s definitely some sort of major outage going on. Google Play Store is having some problems for me currently too.
Pathetic. My single podman container has perfect uptime from when I start it manually with podman desktop to when I shut down my PC. I also allow only the highest security standard, it being not accessible outside of my network and all that. I am clearly a cyber security expert.
Ahaha it’s funny because I initially read that as serious comment X)
The world was a better place for about 20 minutes
Going on a couple hours, it seems. LoL.
I’m glad that lemmy is there so that I know about this stuff.
On super Tuesday. Coincidence?
$100 it’s the Chinese in retaliation for TikTok, too logical to ignore.
You know it is big news when it shows up in the active feed of Lemmy 😅
Why it didn’t appear in the wholesome news community though.
Good. May they forever stay down. Who cares.
I care, it would be lovely. How can we pile on them like a ‘dog pile’, to keep 'em down?
