Still ongoing, I haven't been able to access my personal outlook email for 12+ hours and as a result can't access my accounts that use email 2FA, including my virtual medical appointment!
Along with the daily Github outages recently, it's making me do an audit of my dependence on Microsoft.
Ha nearly the same setup here, FreeBSD/ZFS/opensmtp/dovecot/rspamd
And yes uptime (with short outages(updates etc)) is not that important for a mail-server, since the sender re-transfers in intervals (mine is 5min the 8 then 15 then 60 then 180 then something up to 2days)
So the uptime for the service is 100% (maybe 99.999) over the last 8 years.
Getting a clean ip is hard. I have my personal email server setup and had to get it off of some black lists maintained by baracuda. Although i agree personal self hosted email is fairly straightforward and ideally more people should do it. Hetzner provides cheap and reliable vms.
Reliable is a flexible word. Hetzner does frequent, albeit often brief, maintenance operations that would incur a down time on a high-throughput live service. Not impossible to design around with enough redundancies, but far from "I have one FreeBSD box that's been up for 467 days with 100% uptime" that people used to brag about in the early 2000s. I have few VPSs and bare metal instances that I just `sudo reboot` when there is a kernel update with practically "zero" impact or downtime for myself. Something unimaginable to do in a production system where you need to cordon, traffic drain, update, reboot, uncordon then orchestrate that across an entire fleet of instances with different roles and characteristics in a specific order.
True, i have some hetzner (not vm's but metal), and Vultr (just vm's), i just had one bad IP until now (i just changed it instead of trying to get it removed from a blacklist)
> Still ongoing, I haven't been able to access my personal outlook email for 12+ hours and as a result can't access my accounts that use email 2FA, including my virtual medical appointment!
I guess software is still not solved, I'll let Dario know
Given that MSFT is down just 1.5% pre-market after what surely must be one of their biggest outages in the past decade, no one thinks much will come of this.
Crowdstrike crashed 8 million+ machines and broke many so bad they didn't fix on restart. Their stock is up 3x since then: https://en.wikipedia.org/wiki/2024_CrowdStrike-related_IT_ou... . I don't understand it but I also wouldn't bet on anything changing from this outage.
Businesses rationally won't switch because of one outage (even a bad one) because switching costs for a lot of places will be astronomical.
So again rationally this shouldn't affect the stock price much.
However repeated disruptive outages every few weeks/months (à la GitHub) is a different story, and there's also the invisible effect of businesses maybe being less likely to choose Microsoft in the future
If the downtime persists for more than 72 hours some will switch as they need to conduct their business , whether its email/github etc.
However as long as the outages are more like a snowday very few of the big orgs will have the inclination to get back into the business of self-hosting for accountability/price/biz inertia reasons.
A week long outage though , yeah at that point the decisions becomes a nobrainer and not even network effects will prevent the migration.
It must be 15–20 years now that Mimecast has been offering a protective layer around email services like Microsoft Exchange and Microsoft 365. Even today, you can see companies that are still able to keep their email running during Microsoft outages because they have Mimecast's email continuity service in place. It shows the value of having that extra layer of resilience.
Having an outage, fixing it, and then not having outages in the future, may be a positive signal compared to no outage ever, because you assume it will happen and the recovery time and process is unknown.
> switching costs for a lot of places will be astronomical
Switching to what though? It's not like we didn't have outages before SaaS. It just happened at different times throughout the year in stead of all having an outage at the same time.
I feel like the big difference right now is that management can't call the EoD for progress updates every 10 minutes, they just have to assume that someone at Microsoft is working on fixing the issue.
The reality for most companies, up until now at least, is that if they manage email servers themselves they will have more annualized downtime than if Microsoft does it. Of course if this outage lasts long enough (who knows what's going on over there), that will stop being true.
IMHO a bigger concern is that Russia hacked Microsoft in 2022 and China hacked Microsoft in 2023. Even that wasn't enough to scare companies off MS.
Basically, MS customers are locked in, the cost of migrating to anything else is huge, and there's a "no-one gets fired for buying MS" herd mentality, so MS has little to lose no matter how hard their product fails, just like Cloudstrike.
There's also the question of what alternative would really be better, or at least more reliable. My impression is that Google's infrastructure and engineering are genuinely better, but I'm recently ex-Google so take that with a grain of salt.
When we self hosted, we very rarely had an unplanned outage. It really shouldn’t happen all the time. Plus I could work planned outages around the company’s schedule.
It's ironic that the Internet was designed to accommodate nuclear war and we still decided to consolidate so much critical infrastructure onto monoliths.
It's usually worse. They spend it in many £500k consultancy blocks asking how to fix something, get a report back, don't implement it and then the staff who were working on it get a position in another part of the government.
I mean why is outlook even centralised? If a company has 600,000 employees, why don’t they get their own infrastructure? Why is it all dependent on central Microslop infrastructure in America
It's on days like this that Microsoft could really learn a thing or two from Google's SRE culture. These large tech companies have far too many internal dependencies, creating massive blast radii when something goes wrong.
I have administered both Exim and Exchange and would very much like not to do so again. Especially since in the time since then the number of requirements for effective delivery and antispam has only gone up.
It's been a while (~10y) since I administered windows servers, but I recall they were much harder to automate w.r.t. maintenance. So Microsoft created M365 to fill that niche and have organizations offload operational cost to them.
With a linux fleet can run via SSH/ansible and you can patch thousands of machines programmatically in minutes and control the deploy at a fine level. With Microsoft/Windows it often involves/d logging in to the one AD server where everything runs, back it up via 3rd party software, apply the patches and hope it doesn't break, and if it does, call MS support, because rolling back is really your last resort (I feel it's become better since 10y ago, but I don't have hands-on experience anymore).
Our company runs on-prem Exchange since 2019, and... well... it's a PITA to work with (Legacy Outlook/New Outlook), we downgraded from M365 and it was really a step down in terms of features and reliability. OTOH I guess it's keeping us afloat today when M365 is having a really bad day.
You can run your own email easily and cheaply, but if you wan to run with a team big enough and experienced enough to recover quickly if there's a problem it gets expensive fast.
Keeping your own server(s) up, including spam control, is work that takes time and costs money. Microsoft is probably still cheaper than that,[citation needed] and has more resources to keep everything running than even a medium-sized government. So it is understandable.
Yes. M365 gives "free", secure emails in a scalable, hands-off way. The downside is it practically being a black box, making email issues undebuggable.
It’s Microsoft we’re talking about so I feel like the latter is a given, and the former is a reasonable assumption.
Jokes aside, it does seem that somehow these hyperscaler cloud providers who tout the reliability of their multiple zones and global distribution sure do seem to always have some central dependency or single point of failure. AWS for example seems to have a global outage once every year or two at this point, or like us-east-1 problems always seem to affect other zones too. You’d think with their scale these clouds would be completely decentralized and redundant but apparently not.
Google basically is at this point, right? Not perfect, and obviously some things like DNS, Authn, and load balancing can’t be completely isolated. But overall, I think they put a ton of thought into removing software and hardware dependencies between regions, maintainence planning, and designing systems that don’t have outages correlated between regions.
The problem with decentralized redundancy is that it requires separate update schedules and process controls. This directly conflicts with the desire for convenience in rolling out new updates and code changes, etc. so the single point of failure tends to be deploying new code to everywhere at once.
Also, while they claim it is not a cybersecurity attack (without saying how they know), these are exactly the kinds of things I would expect with the soon/maybenowww future of automated hacking agents wrecking world's cyber infrastructure. Right now, its embarrassing, when it happens to the banks, it will be something more than that.
This take is bizarre and i’m really beginning to think that a scant few people are pushing it in bad faith in an attempt to spread rumours. It has no connection with reality. Extended downtime and faffing about with distributed systems issues all happen without it being caused by an attack.
A successful cybersecurity attack on a major email provider like Outlook or Gmail is an attack on the banks. Access to a user's email account gives attackers most or perhaps all they need to access their bank accounts.
I went to a bank branch here in India today, and apparently their entire email + a bunch of other services are down. This is one of India's biggest banks.
I remember there was a comment from a MSFT-er talking about how unreliable Azure is. Combined with AI coding, is the whole thing collapsing under its own complexity and weight?
We're seeing problems delivering to MS customers now. I'm not sure if it's they are full, busy, or perhaps rate-limiting to try and slow the deluge.
Last Error: 451 4.7.0 Temporary server error. Please try again later. PRX3 NextHop: substrate-int.office.com. Last IP: [MI3PEPF00008491.eurprd03.prod.outlook.com 2026-11-13T10:15:37.721Z
Last Error: 421 4.3.2 The maximum number of concurrent server connections has exceeded a limit, closing transmission channel [MI3PEPF00008487.eurprd02.prod.outlook.com 2026-11-13T10:42:55.252Z ]
We're also seeing messages about alternate failover failing.
It shouldn't be news in the first place, why do we all use such centralized systems where everything breaks down because of one bottleneck is the better question.
Why would any company as big as Microsoft choose to centralize software services so much that if something goes wrong, it effects everyone around the world?
If everyone uses Microsoft and it goes down for Microsoft, nobody blames you, and you're at no disadvantage compared to them.
Much of the productivity gain from this AI boom came from shuffling off the expensive emphasis on robustness and reliability, and this downtime is just one of the externalities of this increase in developer productivity.
As evidenced by Crowdstrike stock quadrupling since it took down half the internet in 2024, vanishingly few people actually care about downtime like this.
Meanwhile, there is decades-long familiarity with Microsoft's product offerings which makes for generational vendor lock in.
I think this is just a new normal. Genuinely speaking, what percentage of businesses will de-Microsoft in the next year over this? I don't think it would even be 2%.
With Europe now being fully woken up and still no emails working alongside Copilot this seems to be a rather more destructive issue then the last ones which mostly were solved at night and lasted ~8hours max in my memory.
We will also discuss and see how to failure proof our infrastructure against Microsoft outages. We made the mistake to base our VPN 2factor on Email. So now no VPN for anyone.
I think in the future, we need stable, minimal implementations of services: Minimal because a small attack surface is crucial in the age of AI and slop coding/attacks.
Minimal also because core services, like the Unix philosophy, do one thing and do it well. We don't need big tech breaking core services by continuously pushing new 'features' (slop, surveillance-ware, ad trackers, etc.).
The future demands small, stable core services, and mail service should be a small core service commodity that has been defined and solved (protocol wise) a long time ago already - we only need to make it an accessible commodity for all without the big tech entanglement.
Running a "simple stable core service" at this sort of scale is far, far harder and more complicated than you understand. Even responding to ICMP ping requests at AWS/GCP/Azure scale requires custom solutions. I've seen very many near-misses in high profile global services due to the complexity of running at this scale that have nothing to do with the causes you're assuming here.
The problem is that making a business out of "less" is hard. Especially when dealing with the hellscape that is "enterprise sales". Both the marketing people and the customer love piling on additional features!
Two decades ago I was at a startup that moved from a Linux email solution to a Microsoft one. Why? For the CEO's blackberry and calendar integration.
There are loads of FOSS email solutions and providers. There are fewer "email+calendar" providers, and there are even fewer "email+calendar+desktop logon SSO" providers.
We need decentralisation coupled with a reliance on shared protocols, like we used to have before platforms took over everything and anti-trust legislation became meaningless.
if this winds up publicized as update-related, i would question the operational decisions taking place within Microsoft.
first - were the updates fully tested in a legitimate test environment? (robust mirror with proper configuration and data) or were they "modeled" on a digital twin?
second - did any canary deployment precede the full rollout which seems to have a performance/capacity issue that borks (highly technical term) servers under load?
third - regardless of the answers for the first two, given their release quality history, should someone be suggesting "hey, this could be an impactful change, we should roll it during a quiet period like 2am Sunday" to mitigate customer impact and provide a longer restoration window option?
or has Microsoft simply asked OpenAI to manage it all...?
Still ongoing, I haven't been able to access my personal outlook email for 12+ hours and as a result can't access my accounts that use email 2FA, including my virtual medical appointment!
Along with the daily Github outages recently, it's making me do an audit of my dependence on Microsoft.
IRN, SEIM logs reporting a high incidence of telnet connections to port 25
Still ongoing, I don't think I remember such a long outage!
Bizzare. My self hosted email, which everyone told me was stupid to host myself has been sending and receiving just fine.
Of course I haven't received a single email because every important email is sent via Microsoft but I still feel superior because I'm not affected
Receiving is easy for personal mail I would say I receive 99% of the time.
Sending out is harder but that 1% when I send something I need it delivered.
Hosting e-mail for friends and family is harder and mostly what I would call stupid.
The solution is to send via some transactional email provider (aka amazon ses).
My iCloud email account is working just fine too. Security through obscurity ;)
Just don’t tell us how many nines you have, hey?
Uptime is not the hard part for self hosting email.
It is not difficult to keep a mail server online.
FreeBSD(ZFS+jails old but gold) + postfix + dovecot.
Ha nearly the same setup here, FreeBSD/ZFS/opensmtp/dovecot/rspamd
And yes uptime (with short outages(updates etc)) is not that important for a mail-server, since the sender re-transfers in intervals (mine is 5min the 8 then 15 then 60 then 180 then something up to 2days)
So the uptime for the service is 100% (maybe 99.999) over the last 8 years.
Mail servers are easy, getting the other mail servers (Microsoft,Gmail,etc) to play nice with yours is harder.
Not that hard, for example thats a good tester:
https://mail-tester.com/
and
https://mecsa.jrc.ec.europa.eu/ (offline atm Azure??)
Getting a clean ip is hard. I have my personal email server setup and had to get it off of some black lists maintained by baracuda. Although i agree personal self hosted email is fairly straightforward and ideally more people should do it. Hetzner provides cheap and reliable vms.
Reliable is a flexible word. Hetzner does frequent, albeit often brief, maintenance operations that would incur a down time on a high-throughput live service. Not impossible to design around with enough redundancies, but far from "I have one FreeBSD box that's been up for 467 days with 100% uptime" that people used to brag about in the early 2000s. I have few VPSs and bare metal instances that I just `sudo reboot` when there is a kernel update with practically "zero" impact or downtime for myself. Something unimaginable to do in a production system where you need to cordon, traffic drain, update, reboot, uncordon then orchestrate that across an entire fleet of instances with different roles and characteristics in a specific order.
>Hetzner provides cheap and reliable vms.
True, i have some hetzner (not vm's but metal), and Vultr (just vm's), i just had one bad IP until now (i just changed it instead of trying to get it removed from a blacklist)
infinitely many, so far...
I mean, I can't speak for him but I also run a small server and I think I just beat MS365 in nines because of this outage.
My non-mirrored Synology NAS full of movies just beat MS365.
> Still ongoing, I haven't been able to access my personal outlook email for 12+ hours and as a result can't access my accounts that use email 2FA, including my virtual medical appointment!
I guess software is still not solved, I'll let Dario know
Officially Microsoft 364
Lmao
its so funny, that service outages might help people get away from american hyperscalers.
Nothing will change. Nobody who is in a decisionmaking position will learn any lesson from this.
If you take a big corp down for a day then decision makers definitely notice.
I'd guess that almost every CEO affected asks themselves whether they should be so reliant on Microsoft.
Given that MSFT is down just 1.5% pre-market after what surely must be one of their biggest outages in the past decade, no one thinks much will come of this.
1.5% is basically random intra-day fluctuations.
they didn't every single other time something similar happened so why would this time be different?
And then switch to what? Exactly.
Stalwart hosted on OpenBSD?
You mean a Microsoft 365 account?
Because "personal outlook" at https://outlook.live.com/mail has been working this whole time.
Because I know people here are devs but you guys dont know IT lmao
This link is not loading up for me, and neither does outlook.office.com, it times out after a while.
It might work for you in your region. It is down in my region for some time.
Interesting because it was down here in Sweden.
365 that is
Peronal outlook has been fine
I've had two ex-colleagues contact me this morning in UK government positions and tell me that they're still out of action with no plan B in place.
I hope this sort of stuff pushes us to a distributed model rather than a centralised mono-provider one.
Crowdstrike crashed 8 million+ machines and broke many so bad they didn't fix on restart. Their stock is up 3x since then: https://en.wikipedia.org/wiki/2024_CrowdStrike-related_IT_ou... . I don't understand it but I also wouldn't bet on anything changing from this outage.
Businesses rationally won't switch because of one outage (even a bad one) because switching costs for a lot of places will be astronomical.
So again rationally this shouldn't affect the stock price much.
However repeated disruptive outages every few weeks/months (à la GitHub) is a different story, and there's also the invisible effect of businesses maybe being less likely to choose Microsoft in the future
If the downtime persists for more than 72 hours some will switch as they need to conduct their business , whether its email/github etc. However as long as the outages are more like a snowday very few of the big orgs will have the inclination to get back into the business of self-hosting for accountability/price/biz inertia reasons.
A week long outage though , yeah at that point the decisions becomes a nobrainer and not even network effects will prevent the migration.
It must be 15–20 years now that Mimecast has been offering a protective layer around email services like Microsoft Exchange and Microsoft 365. Even today, you can see companies that are still able to keep their email running during Microsoft outages because they have Mimecast's email continuity service in place. It shows the value of having that extra layer of resilience.
Having an outage, fixing it, and then not having outages in the future, may be a positive signal compared to no outage ever, because you assume it will happen and the recovery time and process is unknown.
> switching costs for a lot of places will be astronomical
Switching to what though? It's not like we didn't have outages before SaaS. It just happened at different times throughout the year in stead of all having an outage at the same time.
I feel like the big difference right now is that management can't call the EoD for progress updates every 10 minutes, they just have to assume that someone at Microsoft is working on fixing the issue.
Yeah true. Only by heel dragging and refusing to roll it into our production stuff lead to us surviving that.
The reality for most companies, up until now at least, is that if they manage email servers themselves they will have more annualized downtime than if Microsoft does it. Of course if this outage lasts long enough (who knows what's going on over there), that will stop being true.
IMHO a bigger concern is that Russia hacked Microsoft in 2022 and China hacked Microsoft in 2023. Even that wasn't enough to scare companies off MS.
Basically, MS customers are locked in, the cost of migrating to anything else is huge, and there's a "no-one gets fired for buying MS" herd mentality, so MS has little to lose no matter how hard their product fails, just like Cloudstrike.
There's also the question of what alternative would really be better, or at least more reliable. My impression is that Google's infrastructure and engineering are genuinely better, but I'm recently ex-Google so take that with a grain of salt.
That's really not true though. That's what the cloud companies told us.
We've had more downtime from all our SaaS and cloud services in the last year than the 20 years before when self hosting.
When we self hosted, we very rarely had an unplanned outage. It really shouldn’t happen all the time. Plus I could work planned outages around the company’s schedule.
Now it feels like something is always down.
Maybe use a decentralized protocol, something like SMTP...
All good until you run it all in the same business :)
Rather smug with my separate SMTP/IMAP provider now. And hourly offline backups of the mailboxes.
It's ironic that the Internet was designed to accommodate nuclear war and we still decided to consolidate so much critical infrastructure onto monoliths.
It should, but knowing UK gov they'll instead pay Accenture and IBM £10bn for a backup system that'll take 15 years and never get delivered.
It's usually worse. They spend it in many £500k consultancy blocks asking how to fix something, get a report back, don't implement it and then the staff who were working on it get a position in another part of the government.
Exacly, because 10B is big enough that everyone involved can get a bit of slice from it... Corruption is astronomical these days.
I mean why is outlook even centralised? If a company has 600,000 employees, why don’t they get their own infrastructure? Why is it all dependent on central Microslop infrastructure in America
It's on days like this that Microsoft could really learn a thing or two from Google's SRE culture. These large tech companies have far too many internal dependencies, creating massive blast radii when something goes wrong.
Because microsoft sort of pushes you into it.
That is exactly the question that needs asking and answering.
It's a pain to administer.
Not nearly as much pain as those who stand to benefit from outsourcing mail services would like you to believe.
I have administered both Exim and Exchange and would very much like not to do so again. Especially since in the time since then the number of requirements for effective delivery and antispam has only gone up.
As if this isn't solvable with automation.
Paying employees -> bad. Paying corporations -> good.
It’s mental that the entire thing is centralised. Why don’t big clients have their own servers? They’re paying a fortune every year.
The fact that the government can get taken down by a central server is mental.
Is it some sort of spying thing? Incompetence?
It's been a while (~10y) since I administered windows servers, but I recall they were much harder to automate w.r.t. maintenance. So Microsoft created M365 to fill that niche and have organizations offload operational cost to them.
With a linux fleet can run via SSH/ansible and you can patch thousands of machines programmatically in minutes and control the deploy at a fine level. With Microsoft/Windows it often involves/d logging in to the one AD server where everything runs, back it up via 3rd party software, apply the patches and hope it doesn't break, and if it does, call MS support, because rolling back is really your last resort (I feel it's become better since 10y ago, but I don't have hands-on experience anymore).
Our company runs on-prem Exchange since 2019, and... well... it's a PITA to work with (Legacy Outlook/New Outlook), we downgraded from M365 and it was really a step down in terms of features and reliability. OTOH I guess it's keeping us afloat today when M365 is having a really bad day.
They’re paying a fortune every year.
You can run your own email easily and cheaply, but if you wan to run with a team big enough and experienced enough to recover quickly if there's a problem it gets expensive fast.
Keeping your own server(s) up, including spam control, is work that takes time and costs money. Microsoft is probably still cheaper than that,[citation needed] and has more resources to keep everything running than even a medium-sized government. So it is understandable.
Yes. M365 gives "free", secure emails in a scalable, hands-off way. The downside is it practically being a black box, making email issues undebuggable.
> Is it some sort of spying thing? Incompetence
It’s Microsoft we’re talking about so I feel like the latter is a given, and the former is a reasonable assumption.
Jokes aside, it does seem that somehow these hyperscaler cloud providers who tout the reliability of their multiple zones and global distribution sure do seem to always have some central dependency or single point of failure. AWS for example seems to have a global outage once every year or two at this point, or like us-east-1 problems always seem to affect other zones too. You’d think with their scale these clouds would be completely decentralized and redundant but apparently not.
Google basically is at this point, right? Not perfect, and obviously some things like DNS, Authn, and load balancing can’t be completely isolated. But overall, I think they put a ton of thought into removing software and hardware dependencies between regions, maintainence planning, and designing systems that don’t have outages correlated between regions.
The problem with decentralized redundancy is that it requires separate update schedules and process controls. This directly conflicts with the desire for convenience in rolling out new updates and code changes, etc. so the single point of failure tends to be deploying new code to everywhere at once.
Also, while they claim it is not a cybersecurity attack (without saying how they know), these are exactly the kinds of things I would expect with the soon/maybenowww future of automated hacking agents wrecking world's cyber infrastructure. Right now, its embarrassing, when it happens to the banks, it will be something more than that.
This take is bizarre and i’m really beginning to think that a scant few people are pushing it in bad faith in an attempt to spread rumours. It has no connection with reality. Extended downtime and faffing about with distributed systems issues all happen without it being caused by an attack.
When was the last time MS had such a large outage?
The timing is suspicious but could indeed be a coincidence.
A successful cybersecurity attack on a major email provider like Outlook or Gmail is an attack on the banks. Access to a user's email account gives attackers most or perhaps all they need to access their bank accounts.
Yeah it’s either a hack or someone slopcoded something that has broken the servers so badly that they can’t be restored.
It's a huge distributed system, they're not hard to break in weird ways.
Malice or gross incompetence isn't really required to break it, that's its natural state. You need high levels of competence to get it to work
Would a secondary MX be useful here, at least for the receiving of mail?
How feasible would it be to quickly stand up an SMTP server and point your MX to it and get some mail while the outage is ongoing?
I went to a bank branch here in India today, and apparently their entire email + a bunch of other services are down. This is one of India's biggest banks.
Second-order effects of this are immense.
I remember there was a comment from a MSFT-er talking about how unreliable Azure is. Combined with AI coding, is the whole thing collapsing under its own complexity and weight?
Azure is, ironically, mostly unaffected, and Microsoft's issues long predate AI coding, by like half a century almost.
We're seeing problems delivering to MS customers now. I'm not sure if it's they are full, busy, or perhaps rate-limiting to try and slow the deluge.
Last Error: 451 4.7.0 Temporary server error. Please try again later. PRX3 NextHop: substrate-int.office.com. Last IP: [MI3PEPF00008491.eurprd03.prod.outlook.com 2026-11-13T10:15:37.721Z
Last Error: 421 4.3.2 The maximum number of concurrent server connections has exceeded a limit, closing transmission channel [MI3PEPF00008487.eurprd02.prod.outlook.com 2026-11-13T10:42:55.252Z ]
We're also seeing messages about alternate failover failing.
How is this not a bigger news? It's been on and (mostly) off for 14 hours.
It is literally at the top of the front page on all major news services.
Here in the UK it was on Radio 4 main news this morning and is the headline story for the online mainstream newspapers.
It's also got two stories on the BBC Home Page, including one which is live reporting.
Except in The Netherlands, where we have some political trouble.
Curious how such a worldwide issue can manifest, considering Microsoft usually has some pretty good geo availability mirroring?
Even Facebook took itself down with a single configuration change. Mistakes happen.
Well, over here we vote each year for the next four years.
It shouldn't be news in the first place, why do we all use such centralized systems where everything breaks down because of one bottleneck is the better question.
No one ever got fired for buying Oracle and Microsoft...
Getting buried by the Jensen Huang downfall. MS is super lucky it overlapped
Outlook is back for me. Just in time when my Claude session expired. I wish Claude would allow login with password.
I am still broken with that, can't use outlook and claude code.
Websites that don't support password (or passkey) login are infuriating. Sad to hear that Claude is one of them.
Claude has passkey support.
Claude is down.
Why would any company as big as Microsoft choose to centralize software services so much that if something goes wrong, it effects everyone around the world?
If everyone uses Microsoft and it goes down for Microsoft, nobody blames you, and you're at no disadvantage compared to them.
Much of the productivity gain from this AI boom came from shuffling off the expensive emphasis on robustness and reliability, and this downtime is just one of the externalities of this increase in developer productivity.
As evidenced by Crowdstrike stock quadrupling since it took down half the internet in 2024, vanishingly few people actually care about downtime like this.
Meanwhile, there is decades-long familiarity with Microsoft's product offerings which makes for generational vendor lock in.
I think this is just a new normal. Genuinely speaking, what percentage of businesses will de-Microsoft in the next year over this? I don't think it would even be 2%.
If it goes wrong its bad for users.
As long as it works its good for them.
In Canada and still cannot refresh my Outlook email. Not sure what’s going on but it seems to be really big. Is there any insider stories?
Duplicate of https://news.ycombinator.com/item?id=50471731
Unfortunally, this outage broke my entire work:
- No emails, calendar, file sharing, etc. - Claude Code broke as well because I cannot login back. - Multiple services I use with 365 are also broken.
With Europe now being fully woken up and still no emails working alongside Copilot this seems to be a rather more destructive issue then the last ones which mostly were solved at night and lasted ~8hours max in my memory.
We will also discuss and see how to failure proof our infrastructure against Microsoft outages. We made the mistake to base our VPN 2factor on Email. So now no VPN for anyone.
Service is back in Japan if anyone wants to VPN and check their emails
Tried this. Took a while to load, but it did work.
I think in the future, we need stable, minimal implementations of services: Minimal because a small attack surface is crucial in the age of AI and slop coding/attacks. Minimal also because core services, like the Unix philosophy, do one thing and do it well. We don't need big tech breaking core services by continuously pushing new 'features' (slop, surveillance-ware, ad trackers, etc.).
The future demands small, stable core services, and mail service should be a small core service commodity that has been defined and solved (protocol wise) a long time ago already - we only need to make it an accessible commodity for all without the big tech entanglement.
Running a "simple stable core service" at this sort of scale is far, far harder and more complicated than you understand. Even responding to ICMP ping requests at AWS/GCP/Azure scale requires custom solutions. I've seen very many near-misses in high profile global services due to the complexity of running at this scale that have nothing to do with the causes you're assuming here.
The problem is that making a business out of "less" is hard. Especially when dealing with the hellscape that is "enterprise sales". Both the marketing people and the customer love piling on additional features!
Two decades ago I was at a startup that moved from a Linux email solution to a Microsoft one. Why? For the CEO's blackberry and calendar integration.
There are loads of FOSS email solutions and providers. There are fewer "email+calendar" providers, and there are even fewer "email+calendar+desktop logon SSO" providers.
And even fewer that let you see that visio presentation someone tried sharing with you over teams
We need decentralisation coupled with a reliance on shared protocols, like we used to have before platforms took over everything and anti-trust legislation became meaningless.
half our monitoring went dark for like an hour before anyone even noticed it was upstream not us
Finally getting email that was queued up for the day. Then I searched for coverage of the news outage and was led to https://www.newsweek.com/microsofts-outage-cut-quarter-workf...:
> ...
> Newsweek contacted Microsoft for comment via email.
I hope that was biting sarcasm so that those of us who are getting up with restored emails can have something small to smile about.
"The company said it had “observed marginal success” in fixing the problem."
Truly next-level corporate speak.
if this winds up publicized as update-related, i would question the operational decisions taking place within Microsoft.
first - were the updates fully tested in a legitimate test environment? (robust mirror with proper configuration and data) or were they "modeled" on a digital twin?
second - did any canary deployment precede the full rollout which seems to have a performance/capacity issue that borks (highly technical term) servers under load?
third - regardless of the answers for the first two, given their release quality history, should someone be suggesting "hey, this could be an impactful change, we should roll it during a quiet period like 2am Sunday" to mitigate customer impact and provide a longer restoration window option?
or has Microsoft simply asked OpenAI to manage it all...?
Why every company in Europe uses a saas provider that’s run like a damn lemonade stand is beyond me.
Office 364, soon to be Office 363.
I guess some people who know how the system works decided to take that buyout package and Copilot is now in charge.
In the hands of god!
AI reducing productivity globally wasn't really the plan
Have they checked DNS?
Microsoft decided to give a 1 day off globally to all Enterprise customers.
Thank you!
They saw GitHub’s 93% uptime and decided to knock a 9 off theirs
Damn. I should stop advocating for running important stuff on-prem. We should move all the things to azure!
The Future is here they said, you will own nothing and be happy they said.
move to the cloud its always up, they said
What on earth was the cause? I couldn't get anything done all day today either.
cool, more skiing for you!
Guess I should build my own ski plates while I'm at it. Can't afford to buy them, though.
The Agentic Era.
I'm sure this has nothing to do with vibe coding, vibe devops or losing institutional knowledge thanks to AI /s