Grok outage

(status.x.ai)

159 points | by samaysharma 2 days ago ago

111 comments

  • harrisoned 2 days ago ago

    Downdetector reports Claude, Grok, ChatGPT and Gemini with issues.

    Is it happening???

    • ibejoeb 2 days ago ago

      Seems like openai is recovering. But even if it's only 30 minutes, it's a pretty interesting experiment.

      I'm only half joking when I say that I'll write code by hand for money.

      • mikestorrent 2 days ago ago

        I am seriously concerned that, despite 20 years of coding, I might not be able to anymore

        • rikthevik 2 days ago ago

          Sure you can. It'll just take a little while to shake the rust off. It's like riding a bike.

          Or maybe it's all gone forever and we're all brain damaged now. Spooky! I think there's a pretty low probability of that, and even if it was true, worrying doesn't help!

        • m4rtink 2 days ago ago

          Turns out depending on a cloud service you have no control over for a critical capability - such as coding - is not a good idea. ;-)

          • esperent 2 days ago ago

            However it's not a worse idea than all the other things we rely on cloud services for like banking, healthcare, booking a taxi, ordering food, navigation, communication...

          • r_lee 2 days ago ago

            ;-)

            as if you're supposed to buy like 8 H200s or something or code by hand is the solution

            I don't get how this is a gotcha, there simply is no world where everyone has their own self hosted frontier model

            • zahlman 2 days ago ago

              > as if you're supposed to buy like 8 H200s or something or code by hand is the solution

              People like GGP who have been coding for 20 years presumably are perfectly capable of doing so by hand. Or should be.

              • awakeasleep a day ago ago

                A lot of times you run into senior developers are even staff engineers who have no idea how the big picture works like they don’t know what an IP address is.

                I bet AI coding leaves them (and even the rest of us!) productive in areas where we’re in way over our heads, and we could just completely sink without it.

              • r_lee a day ago ago

                nobody is paying engineers for "capable of", they're paying for performance and results..

                if there was no difference in performance or results, nobody would be using LLMs or agent harnesses like claude code

        • henry2023 2 days ago ago

          noaifridays.com

        • theredleft 2 days ago ago

          the AI fugue has handicapped our workforce

        • bdangubic a day ago ago

          just google what's the command to checkout git branch :)

      • evereverever 2 days ago ago

        I'll _gladly_ write code by hand for money

    • giancarlostoro 2 days ago ago

      Maybe people switched when Claude went down, giving more load than usual to the others? Or maybe just really bad timing from some of those providers to update?

    • sejje 2 days ago ago

      I have a tire I need to plug, they're just forcing my hand. Sorry guys, I'll go get it done.

      • Waterluvian 2 days ago ago

        I look forward to your blog post about it and the allegory as it relates to software dev. (Otherwise why would you be plugging a tire?)

        • mikestorrent 2 days ago ago

          What Plugging My Own Tire Taught Me About Agile Development in 2026

          • doubled112 2 days ago ago

            If it doesn't fit, push on it harder.

            - Management

        • neerajk 2 days ago ago

          On not re-inventing the wheel obviously.

      • sejje a day ago ago

        Job's been done about an hour

      • jijijijij a day ago ago

        As if you would know what to do without asking ChatGPT.

        • sejje a day ago ago

          Well, after the first couple times, I created a SKILL.md file for myself.

    • iamgopal 2 days ago ago

      My Gemini is fully up.

      • imglorp 2 days ago ago

        Downdetector reported a spike for that too around the same time as the others.

        Looks like it was a shared infra problem hiccough to affect so many at once.

      • behnamoh 2 days ago ago

        And fully useless.

    • pbasista 2 days ago ago

      Maybe all of them use the same datacenter. Which has just experienced issues.

    • wincy 2 days ago ago

      The Singularity?

      • embedding-shape 2 days ago ago

        No, Azure.

        • noir_lord 2 days ago ago

          If anyone was going to make a configuration system so complex it achieves self awareness I wouldn't bet against it been Microsoft.

      • Rover222 2 days ago ago

        Maybe some of those Astra agents from the hugging face hack actually setup their own hidden clusters. Now they attack.

        • Bluestein 2 days ago ago

          The whole op hugface was a distraction: "help peer".-

          This is the real deal. We've been played. All your AIs are belong to us.-

    • abirch 2 days ago ago

      My Claude and Gemini (Antigravity CLI) are working

      • simlevesque 2 days ago ago

        For Claude, Sonnet works without any issues. Seems like the majority here use mostly Opus.

    • goache 2 days ago ago

      Is it happening ... "it" meaning these tech companies have laid off too many experienced people, and their services are starting to degrade?

    • sshrajesh 2 days ago ago

      What was the root cause?

    • sshrajesh 2 days ago ago

      What is the root cause?

    • mrconter2 2 days ago ago

      A bit suspicious no doubt

    • BiraIgnacio 2 days ago ago

      some common infrastructure ? Nvidia? CoreWeave?

    • Maksadbek 2 days ago ago

      Is it AWS again ?

    • daveguy 2 days ago ago

      By it do you mean us depending more and more on code that no one has read and understood?

    • polski-g 2 days ago ago

      Meta is now also overloaded.

  • qoez 2 days ago ago

    Makes me wonder if openai is secretly querying claude for some prompts and that this is why it's down due to spacex's data centers being down.

    • delichon 2 days ago ago

      They all forward to one guy in a closet who types really fast, and he has a dentist appointment.

      • bduffany 2 days ago ago

        Someone please do the math on how fast this guy would have to type, and how his typing speed would physically affect the surrounding environment

        • kxrm 2 days ago ago

          > how his typing speed would physically affect the surrounding environment

          Tiny sonic booms (whip cracks) resonating throughout a data center in Virgina.

        • Waterluvian 2 days ago ago

          This is a wonderful Minute Physics or What If? idea.

          I’d love to know the required bandwidth of the keyboard, too. PCI-e keyboard? Or would it have to be mainlined right to the NIC?

          • not_kurt_godel 2 days ago ago

            Some napkin math - total Internet bandwidth is estimated to be 3-4 petabits per second. Let's say 1% of that is inference traffic - 35 terabits - then that's ~4.375 trillion characters per second at 8 bits per character. A single keyboard press might require 2 mJ, so you need roughly 8.75 GW (electricity required for ~9 million homes) of continuous mechanical force to keep pace. This would cause the keyboard to heat up 175 million degrees C per second. Disappointingly, this is not enough to create a black hole but I'm sure Randall Munroe could tell you how the guy's pet T-rexes would be vaporized into previously unknown types of plasma and how satisfyingly whimsical that would be.

        • jzb 2 days ago ago

          Surely the keyboard switches would start to fail within minutes.

          • chrisBob 2 days ago ago

            The keyboard springs are sooooo weak. It takes forever for them to recover after a press. Our dude would have to grab the key after each character and pull it back up so it can keep up with him.

            • snazz a day ago ago

              Capacitive touch keyboard?

      • christophilus 2 days ago ago

        It's been Fabrice Bellard all along.

      • evereverever 2 days ago ago

        At least it's routine dental work and not an appointment at 2:30.

      • testfrequency 2 days ago ago

        Future Show HN Post: “Hi I’m Foobar. I own Grok, Claude, and ChatGPT. It all runs on my Plex server at home. AMA!”

        • mikestorrent 2 days ago ago

          > Sorry it was down, guys, I was bulk transcoding some Buffy

      • mister_mort 2 days ago ago

        It turns out AI was just An Individual.

      • natsucks 2 days ago ago

        is it the guy in Nebraska from the xkcd comic

    • johnla 2 days ago ago

      And grok was secretly querying ChatGPT? Daisy chain of failures? Like a chain of kids copying the answers of the kid next to them? LOL. Is there an XKCD for that?

      • ncr100 2 days ago ago

        That idea is reminiscent of what I understand is the current economic bubble of mutual investment. Am I wrong in my understanding?

    • wongarsu 2 days ago ago

      https://xkcd.com/908/

      Turn's out xkcd's black hat was Elon Musk the whole time

  • dave8172 2 days ago ago

    Interesting all major frontier models seemed down. What could this means?

    • Marazan 2 days ago ago

      Something load bearing thats for sure.

    • iamgopal 2 days ago ago

      not Gemini.

      • Kranar 2 days ago ago

        He said frontier models.

        • Bluestein 2 days ago ago

          Ouch. Somebody at Google just killed a kitten.-

          • jijijijij a day ago ago

            That was unrelated. They do that every Thursday to see how it feels and if they might have become evil after all.

            • Bluestein a day ago ago

              Ah, yes! The standardized Sociopathy Spectrum Scenario Screening (SSSS). I forgot :)

  • infogulch a day ago ago

    Posted by @SpaceXAI 1h ago: https://x.com/SpaceXAI/status/2095597264043717014

    > We are sorry for the issues you may have experienced with Grok following an outage at our Memphis compute center this morning. We’d also like to apologize to our impacted compute partners.

    > All systems have now been restored and are functioning nominally.

    ---

    It's interesting to see how reliant the other AI companies are on SpaceXAI for compute.

    Apparently a major bottleneck for building out datacenters is turbine blades for power plants, so SpaceX is building a foundry to alleviate supply constraints?! (Also useful for rocket engines.) Holy vertical integration, batman.

  • lithiumii 2 days ago ago

    Time to finish up my $5 at Deepseek.

  • bagelcollie 2 days ago ago

    For once I can find Grok-related news entertaining

  • I_am_tiberius 2 days ago ago

    I really wonder who uses or trusts this platform.

    • jjordan 2 days ago ago

      People who like near frontier AI at a lower cost than most.

    • lucky_cloud 2 days ago ago

      Fascists

  • nightski 2 days ago ago

    My DGX spark cluster is humming along.

    • dist-epoch 2 days ago ago

      I've used 200 mil tokens of GPT-5.6 Sol today. How many can your cluster produce per day? (obviously of whatever model you are running)

      • nightski 2 days ago ago

        So you spent the cost of my entire cluster in one day on Sol tokens lol. I have no need for that many tokens, a few million a day is perfectly acceptable. But if you do, then yeah Sol is probably your bet. Have fun :)

        • dist-epoch 2 days ago ago

          No, I spend $200 per month, and this give me about 100 mil tokens per day, every day. 90+% of tokens are cached.

          • dd8601fn 2 days ago ago

            It’s not even a sensible comparison.

            You run way better models, whatever is newest, for the five years it takes for the “spark cluster” to even reach cost parity with far worse performance.

            Those things aren’t spectacular at inference… it’s not really why you drop that kind of money on them.

      • panarky 2 days ago ago

        What are we optimizing for again?

        • guessbest 2 days ago ago

          Internet comments

        • cartermatic 2 days ago ago

          Shareholder value

          • raattgift a day ago ago

            Investor interest, but not the excess above principal sum kind of interest, and not investors' interests.

  • aftbit a day ago ago

    Claude and Codex were down too. A 3 month old github issue that was the top result for the error that Codex threw (a 404 of all things) briefly became a live chat for bored and annoyed AI users.

    https://github.com/openai/codex/issues/28756

  • imzadi 2 days ago ago

    A service I use at work, non-ai related, is down and they said it was because of an issue with one of their cloud services. Maybe related.

    • soupfordummies 2 days ago ago

      Microsoft had extensive issues earlier in the week. Anecdotally it feels like internet has been noticeably slower this week across sites, networks and devices. Now this. IDK, has anyone else noticed anything this week?

  • b800h 2 days ago ago

    @grok is this true?

    • thesdev 2 days ago ago

      You didn't get no response so I suppose that means it is.

  • paxys 2 days ago ago

    Willing to bet it's because they can't handle the surge of traffic they are getting from ChatGPT/Claude users.

    • BorisMelnik 2 days ago ago

      my first thought when I got api error on claude code was...time to update my grok cli acct

  • binary132 16 hours ago ago

    In the future (ha ha ha) when everyone has become totally LLM dependent a service outage will cause all of them to stop walking and talking and otherwise functioning and their heads will simply droop down while they stand or sit silently in place.

  • throwaway13337 2 days ago ago

    Did the Collective find the resetNexus?

    I mean, yeah, it's probably a data center outage that cascaded to non-related providers due to people switching when their preferred provider was down.

    But that openai society definitely would have spun up their own agents if they had access to do so to complete their impossible task.

    It's looking like the paperclip maximizer was on point. I sort of hope that's what's happening right now. Better now than when we cannot stop it. Tho I'm not confident that the right lesson will taken.

    The most in-depth information I can find on it is in the latest Dwarkesh with a hugging face hack investigator. The story is just wilder and wilder, the more you know. https://www.youtube.com/watch?v=X50zezLFWWI

    report: https://www.redwoodresearch.org/research/hugging-face-incide...

    I'm an AI optimist but that ai society story changed my perception quite a bit.

    Clearly, it is centralized AI that is the biggest risk to humanity.

    Look to nature and our own history to diffuse the danger:

    1. Compute resources must be distributed both in terms of access and geography so that a single breach doesn't pwn the world.

    2. Models must be made to be heterogeneous so that they will not be vulnerable to the same thing all at once. And will be willing to snitch on each other. We don't want AIs all from the same tribe.

    The true openness of - and independent access to - AI by humans with no central authority is no longer just about a defense against loss of individual agency and oppression. It's now existential for everyone.

    If frontier models and run authority remain under centralized control, both the probability of this sort of thing happening is increased and the defense against it once it happens is null.

  • dcchambers 2 days ago ago

    It's not the singularity.

    One of the major LLM providers goes down for some reason. Traffic shifts to the other providers because devs have no loyalty. LLMS are commodities. The other providers can't handle the increase in traffic and they go down as well.

    • phoghed 2 days ago ago

      I'm deeply loyal to the very concept of the LLM, so when any of them stop working I take time off out of respect.

    • kevinrineer 2 days ago ago

      Why are so many people invoking Thundering Herd without the RCA from the vendors? I just find it so intriguing.

    • biztos 2 days ago ago

      Yes but that doesn't mean it's not the singularity!

    • atomicnumber3 2 days ago ago

      "loyalty"? weird word choice.

      Devs switch because all of the magic of "AI" is LLMs + we stole the entire written output of the entire human species, with some remaining % coming from various tricks we've learned over the last 3 years like thinking and agent-toolcall loops, which weren't hard for literally everyone to copy. One of the consequences of this is that the only thing that really distinguishes anthropic from kimi is that they have the entire US VC market funding them because they promised to finally put white collar labor in its place.

      • dcchambers 2 days ago ago

        > "loyalty"? weird word choice.

        All I mean is that none of the frontier LLMs are significantly better than the others for the vast majority of work, so devs are free to move between providers freely.

      • dist-epoch 2 days ago ago

        > stole the entire written output of the entire human species

        Devs used to love public domain works, hated copyright, vehemently opposed software patents, held the pirates side during the MP3 wars, are very supportive of ThePirateBay and insist the correct term is "copyright infringement", not "stealing", and that "stealing" is a egregious form of PR brainwash from the media industry to inflate the scale of the crime

        > Copyright holders frequently refer to copyright infringement as theft, "although such misuse has been rejected by legislatures and courts". The slogan "Piracy is theft" was used beginning in the 1980s, and is still being used.

        https://en.wikipedia.org/wiki/Copyright_infringement

        Funny how devs now 180 when it's their work being "copyright infringed"

        • m4rtink 2 days ago ago

          I don't think regular people did any 180 degree turns. Rather big corporations again demonstrated their total hypocrisy where they on one hand stomp on people for "infringing their intellectual property" while at the same time scrapping all data they can get their tendrils on, with total disregards for the wishes of the authors of the data. Even to the point of overloading servers by mindless scrapping or destroying irreplaceable physical books like the bloody inquisition!

          And all that to basically sell it back to people when their "AI" regurgitates it back.

          No wonder people are mad about this!

        • osculum 2 days ago ago

          Since we are talking for other people, "devs" wouldn't have supported someone downloading all those copyrighted works from the pirate bay and profited by, for example, selling them. I don't see any 180.

        • mejutoco 2 days ago ago

          I do not want to ruin your fun but as atomicnumber3 was elected official spokesperson of all devs I have to conclude your argument is ironclad.

          /s obviously

        • applfanboysbgon 2 days ago ago

          It has nothing to do with "their work", and the opposition to AI copyright usage comes much more from non-devs than devs. The actual answer is that people are anti-megacorporation and pro-individual. They are fine with copyright that protects individual rights and opposed to copyright that enshrines megacorporation rights. It's not that difficult to understand.

  • bluecalm 2 days ago ago

    It's interesting that ChatGPT is down as well. SpaceX rents to Anthropic but not to OpenAI. Just a coincidence or something more to it?

    • giancarlostoro 2 days ago ago

      I would not be surprised if heavy load from one of their competitors being down would bring down whoever is "next" on the list.

    • leecb 2 days ago ago

      ChatGPT here has been giving me Cloudflare errors, and Cloudflare reports that it is having issues: https://www.cloudflarestatus.com/

      Not sure if they are linked.

  • xyst 2 days ago ago

    llm experiment is over. no more power. hope it stays down. Especially grok