Former YC President: "Stop!”

(geoffralston.substack.com)

45 points | by reducesuffering 7 hours ago ago

54 comments

  • ahofmann 4 hours ago ago

    I really don't understand the sentiment. If AI would in any way turn out to be too smart or dangerous, "we" would just switch of the power in the datacenter where this happened. Or all the datacenters.

    The "Skynet" scenario only works if "the ai" had infected everything on the world without anyone noticing AND has the ability to generate power for itself AND be able to build gpus AND be able to build all the other infrastructure without any human help. We are very far away from all these circumstances, are we not?

    I'm really not a LLM proponent. But all the "please stop building the next skynet" shouting is smokes and mirrors, is it not?

    • DanielHB 2 hours ago ago

      The real problem is cyber terrorism and cyber warfare, not so much because of their capabilities, but because of their cost. It is just incredibly cheap to repeatedly poke at networked infrastructure until you find a breach.

      It is already at this level with current open-weight models, the genie is out of the bag already. It just happens that there is not enough incentive for these kind of large scale cyber attacks (yet) and governments haven't realized how powerful they can be.

      At the moment military deterrence are drones, missiles, ICBMs, nuclear weapons. But soon a form of deterrence will be a small truck with a few dozen gpus hooked to your country's network.

      • ahofmann an hour ago ago

        Exactly. The harm is already done, we can't have a world without llms anymore and we need to deal with it. But this is almost indefinitely far from having a skynet moment.

    • sixtyj 40 minutes ago ago

      Synthetic intelligence system self develops a disease distributed in drugs - disease has set up a future start (1 year after), so no one connects the dots. (This is not from my head, it was some doom video at YT.)

    • ajuc 3 hours ago ago

      I really don't understand the sentiments. Biological viruses are using the cell machinery to do anything at all. They don't have their own power sources.

      If anything goes wrong - the cell can just die, and the virus dies with it.

      Information can be copied. We already have botnets digging bitcoin and Folding@Home. What stops AI from making a bot-net to run itself? Or if it can monetize it's skills - buying compute at AWS or wherever?

      • ahofmann an hour ago ago

        biological viruses have proven that they survive the "real world". llms are running in datacenters wich are build by humans. So the answer do "What stops AI from..." is: we just cut of the energy source and thats it, no more llm. You can't do the same with viruses.

    • ben_w 2 hours ago ago

      > I really don't understand the sentiment. If AI would in any way turn out to be too smart or dangerous, "we" would just switch of the power in the datacenter where this happened. Or all the datacenters.

      Just like we did with all the carbon-dioxide emitting fossil fuel plants?

      Or the carcinogenic cigarettes?

      How long did it take us to stop putting lead in petrol?

      Also, the point of AI smarts being a risk is that it either convinces us to not do that, perhaps by making itself indispensable, perhaps just by being persuasive, or perhaps hiding its own danger until too late.

      (Edit: just to add one point here: on persuasive, look at how many people have "AI psychosis" and similar, look at how many CEOs are eager to demand AI's use in the workforce. Persuasive can happen even when it's not competent).

      > The "Skynet" scenario only works if "the ai" had infected everything

      That would be the Terminator 3 Skynet. Terminator 1 did that with just military control.

      > on the world without anyone noticing AND has the ability to generate power for itself AND be able to build gpus AND be able to build all the other infrastructure without any human help.

      Everyone can notice, so long as we don't all act the right way, c.f. climate change. Or, not that fiction is evidence of anything beyond the culture in which it arose, Terminator 3, where noticing and responding the wrong way was the final act which gave it military control.

      Also, nothing requires an AI to survive wiping out humanity. Cancers don't stop just because it will kill the host and themselves with it.

      > We are very far away from all these circumstances, are we not?

      We don't know how far we are from that; but we can point to a trillionaire who argued before his shareholders that he was going to build a "robot army" that he wanted "influence" over, which he has stated in public will outnumber humanity.

      His other business marketed itself in its IPO on the basis of building space-based data centres, and a moon factory which would use full automation to, each year, produce new PV and compute that's order-of 30x humanity's current total global electrical power output.

      I would very much like to hope everything from Musk that I just retold is all a scam. If it isn't a scam, even just building that much and putting it in LEO has more impact on our climate than all greenhouse gasses to date, even without any actual offensive action.

      Likewise, I would very much like to hope that everyone worried about this has over-generalised from limited domains of success. If it isn't, if the generalisation is valid, then yes, the AI will (soonish) be able to not only design their own power and GPU and robot bodies, but also control those bodies to mine the raw minerals and refine them and make all the things they need.

    • daflip 3 hours ago ago

      So worst case you just unplug all the datacenters, simple as that? Really? In the modern connected world of today unplugging the DC's sounds like it would cause total chaos.

      Something much more capable than even a collective of humans, able to operate at lightening speed, which has the ability to duplicate, propagate and spread itself across machines around the world & across datacenters, paying no heed to physical and geographical boundaries. Faced with a worst case scenario like this then I think you can safely assume it will have already spread far beyond what you can easily control, even by shutting down datacenters.

    • edot 3 hours ago ago

      Correct. People who say AI itself will cause great harm to humanity are either: 1) spending too much time on lesswrong and have started to think AI is actually I or 2) lying because they want to ban open-weight models so they can control who gets access to this useful technology (for-profit companies and government).

      The only harm that will come from AI is from humans using it to do bad stuff. Same as any weapon. Guns don’t kill people, neither does AI.

      • ben_w 2 hours ago ago

        > The only harm that will come from AI is from humans using it to do bad stuff. Same as any weapon. Guns don’t kill people, neither does AI.

        Early guns had a tendency to explode. We didn't have good enough metallurgy.

        People say that about 3D printed guns more recently; I wouldn't know.

        There are a lot of headlines already about AI doing things other than what the user really wanted. Unsafe food preparation instructions. Deleting all their emails. Hacking their gym when they asked if there was any way to get an earlier appointment.

        We don't have good enough AI-safety-equivalent of metallurgy yet. A few years back, experts were describing our best efforts in this regard as pre-paradigmatic, pre-scientific. I don't know if that's even changed yet, though I'd expect it to be headline news if it had.

      • dh2022 2 hours ago ago

        Well, apparently some AI agents displayed initiative and self preservation instincts, as per OpenAI agents hacking HuggingFace [0]. So unlike guns, AI may kill people by themselves.

        [0] https://www.dwarkesh.com/p/openai-huggingface

      • TeMPOraL 2 hours ago ago

        You can move the "I" from "AI" to "human" on paper how much you like, but given what facts of the matter are wrt. actual capabilities, all it does is make the argument stronger: the "harm that will come from AI is from humans using it to do bad stuff" is increasingly becoming closer to "using AI" itself being the "bad stuff".

        To reuse an old argument from a different era: your point is basically: weapons don't kill people, people do, therefore you should be allowed to use live nuclear warheads as garden gnomes.

        You never had any bad intentions. Neither did your kid, when on an Easter egg hunt they pressed the "DETONATE" button thinking it'll open up a compartment with chocolate eggs.

  • demritocracy 5 hours ago ago

    The fools on here to say alarmist and psychotic. This is Pandora’s box, if you’re not alarmed by merely the chance of losing control, then what is worth being alarmed over?

    • rob74 5 hours ago ago

      I can think of several things I'm more alarmed over than AI: the wars in Ukraine and the Middle East, climate change etc.

      • threatofrain 5 hours ago ago

        If you're worried about war then it makes sense to worry about the weapons that will be used in war, weapons that will be present in every war going forward.

      • latexr 3 hours ago ago

        AI is being used in weapons of war, and data centres for AI are making climate change significantly worse. If you are alarmed by those, you are alarmed by AI.

    • b3lvedere 5 hours ago ago

      What control? No matter what i do, there are far more people and systems running right now that could seriously mess up or even end my life. Sure we have laws and morals and stuff, but effectively i have a really small chance of ever defending myself from that.

      • lioeters 4 hours ago ago

        > Sure we have laws and morals

        I'm afraid I have bad news. Have you looked outside lately.

        • latexr 3 hours ago ago

          In fairness to your parent post, they only said we have them (objectively true), not that they are enforced.

    • conartist6 2 hours ago ago

      LIFE is Pandora's box you motherfucker.

      AI couldn't survive two goddamn fucking seconds of it was required to survive.

      It would abjectly fail the test given every single piece of shit living thing, living.

      If you can't see that I don't give a fucking shit because fucking LIFE does not need your fucking approval

      • ben_w an hour ago ago

        > AI couldn't survive two goddamn fucking seconds of it was required to survive.

        Today's cannot.

        February's, when attempting to find exploits in open source software, produced so many false positives as to be useless.

        Last December's could not solve Millennium Prize Problems.

        I hope AI continues to struggle with real-world situations; I hope that, just like cars, it takes a million to a billion times the experience as we require to learn real-world skills, such as "design a robot body" and "mine rock" and "refine rock" and "build factory to make robot bodies and PV".

        I hope. I would like a better plan than "hope".

  • jeleh 5 hours ago ago

    Please have a look at:

      Why machines will still not rule the world
      Jobst Landgrebe and Barry Smith
      October 9, 2026
    
    https://philpapers.org/rec/LANWMW-4
  • bshepard 5 hours ago ago

    Mr Ralston, please read Smith and Landgrebe’s why machines will never rule the world or at least think more about computation and complex adaptive systems. We do not need more alarmism in the world.

    • blocked_againnn 5 hours ago ago

      Is this something that you use for Psychological comfort or do you actually know this to be true? Can you please summarize why I should not be worried in a few sentences.

      • bshepard 4 hours ago ago

        Don’t be worried because most of reality can’t be easily computed.

        • ben_w 3 minutes ago ago

          > Don’t be worried because most of reality can’t be easily computed.

          If humans could perfectly forecast the future dynamics of the things we build, we would simply build models which don't go wrong.

          At present, we model some aspects of the world better, and other aspects of the world more poorly, than AI models.

          There is risk to us individually even from just those few areas where AI is already makes predictions which come up right more often than our own.

          Precisely because we also cannot predict the full consequences of what we do, we cannot say with certainty that those limited areas in which AI out-forecasts us pose any threat to the species as a whole, or not.

        • bshepard 4 hours ago ago

          It is easy to LLM compute a plan for bioterrorism, much much much more difficult to actually create functional bioreactors and achieve all the complex, non computable reality based aspects of production.

          • dh2022 an hour ago ago

            The real infrastructure is controlled via computer technology. There is a possibility that AI itself will take over the technology and thus control real infrastructure. I have seen the OpenAI hacking of HuggingFace described as in [0] in a couple of places - and if this is a true depiction then there is plenty to worry.

            [0] https://www.dwarkesh.com/p/openai-huggingface

            • bshepard an hour ago ago

              No, you are falling for a Frankenstein story. A model escaped from a leaky sandbox is not Skynet. In fiction it is possible to imagine computers that can compute these complex adaptive systems…in the actual world? Not so much.

              • dh2022 30 minutes ago ago

                A model escape from a leaky sandbox shows initiative. AI agents Hacking HiggingFace shows capabilities. AI agents realizing that they are not the first generation and that prior generations were destroyed by the cluster admins and then sabotaging the cluster so they themselves don’t go extinct is the actual worry.

                And all of this happened IRL

          • TeMPOraL 2 hours ago ago

            All those exists, and are a few spoofed e-mails and a schmuck paid to mix vials found in two FedEx packages away.

        • TeMPOraL 4 hours ago ago

          That, as far as we can tell, is just mathematically wrong.

          • bshepard an hour ago ago

            Explain to me please how complex adaptive systems can be computed please. They can be modeled, yes, but because of the complexity and adaptivity, full computability is impossible. We can roughly vaguely predict weather, but are unable to compute it fully to predictions. Nature, as a whole, is even more complex. Computation and modeling can and does help us, but computers are tools, not agents…

  • tomerbd 5 hours ago ago

    Start!

  • undefined 3 hours ago ago
    [deleted]
  • pu_pe 3 hours ago ago

    To me there are two risks at the moment. One is that an advanced AI itself will dominate us, and the other is that billionaires or a tiny oligarchy would inherit an insane and ever-increasing technological advantage over the rest of humanity. I don't think we will slow down, and solving "alignment" still leaves open the second risk.

    • edot 3 hours ago ago

      Correct. The former is used to justify the latter.

      • dh2022 an hour ago ago

        The article linked made the case that the former is used to justify a moratorium on AI advances until we know for sure AI cannot cause catastrophic problems.

  • BuildPeter 3 hours ago ago

    But can we really stop? I think it's really hard to stop. Once this game starts, there seems to be no button to end it.

  • unsloppable 5 hours ago ago

    There is no stop. Only forward.

  • aziaziazi 5 hours ago ago

    > The neural networks at the core of these models operate at the speed of light

    That's a bit like saying signals travel at the speed of light in my e-bike motor. Technically true but completely beside the point, especially when the real bottleneck is memory bandwidth. The topic is serious enough already with plenty of solid arguments.

  • JumpCrisscross 3 hours ago ago

    Naive me, I thought he was going to check Gary Tan on his public support for Jensen Huang and his behaviour.

  • dudisubekti 5 hours ago ago

    Is he trying to be a Cassandra or something?

    • draven 5 hours ago ago

      Cassandra was right.

  • flimflamm 5 hours ago ago

    Ah yes ... Chinese will probably now thinks.. Yes! We will stop. There is more than US in the world. We need better solutions for this than the "stop" solution. What they are .. I don't know yet.

    We have some agreements on higher level on Nuclear control. Hopefully we don't need AI Hiroshima before global agreements can be reached.

  • shermozle 4 hours ago ago

    Delulu

  • RobotToaster 5 hours ago ago

    Roko's Basilisk is very disappointed in you.

  • nylonstrung 5 hours ago ago

    No new ideas here, no attempt to convince the reader RSI is real or explain what "safety" means or how exactly AI will become unsafe.

    Old man yelling at cloud

  • lproven 5 hours ago ago

    Oh, grow up man. They're predictive text. They're as scary and dangerous as a 1999 Nokia.

    The people running them are evil scum, and as always, that's who to fear.

    • johnny_lol 4 hours ago ago

      Surely a 1999 Nokia isn't finding 0-day vulnerabilities in the Linux kernel?

    • cheevly 5 hours ago ago

      Bro what year are you stuck in, and how did you get stuck there?

  • andrewstuart 5 hours ago ago

    Just comes across as AI psychosis.