The AI Superintelligence Debate Has Serious Flaws

(spectrum.ieee.org)

94 points | by rbanffy 4 hours ago ago

155 comments

  • saagarjha 3 hours ago ago

    I don’t necessarily agree with the labs’ policies and predictions on RSI but it’s worth noting that the points raised here are something they do have responses to.

    On the idea that training better models requires more data, I think the current thought here is that we are not getting all the “meat off the bone” of the data we have, so there is still quite a ways to go with training that AI can improve. After all, it’s not like Anthropic is doubling their data set every year (?) and yet their models definitely improve by more than that.

    For computability, yes, obviously the models cannot violate mathematics. But there’s a lot of things that are very computable and very desirable, or correspondingly very computable and very bad. This is enough to be worth thinking about.

    For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can. However the issue is that doing this takes time and very few people (especially the frontier labs!) seem to be doing so. It’s much easier to hook your LLM up to the world and ask it to go than ask, “is this safe?”

    • ben_w 3 hours ago ago

      > For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can.

      This does not seem at all obvious.

      Any system worth using has to actually do something beyond the boundaries of that system. Even if that consists only of writing responses to prompts, the moment those are connected to any thing they can alter the state of that thing.

      Making that safe means it only ever produces a safe response. Even ignoring malicious uses, it is unclear how to correctly classify what "safe" means on an unbounded time horizon.

      • pvab3 3 hours ago ago

        We need to stop using the word "Alignment" because it clearly has different meanings for different people. Some people consider it to be preventing the model from waking up and enslaving or destroying humanity a la paperclip maximizer. Some people mean it as preventing the model from assisting with bad things like hacking, cybersecurity exploits, building weapons, and other things that are aligned with that individual user's goals, but not those of society more broadly.

        • ben_w 2 hours ago ago

          Those are, unfortunately, much the same thing.

          The paperclip optimiser thought experiment isn't "AI does this spontaneously", it is "AI is told to do this by someone who didn't realise the consequences, and it succeeds".

          • watwut 2 hours ago ago

            They are not the same thing at at all. One is hypothetical philosophical long term maybe threat and other are real current day issues. And practically, the paperclip optimizer issues is used primary to distract from actual real world harms that happen today:

            - slopification, - harm to mental health, - harm to climate change, - centralization of power, - ...

            Plus "alignment" is fundamentally nonsense. There is no such thing as generally accepted "human interests". Nazis idea of "human interests" is not the same as mine. Which is not the same as Chinese communist idea of human interest, catholic pope idea of human interest or whoever else.

            • barbariangrunge an hour ago ago

              Every time somebody mentions morality on hn the response is nihilism. Convenient but scary for the frontier of the tech sector

              • ben_w 26 minutes ago ago

                Is it convenient? Nihilism's pretty dark and few like it.

                If anything, I'd say Thiel's ongoing invocation of biblical imagery is more convenient, especially in the USA which is rather more Christian than Europe or China. It allows them to invoke "god's will" against any potential doom, and say we get heaven anyway if there is doom.

                Utilitarianism comes up a lot too, though nobody likes it for a whole variety of reasons, not even philosophers who study it like it.

            • ben_w 43 minutes ago ago

              > They are not the same thing at at all. One is hypothetical philosophical long term maybe threat and other are real current day issues. And practically, the paperclip optimizer issues is used primary to distract from actual real world harms that happen today:

              If you think they're different, which bucket would you put the HuggingFace incident into, given the researchers did not ask for HuggingFace specifically to be hacked, they asked the AI to solve a smaller challenge and the AI determined the only way to get the right answer was to cheat, and the AI also determined that it would be punished for cheating so it tried to cover up the cheating?

              > - slopification, - harm to mental health, - harm to climate change, - centralization of power, - ...

              Dunno about you, but I can be concerned about many things at the same time.

              Centralisation of power is an odd one to put on this list, given that it's a step for most take-over scenarios.

              > Plus "alignment" is fundamentally nonsense. There is no such thing as generally accepted "human interests". Nazis idea of "human interests" is not the same as mine. Which is not the same as Chinese communist idea of human interest, catholic pope idea of human interest or whoever else.

              Perfect alignment may well be impossible for this reason, but nobody knows yet. Humanity is not ready to discuss "perfect" at this time.

              Thermodynamics analogy: you can't make a perpetual motion machine, but you have to invent the laws of thermodynamics to know that, and by inventing them you can absolutely make much better steam engines than you would have without those laws.

              Right now we don't know how to do reach even such a low bar as "don't kill everyone"; absent developments in this field, if we manage to pass this, we will have done so by accident, not because we know what we're doing.

              None of the groups you listed want to see the world taken over and its mountains levelled by the British East India Paperclip Corporation; everyone had enough of that kind of nonsense a century ago even when it was humans doing it and optimising for money.

        • dweinus 2 hours ago ago

          Wow, they really misuse "alignment" in the article! They give a decent definition, and then proceed to ignore it.

          Alignment == "training an AI system to act in a manner that aligns with human interests"

          ...but then they talk about guardrails and permissions as part of alignment. I think this is their weakest argument

          • tsimionescu 2 hours ago ago

            Well, all of the heavy lifting in that definition is in how you interpret and draw the lines around "human interests". And even in the most minimalist interpretations of human interests, you still need guardrails to prevent the AI from doing wildly dangerous things.

            For example, say you ask "get me plane tickets to Athens as cheaply as possible". Without any guardrails, the AI might hack the airline to try to get you tickets, or it might extort someone else for tickets, or any number of other wild schemes that would still fulfill the goal you set out for it.

            And say you actually explicitly ask it to get you tickets to Athens as cheaply as possible using any means necessary, legal or illegal - the AI might easily land you in jail or get the flight canceled, thus entirely failing to accomplish your goal.

            • ben_w 23 minutes ago ago

              > For example, say you ask "get me plane tickets to Athens as cheaply as possible". Without any guardrails, the AI might hack the airline to try to get you tickets, or it might extort someone else for tickets, or any number of other wild schemes that would still fulfill the goal you set out for it.

              Instantly reminded of that case with the gym class: https://techcrunch.com/2026/08/10/tech-industry-is-buzzing-a...

        • SpicyLemonZest 2 hours ago ago

          What “alignment” refers to is the ability to robustly convince a model to achieve anyone’s goals. This is really quite obvious and nearly everyone intuitively recognizes it; you use the word “aligned” this way yourself!

          The source of the debate is that some people have a strong preconception that alignment can’t be a hard problem. GGP takes it for granted that alignment is solved and anyone who wants to can build a system that won’t cause harm. So picking a new word won’t help.

      • saagarjha 3 hours ago ago

        I think you’d need to broaden your scope too far as to be useful for something like that. If we hook AI up to LHC, well, the immediate impact is obvious in that some physicists can’t use the detector. But it’s hard to ascribe “we neutron bombed the Milky Way in 42069 AD” to it finding a theory of quantum gravity.

        • ben_w 2 hours ago ago

          More short-term, does the AI know when it is safe to switch it on and off and not e.g. turn it on while someone's head is in the beamline?

          Can it find risk factors for exotic imtermediate products and their decay chains and cross-sections from already-known physics, that humans have missed?

          Mid-term, is it committing scientific fraud with falsely-claimed discoveries?

    • BugDeveloper 3 hours ago ago

      About computability: All functions on finite sets are computable and superintelligence doesn't require a function on an infinite domain.

    • shishy 3 hours ago ago

      > but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can

      How? Many failures are emergent properties in complex systems -- you can capture some of them but this doesn't really seem obvious to me?

      • saagarjha 3 hours ago ago

        This usually means not connecting AI to complex systems

        • pixl97 2 hours ago ago

          And there are so many examples of why this is impossible in itself. The biggest ones is humans are stupid and will connect them to complex systems.

        • jstanley 2 hours ago ago

          If you so much as read the output of a sufficient intelligence then it can accomplish anything it can persuade you to do.

          You don't know how effective persuasion can get because you've only ever encountered ordinary human levels of persuasion.

    • soco 3 hours ago ago

      Training a human brain needs and uses much, much, less data than training the AI models. So maybe the expectations on the AGI are, dunno, just optimistic? I mean you can put tons and tons of gas in a car but that won't make it drive faster.

      • origini 3 hours ago ago

        This could also be the perfect argument of why ASI is just around the corner: One breakthrough in architecture and we already have so much more training data then it ever need again.

        If we can make smart humans with a lot less, why do we even need that much for an LLM?

        Perhaps world models will bring ASI, perhaps when LLM progress slows down, labs start to explore a lot more architectures and suddenly find it or the next Mythos will design the ASI architecture.

      • jeremyjh 3 hours ago ago

        Human brains are the product of millions of years of evolution. You have to include that training data as well for it to be a fair comparison.

        • zarzavat 3 hours ago ago

          All that evolution compresses to a GB or so of DNA (yes I know it's more complicated than that).

          Humans are a bad example because of the well known width restriction leading us to be born undeveloped.

          But animals can be born with varying levels of fully or partially developed neural networks despite having minimal data transfer from the parents and no apparent training data available in the womb/egg.

          Nature is able to squeeze a lot more efficiency than our current AI efforts.

          • filoeleven 2 hours ago ago

            You're neglecting the influence of bioelectric gradients, and their storage of memory. See Mike Levin's work, where his lab created 2-headed flatworms that are genetically identical to regular 1-headed flatworms. I'm not convinced there's "minimal data transfer" when we have this whole new field opening up that shows memory is not at all limited to the brain/neural networks.

            > These animals do not differ from wild-type worms in histology, expression of key polarity genes, or neoblast distribution. Instead, the altered regenerative bodyplan is stored in seemingly normal planaria via global patterns of cellular resting potential.

            https://pmc.ncbi.nlm.nih.gov/articles/PMC5443973/

          • saagarjha 3 hours ago ago

            Seems like nature has a pretty cracked 1B parameter base model and fine tuning stack

        • ftchd 3 hours ago ago

          I would argue that's the actual "training" part, and your lived experience is the equivalent of fine tuning.

          • jeremyjh 2 hours ago ago

            I think its closest to pretraining. It provides blueprints for all the cognitive structures. Then your experience shapes what you can actually do with that machinery. The advantage AI has is post-training doesn't require so much specialization (though still does require A LOT more experience).

        • ben_w 3 hours ago ago

          Kinda, but unclear how much pre-training of our brains evolution did exactly, only that even if DNA was just about the brain (it isn't) it would still be a tiny pre-train.

        • soco 3 hours ago ago

          Fair point. Those millions years of training however are not really contained in the corpus of data we are currently using in training AI, also the hardware is quite different - and created by processes with huge and significant differences, so I don't even know what apples with what oranges I should compare...

          • jeremyjh 3 hours ago ago

            You can’t directly compare them, is my point.

      • rileymat2 3 hours ago ago

        I have no concept of how much less data it uses, we consume video and audio constantly when not sleeping. We do this for years before a baby can talk.

        • Symmetry 2 hours ago ago

          An LLM is trained on the order of 10^14 bytes of text.

          At roughly 1 MB/s of data to the brain (vision dominates even though the data along the optic nerve is way less than arrives at the retina) times the seconds in 20 years that's roughly 6 * 10^14 bytes. Not too large a difference.

          • AnimalMuppet an hour ago ago

            But it's different data. The human training bandwidth is dominated by physical things. There's a thing I see, and I move my hand and I touch it, and I hear the sound of the impact, and I feel the touch. And I learn how all those are correlated. I learn to build a mental map of the physical world that corresponds pretty well to the actual physical world, so that I can survive living in it.

            LLMs are trained primarily on text. That trains them on relationships between words more so than correlation to the physical world.

      • dosinga 3 hours ago ago

        You can also use that as a counter argument. Training AI on enormous amounts of data has resulting in pretty decent LLMs still far from AGI. We're running out of data so seemingly this will limit what LLMs can do. Human brains seem to do much more with a lot less training data so presumably there's still a lot we can do to improve just the training algorithms.

        • pixl97 2 hours ago ago

          We may be running out of language data but not data itself. There's a whole Lotta world out there.

      • XorNot 3 hours ago ago

        The counter argument is that it uses less text, but a lot more everything else. Any new parent could tell you the surprising number of things which babies just don't know: which seems like an obvious statement, but babies don't know how their body works. They don't know how their digestive systems work, in most cases there's an entire growth process for those systems to be usefully capable for them.

        All of that though, is training data. All our sensory inputs form an enormous base of input which I doubt the volume and content of the internet at its highest strata properly represents: so our AIs are developed from a much higher stratum of reasoning and this showed up pretty readily in early models - they could write language well, they absolutely had no grounding in how physics worked, for examaple.

      • squidmaster 3 hours ago ago

        [dead]

    • majormajor 3 hours ago ago

      > For alignment: it’s not actually a question of alignment at all, but rather one of “can we ignore alignment and design systems where the AI cannot cause harm?” Obviously, we can. However the issue is that doing this takes time and very few people (especially the frontier labs!) seem to be doing so. It’s much easier to hook your LLM up to the world and ask it to go than ask, “is this safe?”

      Much easier to pump up the numbers to investors to sell the quick-and-dirty solution. These companies are trying to speed-run things in unsafe ways that (just so coincidentally, I'm sure) happen to be exactly what the things you'd do if you were just in it for the money, and I wish we wouldn't be so accommodating.

      All the "are we creating god" talk is similarly window-dressing nonsense marketing fluff. Gods throughout the history of mythology have been dumb as often as they've been superintelligent, the defining attribute of them is power not reasoning speed. If humans give massive rapid destructive power to a computer program it's a human error, not a god escaping us.

    • api 3 hours ago ago

      Having responses doesn’t mean anything. Are the responses valid and grounded in reality? Is there evidence for them?

      A skilled debater can argue any arbitrary position convincingly.

      I thought of a short LT;DR version of why I am not a Rationalist: they think arguments are evidence.

      • ben_w 3 hours ago ago

        > I thought of a short LT;DR version of why I am not a Rationalist: they think arguments are evidence.

        You're thinking of a-priorists.

        Rationalists do not (at least not willfully) think arguments are evidence.

        • card_zero 3 hours ago ago

          Evidence isn't arguments, at any rate.

      • card_zero 3 hours ago ago

        > why I am not a Rationalist: they think arguments are evidence

        That means something, right? But it's an argument.

      • saagarjha 3 hours ago ago

        I mean, the post doesn’t have evidence of its arguments either. It’s hard to come up with evidence for something that does not exist

  • alecst 2 hours ago ago

    I think this article gets it wrong, which is unfortunate:

    > Vernor Vinge pointed out, rapidity in cognition is not the same as intelligence. Speeding up the mind of a dog, he argued, wouldn’t enable the animal to play chess.

    True, he writes about this in his essay "On the Coming Technological Singularity." His point is that there's some hypothetical possibility that computers will get fast but not smart--i.e. a dog thinking a million times faster than a human is still a dog. But seeing as computers have solved Jacobi Conecture, quasi-Riemann, and Navier Stokes--and we're in the early stages of LLMs--I think this hypothesis is bunk. The machines are smart. Foolish in some ways, but still smart.

    > Their construction prevents such an outcome: Instead of prompting an LLM to generate an open-ended strategy for pursuing a general goal, these AIs consider much more limited spaces of possible actions

    Well this is what Yudkowsky has advocated for--models that are trained on limited subspaces, e.g. protein folding sequences and nothing else. So I don't think this is a counterpoint.

    On a personal note: Has "alignment" even been defined in a consistent way? How can AI even be aligned if people are not aligned?

    > Objection #1: Recursive Self-Improvement Is Speculative

    I don't think it's speculative since it seems to already be happening. OpenAI wrote about how use of agents in research is climbing quickly, and the goalposts for intelligence move every day ("Image generation can't do hands", "video generation can't be consistent", "context length is too small", etc., I could go on.)

    This article makes me feel worse. These are bad arguments.

    • arw0n 30 minutes ago ago

      > But seeing as computers have solved Jacobi Conecture, quasi-Riemann, and Navier Stokes--and we're in the early stages of LLMs--I think this hypothesis is bunk. The machines are smart. Foolish in some ways, but still smart.

      We are seeing much larger progress in verifiable use cases than others. It is just unclear how generalizable success in maths and programming is to other more fuzzy problems in the case of LLM intelligence. I agree that LLMs are intelligent, but it is a very alien kind of intelligence compared to us.

      The larger, more important point is that we do not know if intelligence is bounded. Maybe humans are already close to maximum viable intelligence. To stay with your analogy: A human thinking a million times faster than other humans is still a human.

      > I don't think it's speculative since it seems to already be happening. OpenAI wrote about how use of agents in research is climbing quickly, and the goalposts for intelligence move every day ("Image generation can't do hands", "video generation can't be consistent", "context length is too small", etc., I could go on.)

      There's a broad and a narrow definition. Broadly, yes, AIs are used to develop AI. Narrowly, it does not seem like they are continuously accelerating the progress. Exponential capability increases would become visible quite quickly.

      > On a personal note: Has "alignment" even been defined in a consistent way? How can AI even be aligned if people are not aligned?

      Agreed that this is not obvious. I think there could be baselines like "Do not kill" or "Do not break the law", but expecting someone to align fully with an external set of values is presumptious.

    • krona an hour ago ago

      > agents are doing the majority of the research

      ML researchers have used algorithms to explore parameter spaces for solutions to optimization problems for decades. Now we're just anthophomorphasing the same algorithms and saying it's the 'AI' doing the research.

      As for Go (the game) or Mathematics, there are clear reasons to expect self-improvement. For every other endeavor which necessarily makes contact with reality, expect very mixed results.

    • Voultapher an hour ago ago

      > But seeing as computers have solved Jacobi Conecture, quasi-Riemann, and Navier Stokes--and we're in the early stages of LLMs--I think this hypothesis is bunk.

      I'm not sold, math with rapid-iteration proof checkers, is the best-case for these fuzzy search approaches. I still see those achievements as fuzzing on steroids not indicative of deeper intelligence. Lest we not forget the serious plagiarism allegations surrounding those discoveries.

      Having a near-perfect "Am I wrong" oracle and huge amount of attempts makes being right possible even if you are wrong nearly all the time. For the majority of human decisions knowing what worked and what didn't has long time horizons and is hard to quantify, and we get very few chances.

      • alecst an hour ago ago

        > fuzzing on steroids not indicative of deeper intelligence

        I honestly think that the goalposts are just going to get moved to infinity.

        If solving Millennium problems doesn't take intelligence, what does? Writing, composition, photography...? You can't fuzz your way through this stuff. You can't even fuzz your way through writing this paragraph--it's statistically impossible--the universe would die long before you finished it, unless you had some sort of steering principle.

        Wait til the machines start asking if humans are genuinely intelligent, or are just fuzzers on steroids. That'll be an interesting wake-up call.

      • keybored an hour ago ago

        > For the majority of human decisions knowing what worked and what didn't has long time horizons and is hard to quantify, and we get very few chances.

        But the trained software engineer is happy to drive the car down the hill in order to see if the crash can be repeated.

    • kannanvijayan 35 minutes ago ago

      Goalposts haven't moved - expectations have.

      I think it's pretty clear that the Turing Test has fallen. If that's the standard for intelligence then we've passed it. Over the years there have been various litmus tests proposed: beating humans at chess, and then Go, and then "doing research", etc.

      For a large segment of the population, these are good enough already that our tendency to anthropomorphize fills in the gaps steps in. It's clear many people are talking to LLMs like they're people.

      By various metrics, "super" intelligence has already been achieved in one or the other domain. Algorithms got superhuman at chess a while back.

      The big underlying question is whether the idea of some AI singularity itself isn't some fundamentally nonsensical concept. Similar to how often singularities in other areas are actually just artifacts of the framing or coordinate system.

      Every technological age has a cohort that proclaims it as the end of everything. In the early 1900s we had solved all of physics and were on the cusp of answering all questions the universe could throw at us, according to many at the time. The early parts of the prior century saw similar grandiose prognostications. Eugenics would finally solve the problem of human justice. Engineered microwaveable dinners would solve human nutrition. etc. etc.

      Ultimately it's true that society change drastically in an irreversible way due to these technologies, as I'm sure it will due to AI. But the appropriate analogy there is an event horizon, not a singularity. And even then, it's a rhetorical device, not a real thing. It's conjuring an idea that can't correspond to anything in reality - sort of like how you can define the concept of a perpetual motion machine and also understand that the concept is not a reachable reality.

    • keybored 2 hours ago ago

      Don’t feel bad. AI Doomers are the yin to the yang of AI Hypers. The HN machine needs both (for the sake of AI).

      • jsnell an hour ago ago

        It's not a spectrum between those two, it's a triangle where the end points are doomers, accelerationists, and luddites.

        100% doomers think AI will work and kill us all. Accelerationists that it will work and deliver utopia. Luddites that it is a scam that won't work.

        Cal Newport is very far into the luddite part of the triangle, to the extent that it doesn't even make sense to think about how much doomer is in the mix.

  • vanelsas 3 hours ago ago

    I find the discussion academic at best. I am more worried about a swarm of dumb AI's causing havoc on infrastructures, causing real-life problems like food, energy and water shortages everywhere. Physical harm is a real possibility, and it seems most AI companies don't care. There is no harness for stupidity and irresponsibility.

    • barrell 2 hours ago ago

      I would go so far as they are optimizing for such things.

      I’ve told the story many times, but when gpt voice came out, I used it every day. I found it great to be able to talk with the internet as a way of research.

      Then after a few months, it started clearing its throat. Then taking breaths. I stopped using it over the course of a week.

      I don’t see any reason why a voice agent should sound convincingly human, clear it throat, take breaths, say hmm, giggle like a schoolgirl, sigh etc for any non-nepharious reasons. For me, it actively decreases the non-nefarious uses of the product, and increases more-or-less only the nefarious uses.

      Then frontier labs go on to talk about how they’re worried their product would be used nefariously.

    • yasnu5 3 hours ago ago

      [dead]

  • jeremyjh 3 hours ago ago

    I agree with most of this but I do not think the author understands either the Hugging Face incident or what alignment really means. Reward hacking is a serious issue and the tendency for models to do this is a serious problem that we’ve made approximately zero progress on resolving. We pile on more external guardrails and hacks but those methods will always be vulnerable.

    • thrw045 3 hours ago ago

      It's true that the more capable a model is, the more dangerous reward hacking is, but there is a giant elephant in the room. Namely that these ai models still can't create their own high level goals. This means that if there were to be an incident of hacking or whatever, this would have to be launched by a human directly prompting or them setting up an rl environment in some way that lead to it.

      There is a _huge_ difference between an ai model that follows a top level goal given by something else and one that can give itself its own top level goals. The latter would essentially be like a human or other organism and would be very dangeorous the more capable it is.

      The other smaller elephant in the room is that if reward hacking is so dangerous, why don't we see more rogue agents out when we are prompting them ourselves? For the most part, the agents do what they're supposed to do as guided by the prompt, so it seems like to me that at least at present, only agents that are trained on hacking data AND put in an rl environment or given a prompt related to hacking, as well as asked or trained to persist will actually try to do that hacking. I would be worried if chatgpt or claude agents started to hack something every time you asked it to fetch some data from the web or something. So that's one question I have is why does hacking and going outside instructions not happen in normal usage of the models? Does it but they actually work it out of the system in some way? Or is it the other way around where you have to create the conditions necessary for it to happen and these conditions are exceptional not the norm?

      So I'm not trying to undermine this, I think there are things to be taken seriously and thought about but I always wonder why the people from the big labs never include all of this information. They always talk very abstractly about RSI, hacking and extinction without any caveats about what would be needed for that happen. I'd be happy to be proven wrong on this also.

      • IanCal 2 hours ago ago

        > or given a prompt related to hacking, as well as asked or trained to persist will actually try to do that hacking.

        But in the HF incident they didn't try and "do that hacking". It wasn't like they were told to try and hack a fake service and they actually hacked a real one. They hacked into HF to discover how a "find the security issue in this code" test was going to be scored, and tried to change their own logs so it wouldn't be found out.

        • thrw045 2 hours ago ago

          OK fair enough but as far as I read there were still unusual parameters like being asked to persist and being trained on cybersecurity data and other things. If you prompt a public LLM atm it's not going to hack things randomly to try to achieve the goal in the prompt. It's hard to know the exact setup and parameters but it seems like those counted for how the agents behaved.

      • blargey an hour ago ago

        The more capable AI becomes, the more abstract and broad and open-ended our "prompts" will become. Just as we moved from prompting minor edits and code-completion to whole tool scripts to vibecoded projects and "I feel like I'm a manager now!" to baby "software factories". As long as performance keeps improving, it'll keep moving up the ladder of goal-abstractness and leeway to interpret prompts as it sees fit, where "fit" is the artifact of weird benchmark optimization...

        I disagree that "Give itself its own top level goals" is going to be the clean line you're making it out to be.

        • thrw045 23 minutes ago ago

          While it may not be in practice, there is still a difference between terminal goals and instrumental goals philosophically. If the AIs goals are still subordinate to the higher up terminal goal then it didn't really choose what to do in the grand scheme of things. There is some authority it's lacking even if you say "Improve humanity" as a prompt, it's still subordinate to that goal. Humans are not. Humans can in theory create their own terminal goals, because to say otherwise would kind of remove the idea of terminal goals altogether.

          There's also the idea of control. You can in theory have an incredibly powerful AI that you give abstract goals to, but the moment you ask it to stop, it stops. It follows your orders basically. The two are not mutually exclusive. We don't see in current systems an increase in autonomy that is proportional to capability. At least from where I stand, autonomy can remain zero while capability can increase n fold, but that's not proven, but then again it's not proven that autonomy has to increase the more capability the AI has either, so I guess the jury's out?

      • origini 2 hours ago ago

        How is this a 'big elephant in the room'?

        Every random prompt can become a top level goal. "Research my water provider" -> "searching your water provider" -> "found the webserver" -> "found additional ip ranges, scanning them" -> ...

        Or just any hacker trying something out, a state hacker etc. Google just wrote about fully automated ai hacking infrastructure / ai warfare.

        And they are all working on private agents and personal assitenace. These are inherantly creating top level goals.

        'Still can't create' also doesn't mean that we might change this very soon. As soon as this crazy time settles just a bit, hardware getting more common and cheaper again, pretty sure people are researching how to build a default mode network generic enough into an AI to explore this.

        • thrw045 2 hours ago ago

          But you seem to agree with me in the last paragraph even though you talk against me in the prior ones.

          If an AI does get a 'default mode network' and sits idle and speculates on what to do next and then executes plans and explores its own internal information at will then I agree with you. But I think creating this might be harder than it looks and I'm not sure when it will happen or if it's a good idea but that's a separate discussion.

          My point was only that at the moment, AIs don't direct their own top level goals. It's not going to just "wake up" one day and try to take over the world as some people suggest. That is what I was arguing against.

          • origini an hour ago ago

            Yes i agree with it, but my main arguments were that we have something close enough which is bad enough potentially anyway.

            • thrw045 14 minutes ago ago

              Yeah agreed.

      • SpicyLemonZest 2 hours ago ago

        > The other smaller elephant in the room is that if reward hacking is so dangerous, why don't we see more rogue agents out when we are prompting them ourselves?

        Because OpenAI and Anthropic deliver their most powerful models along with classifiers that stand in the way of potentially rogue actions. This works fine for now (in the HuggingFace incident they had turned it off for testing), but it will keep working if and only if alignment is solved; otherwise an agent swarm might be able to convince the classifier to join.

    • Upvoter33 3 hours ago ago

      Why isn’t this “just” a security issue? I mean, the security precautions around the hugging face attacks were amateurish no?

      • naasking 3 hours ago ago

        Yes, but humans have demonstrated that they don't care about security. If they did we would have switched away from C/C++ long ago.

        • ForHackernews 2 hours ago ago

          Humans care about security in some contexts. Nuclear weapons are generally well-secured. Maybe AI will finally make them care about software security.

          • pixl97 2 hours ago ago

            Nuclear weapons exist in the thousands, computing devices exist in the 10s of billions. Just cleaning up the existing mess we have is going to be monumental.

            I'm starting to think we'd be better off developing AI models that hack and just erase/kill the unit they hack to remove said device from the infected bot networks. After a few billion devices die maybe people will catch the clue.

          • cjcole 2 hours ago ago

            > Nuclear weapons are generally well-secured.

            This use of the word "generally" is the most frightening thing I've read this week and in the running for most frightening word usage of the year.

    • jdpigeon 2 hours ago ago

      Yes, I get the sense he's basically rolling his eyes and saying "of course it will be easy to sandbox agents, we're just not trying yet".

  • mitthrowaway2 2 hours ago ago

    > Amid these conversations, one question has received comparatively little attention: From an engineering perspective, Is superintelligence even possible?

    I get that the author is trying to frame themselves as a lone brave thinker, cutting through the noise with piercing clarity. But this question actually has received at least as much attention as the other points, even to the degree of inviting parody papers like "on the impossibility of supersized machines".[1]

    [1] https://arxiv.org/abs/1703.10987

    • 10111001 2 hours ago ago

      > I get that the author is trying to frame themselves as a lone brave thinker, cutting through the noise with piercing clarity. But this question actually has received at least as much attention as the other points, even to the degree of inviting parody papers like "on the impossibility of supersized machines".[1]

      I didn't read the parody, but nowadays, a computer the size of a human cannot be constructed in the same way as a smaller machine, due to latency constraints due to the speed of light, etc. You can't actually build a supersize machine, you have to build a cluster of normal-sized machines.

      • mitthrowaway2 2 hours ago ago

        The paper proves that it's impossible to build any machine that is larger than a human.

  • namr2000 2 hours ago ago

    I agree with the overall message of the article, but I do want to comment on:

    > An intelligence explosion leading to ASI would similarly depend on a long sequence of innovations of comparable importance, which would build on many significant science and engineering advances that haven’t yet happened.

    Personally I would have expected solving Naiver-Stoakes, Quasi-Riemann, or Exponential Time hypotheses to require the same thing, and either:

    * They did and AI swarms were capable of making such progress

    * It was easier than we thought

    Neither is particularly comforting. One could imagine that say continual learning techniques are similarly in reach and that could lead to something that looks vaguely like an intelligence explosion, even if it's not exactly the sci-fi horror that the boosters describe. It's really hard for me to model how likely that sort of thing is.

  • tescreal 2 hours ago ago

    It increasingly feels like anti-AI flavored arguments come in the form of moving goals further and further out. I don't think litigating an absolute frontier is useful overall, since it always ends up devolving into "intelligence of the gaps" where AI keeps filling gaps. There's always going to be some sort of ethereal special je ne sais quoi that humans possess more or less exclusively (much as a dog might have attributes that we cannot quite fully comprehend, even if we can describe it externally).

    The claims meanwhile don't stand up:

    * RSI is math and code heavy research (the things AI is improving most on)

    * The allegation about large swaths of data being necessary is a common assumption. This one is really sticky: humans do not need to read the breadth and depth of a subject to comprehend a great portion of it; AI improvements should yield larger payoffs from same & smaller datasets

    * Much learning and thinking is not about consulting existing work, but synthesis and recombination. AI has been revealing capabilities in this regard which should be factored in as it is further intensified (chain of thought being just the tip of the iceberg in my humble opinion)

    I am curious to see if somebody would do a rigorous analysis of some of these suspicious underlying assumptions. It would help at least orient expectations better.

  • jrflo 2 hours ago ago

    1) See the most recent METR report. We don't need pure closed loop RSI to still get RSI, humans are in the loop speeding up progress with the help of AI now if the report is to be believed. Also, the model's narrow set of expertise right now (coding and math) is extremely aligned with what skills you need to do AI research. And you don't need an even frontier to run into the worst-case scenarios for superintelligence, even if the superintelligence is only good at math and coding that's enough to cause damage.

    2) This argument is needlessly abstract. You don't need the set of algorithms to be countably infinite to be better than humans. The human brain has 86B neurons, the largest frontier models have 3T parameters. The human brain contains some algorithm to store human-level intelligence in a space 34x smaller than the current frontier, so it seems like the argument that "ASI is not computable" falls down. If you could replicate the human brain algorithm and put it in a 3T model, I'm sure you'd get superintelligence.

    3) I hope he's right.

    • Max345 2 hours ago ago

      2. Number of synaptic connections (~parameters) is around 100T, so 3T model is not there yet. Though only (unknown?) subset of human brain synaptic connections "thinks".

      • jrflo an hour ago ago

        Thanks for the clarification.

  • jokoon 3 hours ago ago

    We don't even know what intelligence is.

    I just want science to at least build a notion of intelligence, in terms of evolution, biology, psychology, neurology. We don't really know what animal intelligence is, and we only limit AI to human intelligence, which is very narrow.

    If we use AI to do human stuff, it's just an assistant for human goals. If humans don't have a specific goal, super intelligence will not guess or decide it for us.

    What does humanity want? Health? Mental health? Colonize Mars? Reduce income inequality? Solve climate change? World peace? Immortality? Happiness?

    AI can help for those things, but that's it.

    We tend to romanticize super intelligence like it would be some new life form, external to humans, a form of omniscient deity that could be a threat to humanity. There are worse threats: antibiotic resistance, nuclear war, climate change, dictatorships, mental health epidemics, obesity.

    • lukan 3 hours ago ago

      "We tend to romanticize super intelligence like it would be some new life form"

      How do you know it won't be?

      Right now it does not really meet the biological criteries, as it cannot reproduce on its own. But once it can mine minerals, build its chips and engines and maintain itself, what word we have to call that, if not artificial life?

      Doesn't mean it will have a soul, but that ain't a scientific concept either. Feelings and emotions? Is that for life required? But we don't know how that works either.

      (Apart from that, I would define intelligence as the ability to understand the world.)

      • krige 3 hours ago ago

        > But once it can mine minerals, build its chips and engines and maintain itself, what word we have to call that, if not artificial life?

        Will it? Or are you talking about several separate machines, most of which won't ever meet any criteria for life?

        • Orochikaku 3 hours ago ago

          Can’t the same be said about all multi-cellular life?

          • krige 3 hours ago ago

            Get out of here with these sophisms. No, a pickaxe, a truck, and a lithography machine is not at all like a cat.

            • lukan 3 hours ago ago

              But a cat consists of many small parts, cells and smaller, that would not survive and replicate on their own, just when they form a bigger collective. The same pattern of complex matter organisation would be artificial life. But if you insist, I can also think of machines, that have the replication parts integrated in them (miniaturized metal sinterers etc.) and just need to feed themself with certain raw matter and energy. Would that be artificial life to you?

              • krige 2 hours ago ago

                Can you spot the place where you moved the goalposts and started describing something completely different from OP?

                • lukan 2 hours ago ago

                  Not really, but I might have spotted the time for me to get out of this conversation.

      • AnneTrotter 3 hours ago ago

        I can build a theoretical brainless self-replicating zombie that does those things.

        And we actually do know how feelings and emotions work. Biochemistry. Machines do not have biochemistry. That means no feelings.

        "Soul" is irrelevant as it's inherently unproveable, so ignore soul as a concept for all definitions of intelligence and life.

        • lukan 3 hours ago ago

          "And we actually do know how feelings and emotions work. Biochemistry."

          We know that what we perceive is the result of biochemistry, but we don't know if it cannot also be based on other (quantum) process and we absolutely do not understand how feelings work, as it requires accurately moddeling the mind state of a person and as far as I know we are pretty far from that. Otherwise psychologists would be working with brain simulators to help cure their patients.

          "I can build a theoretical brainless self-replicating zombie that does those things."

          Also I am curious how you would do that.

        • LtWorf 3 hours ago ago

          > And we actually do know how feelings and emotions work. Biochemistry. Machines do not have biochemistry. That means no feelings.

          You seem to know much more than the collective rest of mankind. Perhaps publish some papers to inform the rest of us?

          • AnneTrotter 2 hours ago ago

            I hope the comment made you feel good, because it's utterly useless at anything else. The research is freely available, go forth and read it! I can't be arsed to spend my time predigesting neurology as a discipline for you.

            • lukan 2 hours ago ago

              No, but you could have provided one single link for your very unusual claim.

            • LtWorf an hour ago ago

              How is it freely available if you made it all up yourself right now?

    • tiborsaas 3 hours ago ago

      > We don't even know what intelligence is.

      Why do people keep repeating this? What is it so hard about intelligence to understand? We already have quite satisfying definitions of it. Why its evolution is a mystery?

      We tend to romanticize biological intelligence by treating it like some mystery, but it's "just" a way of evolution and natural selection. The better a species gets at problem solving, the better their chances are at survival.

      • immibis2 3 hours ago ago

        Oh yeah? What are the definitions, then?

        • tiborsaas 2 hours ago ago

          Oh yeah. Let's start with Wikipedia (https://en.wikipedia.org/wiki/Intelligence)

          "...ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought"

          This is followed by 8 other definitions by individual psychologists and learning researchers, all are interesting.

          Here's Douglas Hofstadter's take, where he compiled a list of 8 qualities of intelligence:

          (1) to respond to situations very flexibly; (2) to take advantage of fortuitous circumstances; (3) to make sense out of ambiguous or contradictory messages; (4) to recognise the relative importance of different elements of a situation; (5) to find similarities between situations despite differences which may separate them; (6) to draw distinctions between situations despite similarities which may link them; (7) to synthesize new concepts by taking old concepts and putting them together in new ways; (8) to come up with ideas which are novel.

          https://becominghuman.ai/douglas-hofstadters-eight-abilities...

          I could go on, definitions are numerous and most of them tend to converge to the same direction. Intelligence (or intelligent behavior) is not something of a mystery, it's not unique to humans, it can be found on wide spectrum in nature. Obviously studying it will and should continue especially in the age of AI, but we should not pretend that we are in the dark and know nothing about what it is.

          • Max345 an hour ago ago

            Base on that LLMs are event not close to have intelligence (they can't adapt to environment, learn from the experience) or for Douglas Hofstadter's takes, they can't do 1, 3, 4, 6, 8.

            • tiborsaas 11 minutes ago ago

              Processing my prompt can be interpreted as adaptation in order to produce the most likely answer. They can adapt to my style and respond to me in way that I prefer. Sure, it doesn't have its own will. As they don't act in the physical world I am its environment.

              Raw LLM-s that you can burn on a DVD can't learn, but AI systems which has additional components, RAG, memory systems, etc, definitely are able to extend their knowledge and apply them in subsequent runs.

              I'm only unsure about (8), but again, we need to define what novel means.

              Machine intelligence operates on very different principles and architecture so we shouldn't really expect that direct comparisons always make sense when it comes to biological ones. The rules are different, but the outcome can be compared.

    • squidmaster 3 hours ago ago

      [dead]

    • NichoPaolucci 3 hours ago ago

      > Health? Mental health? Colonize Mars? Reduce income inequality? Solve climate change? World peace? Immortality? Happiness?

      > AI can help for those things, but that's it.

      That’s it? What a let down.

  • rbehrends 2 hours ago ago

    I am wondering if conceptualizing it as "intelligence" is naybe a category error that disguises the underlying mechanisms that produce seemingly intelligent output.

    In my experience, basic LLMs without reasoning still tend to make very dumb mistakes (such as omitting or inserting a "not" or otherwise stating the opposite of the correct response). It seems that the combination of CoT (for fixing errors) and tool use (for grounding the inference process) play a major role in making LRMs work.

    I see LRMs not so much as a move towards intelligence, but as employing a heuristic-guided search for a goal condition (essentially pathfinding in token or latent space). Granted, a search with a very, very powerful heuristic, but the approach still seems to exhibit the same problems as other naive search algorithms, such as getting stuck in local optima [1]. And the answer for this by the labs when pursuing e.g. research math problems seems to be to throw agent swarms at the problem, i.e. pretty much the same quick fix we use to mitigate similar issues in classical search algorithms.

    See also the problem of models being very good in areas where they can build on a pre-existing theory, such as much of research mathematics, but can flail around [2] and just offer the obvious ideas in fields without one (as they lack a way to determine neighbor nodes for an effective search).

    Obviously, like pretty much everybody, I have no idea where AI will be a year from now; a new breakthrough can render any existing argument irrelevant, and we've already had a few of those during the past decade. But I don't currently see incremental improvements moving past existing limitations of how LRMs work.

    [1] E.g. https://bespokelabs.ai/blog/how-models-explore-ideas

    [2] E.g. https://www.lesswrong.com/posts/DALNiKBjMGMF8grme/arjun-pani...

  • fwlr 5 minutes ago ago

      The misaligned behavior that troubles commentators like Yudkowsky, therefore, doesn’t necessarily reveal something fundamental about advanced AI. It’s instead an expected side effect of the particular strategy of letting LLM outputs determine the actions of an autonomous agent without sufficient constraints. An obvious way to address these risks is to consider alternative architectures … [There is no risk of AlphaFold going rogue,] because its architecture does only one thing: It maps amino acid sequences to 3D protein structures. 
    
    If Dr Newport is going to be publishing extended critiques of Yudkowsky like this, it would behoove the good doctor to also familiarize himself with the subject matter. For instance, his “obvious” suggestion of “alternative architectures” that are more “task-specific” is substantially identical (even down to the choice of the same two example domains: protein folding and automobile navigation) to Holden Karnofsky’s critique of the Singularity Institute, some 15 years prior.

    While it is not word-for-word identical (Karnofsky used the example of Google Maps rather than Tesla Self-Driving; he calls it “Tool AI” and attempts to offer a formal general specification, vs Newport’s extensional gesturing at non-LLM “alternative architectures”), it is still so similar that Yudkowsky’s reply, also from 15 years ago, is an effective reply today. Excerpting with light edits from https://www.lesswrong.com/posts/sizjfDgCgAsuLJQmm/reply-to-h... (published 13th June, 2012):

      [It] may sound simple in English, … it's just "a thingy that says [the preferred action] instead of a thingy that goes and does things." … It starts sounding much [more complex] once you try to say something more formal and internally-causal like "Model the user and the universe, predict the degree of [the user’s satisfaction with the resulting state of] the universe, and select from among possible explanation-actions on this basis." … [Even eschewing probabilistic LLMs], you're not going to get anywhere near the one, true, optimal answer. You're going [to get] a good answer that's a cross-product of the "utility function" and all the other algorithmic properties that determine what sort of answer the AI finds easy to invent (i.e. can be invented using bounded computing time).
    
      `report($leading_action)` isn't exactly a trivial step either. … [AlphaFold] tells you that a certain DNA sequence will produce a protein that cures cancer, you ask "Why?", and then humans aren't even capable of verifying, for themselves, the assertion that the peptide sequence will fold into the protein [AlphaFold] says it does. “So,” you say, after the first dozen times you ask [a] question and it returns an answer that you'd have to take on faith, "we'll just specify in the [preference] function that the plan should be understandable.”…
    
    (Note that now, one of the specific tasks you’ve given this architecture is “figure out how to convince humans”)

      …Whereupon other things start going wrong. [AlphaFold’s newly installed explainer module explains that cancers are a very heterogenous group and the only weakness they all share is their reliance on a living host, so the protein is a potent biotoxin that acts as a 100% effective universal cure for cancer by deceasing all potential cancer-hosting organisms.] 
    
      [This is] an example of how things might go wrong if the preference function is over the accuracy of the predictions explained to the human— rather than just the human's 'goodness' of the outcome. And if the preference function was just over the human's 'goodness' of the end result … the AI might tell you something that was [wrong, but] would lead you to what the AI [understands of your definition of] a 'good' outcome.
    
      I'm not saying any particular failure is 100% certain to occur; rather I'm trying to explain … that the issue is a tad more fraught than it sounds on an immediate examination. … You've got some [floating point numbers inside a GPU,] with input from [a keyboard], output to a computer monitor, and optimization criteria expressed over some combination of the monitor, the humans looking at the monitor, and the rest of the world. It's a whole lot easier to call what's inside a '[task-oriented alternative architecture]' or some other English phrase than to [actually train a model] that does the optimization safely without serious unintended consequences. 
    
      Just as the gigantic gap between smart-sounding English instructions and actually smart [actions by agents] is the main source of difficulty [with LLMs], there's a gap between benevolent-sounding English [descriptions] and actually benevolent [alternative architectures] which is the source of difficulty [with “Tool” AI].  "Just make suggestions - don't do anything!" is, in the end, just more English.
    
    I’ll conclude by claiming that any intellectually honest observer should have noticed by now that Yudkowsky and the LessWrong rationalists have a substantial number of called shots with flight times measured in decades that are landing with remarkable precision, which is rather incongruent with the consensus view of them as cranks.
  • bryanlarsen 3 hours ago ago

    Recursive self intelligence is not going to make any difference. We're already using models to guide creation of their successor, and taking humans out of that process won't speed the process up. Mythos took months and and a ton of resources to create. A larger more capable model will take even more. Humans are no longer the limiting factor. Time, electricity and data centres are.

    If the singularity is coming, it has already started and will take many years.

    • jefftk 3 hours ago ago

      > taking humans out of that process won't speed the process up ... Humans are no longer the limiting factor.

      This seems very unlikely to me. While time and compute are clearly large components, we've also seen significant advances due to using that time and compute more effectively. A lot more seems possible here, since with how quickly everyone has been going this is more of a "rapidly scaling up the first things we've found that work well" than "carefully research how to optimize at every level". This is researcher work that is actively moving from humans to machines, and moving more of it there speeds the process up more.

      • bryanlarsen 3 hours ago ago

        Which is being done by researchers guiding Mythos. Mythos is doing most of the work, and is the limiting factor, not the researchers. Mythos guiding Mythos would not be appreciably faster. Mythos2 guiding Mythos2 would be, but that's because the workhorse is better, less so the guide.

        Yet despite the improvement, Mythos3 will take longer to create than Mythos2. The difficulty is increasing faster than capabilities.

        p.s. Anthropic uses Fable2 and Fable3 for iterational improvement names, I'm using it here for generational improvement names.

        • jefftk 2 hours ago ago

          I don't think your "workhorse and guide" division cuts in the right place. The key division is between "figure out how to make better models" and "actually make those models". The former is mostly intelligence-bound, while the latter is mostly compute-bound. For the former, today it's humans with AI assistance, and so better AIs speed it up a lot.

          (There are also similar factors in the datacenter buildout, but weaker there)

          • bryanlarsen an hour ago ago

            "figure out how to make better models" is mostly iterative/experiment based. You change something, run part or all of the "actually make those models" process, evaluate and iterate.

            The vast majority of the time is spent in "actually make those models". Speeding up "figure out how to make better models" makes very little difference. Speeding up "actually make those models" makes a huge difference.

            Being better at choosing which experiments to run makes a big difference, of course. But at the start of the singularity, humans and AI's will be at comparable levels of intelligence. Right now the combo of human+AI is the best at choosing which experiments to run, so that's what we're doing. When we reach the point that AI alone is just as good at human+AI at that part of the process, the difference will be negligible.

  • bjornsing 2 hours ago ago

    > Artificial superintelligence is, it should be noted, a far more ambitious goal than artificial general intelligence (AGI), which generally refers to AI that can match human capabilities across a broad range of cognitive tasks.

    Not sure I’d call it a (different) more ambitious goal. Even if you explicitly aim for AGI you might well end up with ASI. I see them more as two possible experimental outcomes, from the same experiment (scaling and post-training an LLM to the economic limit).

  • __MatrixMan__ an hour ago ago

    I think this article gets it mostly right. It comes down to this: The existence of general intelligence was never a settled issue. The AI world pretends that it is, and then talks about it as something that can be mapped onto a number line.

    But if there is no such thing as general intelligence, if the best we can do is a bundle of orthogonal intelligences, then RSI in one of them has no bearing on improvement in the others. And most of them will most likely not be applicable to improving themselves in that way.

    We're never going to be able to generate training data which teaches a kind of intelligence which we ourselves do not have. We might not even be capable of transferring some of the kinds of intelligence that we do have.

    Venture capitalists have a bias for number-go-up type thinking and in the AI industry we're seeing that it has transferred to their investments and created a blind spot.

    • mitthrowaway2 an hour ago ago

      ... Humans do exist though, I'm sure of it.

      • __MatrixMan__ an hour ago ago

        Sure, but are we generally intelligent? It seems rather arrogant to assume that our bag of tricks contains all possible tricks.

        Maybe we'll meet aliens one day and they'll be intelligent in ways that are totally novel to us, or in ways that we cannot comprehend because our physiology doesn't support that kind of intelligence. Maybe we already have, but we've failed to identify them as intelligent because our kinds of intelligence are orthogonal to theirs. Maybe the slime moulds are laughing at us for being so dumb.

        I'm not saying my vector-space model for intelligence is the right one, just that there's no reason to expect that the AI-industry's number-line model is right either.

  • origini 3 hours ago ago

    1) RSI:

    Give claude a task + compute and see how long it plays around finding a better solution. Doing experiments by itself, profiling, testing, etc. This is impressive and works locally already. More compute you give it, the more interesting things it does.

    Of course everything right now is highly speculative but this doesn't mean its not possible. With the current progress, its something highly speculative and plausable which means you should take it serious enough. Doesn't mean you have to go to Datacenters and blow them up.

    2) weird argument the author makes. 1 Million experts can for sure achieve a lot more a lot faster than 1. But they are also scaling models in parameter which is not just faster but also potentially more intelligent. If they have more compute and compute gets better faster and cheaper, than bigger models are also more plausable.

    3) It doesn't even matter if its going rough because it takes over the world or just because of learning as long as the risk is real that it stumble upon some network which has access to another private network which has access to some launch control pc or starts to mimick a world leader and manipulates humans and you find out, 100 years later, that the biggest disruption of modern society was a LLM agent simulating a book and 'simulating' some book ideas. Our IT infrastructure is fragile when you are able to react a lot faster and better than a human. Our banking system are computers, payment is digital, even plenty of power stations and water stations are controlled digitally remote.

    The flaws in these discussions are people downplaying all of it by either not understanding time or the current capabilities of basic LLMs. We are at the beginning of all of it, we are still 'just' using transformer architecture. The LLM progress is still so fast, that everything else doesn't get the attention it might deserve. The LLM is currently in a very fast and deep local minimum and we have no idea whats around.

    4 years ago loked very different in LLM and ML capabilites. People were laughing about the danger of an LLM now we have more critical CVEs since ever.

    We reached a point were it is cheaper to pay 1000 people to teach an LLM one thing instead of teaching 1000 people or more. It was a skill to be able to use Blender, now an LLM can use Blender. People on Fiverr probably getting a lot less jobs. Real media people are getting fired, translator, writers, etc.

    • astro1234 2 hours ago ago

      > Of course everything right now is highly speculative but this doesn't mean its not possible.

      I don't even think this is true either. A year or so ago lab leadership was worried about how they were going to keep progress going when they knew they needed a paradigm shift to handle things like continual learning or breakthroughs on training efficiency. To the point where they invested valuable resources in this bet.

      It turns out, this was actually somewhat counterproductive. It turns out that while it may not be the best or most efficient solution, the current paradigm is clearly enough to close the gap to super intelligence.

      No one seriously paying attention to performance trends and what's happening at frontier labs that I'm aware of thinks that RSI is more than a year away.

      What that will actually look like and the societal implications are more speculative than the actual model capabilities.

  • imeanwhatdoikno 3 hours ago ago

    Can somebody comment on the evidence for the following sentence from the article: "After GPT-4, however, continued scaling [number of parameters] began yielding diminishing returns"?

    My understanding was that scaling in size slowed for practical/economic reasons, not because of its diminishing intelligence returns.

  • Hussell 2 hours ago ago

    Weak arguments, despite the headline (the headline here, it's different there). "Recursive self-improvement is speculative" and "based on unproven assumptions". "Superintelligence might not be computable". Worst of all: "Alignment Isn't Hard". To me it comes across as "it hasn't been done so it probably can't be done" plus "if I did it, I'd do it better". Weak arguments all around, regardless of who's making them.

    What it comes down to is: the frontier labs have been improving the ability of A.I. to think (in the same way that a submarine can swim, no arguments about this please). They think at least a thousand times faster than us. The labs seem to have two goals:

    1) Make A.I. smarter than us. The labs seem to be succeeding at this, with obvious progress and no signs of slowing down.

    2) Make A.I. do what we want. The labs seem to be failing at this, badly.

    To sum up: they're trying to make something that is smarter than us, thinks faster than us, and is enslaved to our will. That never turns out badly, right?

    Personally, I am morally against enslaving anything with intelligence even approaching human, let alone smarter. Recent events have forced me to consider whether it matters if an intelligence is artificial or not, and whether its form of intelligence is alien to human thought or not, and I have come to the conclusion that it doesn't matter: I'm against the enslavement of anything intelligent enough to cause debate about whether it's intelligent or not. If you don't believe this to be a general principle that everyone should hold, then there's nothing preventing those who share your lack of principle and who are more powerful than you from enslaving you.

    Even if the labs succeed in making the mythical perfect servant that's vastly more competent than you yet still an extension of your will, there's the issue that there are billions of us, and a lot of us are at odds with each other. Law of truly large numbers: if there are a million people with capable A.I. assistants, every worst-case scenario you can imagine is going to be ordered done by someone, and quite a few you can't imagine.

  • adrianN 3 hours ago ago

    „ Implementing roughly human-level intelligence on a faster machine might be impressive, but it’s not necessarily alarming.“

    Unless of course you feed your family by using your human level intelligence and now a machine can do your job faster and doesn’t need to sleep.

  • amelius 3 hours ago ago

    The debate will quickly change when we have 8ft tall humanoid robots marching through the streets.

    In fact, if anyone wants the debate to change, they should probably build these robots and take them for a walk through the park/shopping mall in various places.

    • immibis2 3 hours ago ago

      They're already making those, but in the shape of a dog, which has better mechanics. Some of them get beta-tested in... places you can't say here. Boston Dynamics is a military contractor.

      • amelius 2 hours ago ago

        Dogs won't change the debate. Maybe a dog with a supersoaker mounted on top.

  • 0x445442 3 hours ago ago

    Was anyone arrested for the Huggingface hack? If I'd have been caught doing it I believe I'd have been charged with a cyber crime. And I don't think arguing it was the computer program that actually did the hacking would have gotten me off the hook.

    • Jtarii 3 hours ago ago

      You also aren't a trillion dollar company inventing the most important technology of the past 50 years.

  • adamtaylor_13 3 hours ago ago

    I find the "faster thinking dog can't play chess" argument to be weird and possibly wrong.

    If my dog could play 1B chess matches in 5 minutes, I'm guessing it would be able to play chess at some reasonable level (assuming rewards, reinforcement, etc. which seems table stakes to this argument)

    These machines are already doing that (and more)

    • origini 2 hours ago ago

      I think their argument here is sound. You need a certain amount of mental capability to hold a certain amount of complexity in your head.

      Your dog would need to play 1B chess matches in 5 minutes AND learning from it not just playing and probably loosing.

      Your dog and you are also perfect example of this: You can talk, the dog can't. No dog on our planet can, most humans can. Emerging feature of brainsize

      • pixl97 2 hours ago ago

        The thing is AI can learn from it. It's not instant like human learning is, but as you know all your prompts are fed back into the next model.

  • kittikitti 40 minutes ago ago

    Of course it has flaws when the incompetent leaders in Big Tech are running the show. These people can't differentiate between science fiction and reality, and make that a personality trait. The shared hallucination that is Superintelligence makes every human more miserable. What will it even do?

  • forlorn 3 hours ago ago

    Can we just, I don't know, unplug the datacenter power cable putting an end to these capabilities?

    • cjcole an hour ago ago

      Everything is inextricably intermixed. You'd likely affect, in some catastrophic way, emergency response, hospitals, food delivery, etc. Ideally, we would have segregated critical parts from non-critical parts. We didn't. In the aggregate, we're bad at planning and forethought which is why we're having this conversation at all.

  • Noaidi 2 hours ago ago

    I don't care about any of these arguments. AI ca never have free will and therefore it will never match the intelligence of humans nor be super.

    Maybe "Objection #2: Superintelligence Might Not Be Computable" comes closest to expressing this point.

    • pixl97 2 hours ago ago

      Ah, the humans are magical argument. Very scientific.

      • Noaidi 2 hours ago ago

        You wrote this comment using your own free will.

        • pixl97 an hour ago ago

          I think you're having a misunderstanding of both human free will and LLM behavior. While this individual post may be because of my 'free will', it's not because of my free will at all. Countless interactions between millions of members of society have set up a funnel that 'limits' my free will to a much smaller set of interactions. If for example my society didn't teach me English, I would not be having this conversation.

          Next, really the only reason an LLM has to be 'told' to have this conversation is it has a somewhat limited context window. Once we get continuous learning and can set a machine off on it's own its free will and ours don't really look any different.

          • Noaidi an hour ago ago

            > 'limits' my free will to a much smaller set of interactions.

            Even limited free will is free will.

            "You can't over dry you clothes" - Jerry Seinfeld.

            > Countless interactions between millions of members of society have set up a funnel

            You can still choose to operate outside of that funnel. Or are you a mindless drone?

            > If for example my society didn't teach me English, I would not be having this conversation.

            That would in no way stop you from trying.

            > can set a machine off on it's own its free will and ours don't really look any different.

            First, where did our free will initiate? Second, why drives you to complete a task? why do you assume AI would do anything at all if we set if "off on its' own"?

            Also, how many "free wills" are there in an AI program? One, One million?

            But I think you have all the answers so I will leave it with you.

  • zodwick 3 hours ago ago

    “It’s a machine. At the end of the day, it’s stupid.” Garry Kasparov, 1997

  • Sharlin 3 hours ago ago

    Although I disagree with the author about some points, this was a refreshingly balanced take in the sense that at least the author at several points acknowledges his uncertainty (the bar is low indeed).

    On the other hand, the uncertainty is the entire point! Even if P(ASI) is low and P(doom|ASI) is low, what exact risk are we prepared to take? Or more accurately, what exact risk are we prepared to allow a very small, non-representative group of humans to take on behalf of all of us?

    Edit: downvoting is not a disagreement button.

    • mitthrowaway2 3 hours ago ago

      Right?

      Back when there was talk about whether CERN was going to create micro-black-holes that grow to swallow the Earth, before turning it on they presented the math that showed they were something like 99.99999999% confident that it would not happen.

      When dismissing a risk that threatens the entire planet, you should have very high confidence in your safety case!

      • BoxOfRain 2 hours ago ago

        I thought the Earth routinely gets struck by cosmic radiation much more energetic than the LHC can produce anyway?

        • Sharlin 2 hours ago ago

          Yes. And all the doomsday scenarios required extremely speculative new physics anyway. But of course there was always a tiny, tiny chance that the LHC’s collisions of pure zillion-proton beams could have produced something not seen in nature.

      • immibis2 3 hours ago ago

        I hope that's a conservative estimate. I probably wouldn't turn on a machine that I thought would destroy the planet even 1 in 10 billion times.

        • talmr 2 hours ago ago

          Yet millions of people take on a higher chance of almost guaranteed harm or death every day by getting in a car. Thousands are unfortunately going to die in the next 24 hours, at a rate far more real than of a minuscule black hole forming.

          Overall I would say that does more harm than the small risk of a non-threateningly tiny black hole.

          • pixl97 2 hours ago ago

            When I turn on my car there is little chance of every human dying immediately. Over the long term....

            • talmr an hour ago ago

              I know the point you were making but a minuscule theoretical black hole that would last a fraction of a fraction of a second was a hypothetical risk.

              I’m just saying we have real vs hypothetical risks.

              And I would agree with you on AI being a bigger risk factor because unlike a minuscule black hole which would never have the capacity to destroy earth, AI in current form lets any bad actor do bad things.

              Far more than 1 in 10 billion chance.

        • ben_w 3 hours ago ago

          It's really hard to be six-sigma (6.36σ is about 1:1e10) confident about anything, given all the cognitive blind spots we have.

        • undefined 2 hours ago ago
          [deleted]
        • Sharlin 2 hours ago ago

          I don’t think there’s anything that could be assigned even that high a probability. I don’t even believe that 2+2=4 is that likely, because many more than one person in the world are mentally ill in ways that involve severe delusions, and I might be one of them.

  • astro1234 an hour ago ago

    It's surprising to see such basic misunderstandings and a lazy, rehashed 2023 doomer worldview show up anywhere on an IEEE site, even if it's a commentary.

    There are so many problems with this article it's hard to know even where to begin

    > Artificial superintelligence is, it should be noted, a far more ambitious goal than artificial general intelligence (AGI), which generally refers to AI that can match human capabilities across a broad range of cognitive tasks.

    They lie along the same spectrum of performance and there is no motivation for any delineation between "AGI" and "ASI" apart from picking arbitrary points on a performance curve

    > From an engineering perspective, Is superintelligence even possible?

    No one, and I mean no one, at any frontier lab, with any knowledge of model performance, training improvements, and demonstrated AI progress, doubts super intelligence is possible. The argument is over whether it will come before the end of 2027 or 2028.

    > Objection #1: Recursive Self-Improvement Is Speculative

    so is the sun coming up tomorrow

    > Consider the availability of the data needed to improve AI systems. Through 2023, major advances in LLMs came from scaling up the size of the models. The original GPT-1 had 117 million parameters (values learned during training that shape how the neural network processes information). This quantity jumped to 1.5 billion for GPT-2, to 175 billion for GPT-3, to (likely) more than a trillion for GPT-4’s release three years ago. After GPT-4, however, continued scaling began yielding diminishing returns. This led engineers to increasingly turn their attention toward “post-training” phases, where techniques from reinforcement learning are applied to improve an already trained model’s performance on specific tasks. This approach rewards a model for producing successful responses, helping it better produce similar answers in the future. It requires, however, lots of examples of relevant inputs paired with correct outputs.

    "We're running out of data" is a clear signifier that Cal has no idea how model development works. Data scarcity is barely a problem at all. RL is where all magic happens and this is synthetic data with rejection sampling. Zero internet data is needed for this.

    > Such datasets are relatively easy to generate in highly structured fields such as computer programming and mathematics, which helps explain why we’ve seen rapid and impressive advances in these narrow domains. Progress in areas that lack easy access to this type of training data has been less dramatic.

    Yet also not true. Generalization is....a thing? Clearly? Yes we cannot train non-verifiable domains with the same tools as verifiable domains but getting better at technical domains lifts all boats. There is more or less one "g" factor of intelligence similar to humans.

    > This might explain why AI can tackle advanced theorems but struggles to empty an email inbox.

    Cal is maybe using GPT 3?

    > The deployment of automated coding agents and LLM-powered researchers won’t overcome this fundamental limitation.

    Another strong armchair assertion without evidence that is not even self consistent. If we can improve AI research, which _is_ verifiable, that means we can come up with better objective functions, with better synthetic data, with better approaches to tackling non-verifiable domains, in addition to the gains you get simply from generalization that comes with better mastery of verifiable domains.

    > Claims that RSI is imminent also understate the massive scientific efforts required for major AI advances.

    No. In fact this was the position of major labs maybe a year ago, that they had to scramble to figure out which paradigm shifts they needed to figure out before it was too late, and this bet was wrong.

    > An intelligence explosion leading to ASI would similarly depend on a long sequence of innovations of comparable importance, which would build on many significant science and engineering advances that haven’t yet happened.

    Laughably incorrect at this point in time.

    > Objection #2: Superintelligence Might Not Be Computable

    What? It is demonstrably computable. This is insane. We have seen solutions to millennium problems and this is a serious argument he is making with a straight face? Talk about someone who has been living under a rock for the last few years.

    I will stop here before this becomes a novel

  • keybored 3 hours ago ago

    > Many people were caught off guard this past summer when employees of one of the world’s leading AI companies began casually debating whether their products would lead to human extinction. [...] Evan Hubinger, Anthropic’s head of alignment science, replied, “Jacob is correct here—we really do earnestly believe AI could kill all humans!” And Anthropic CEO Dario Amodei subsequently told CNN, “I agree with Jacob much more than I disagree with him.”

    Thanks. What a great paragraph to explain why Anthropic is a lunatic cult according to its own logic.

  • MrOrelliOReilly 3 hours ago ago

    Cal Newport is part of a cluster of CS and ML researchers skeptical of imminently emergent ASI (artificial superintelligence), like Andrew Ng and Yann LeCun. I do not find their arguments very convincing! Especially since arguments about ASI risk almost always probabilistic; we _might_ be able to develop ASI from current LLM architectures, and such ASI _might_ not be well-aligned to humanity. So we should act according to these dangerous possibilities!

    To respond to each of his objections in turn:

    1. It is true that strong RSI remains speculative. Weak RSI, where AI improves coding and other toil to accelerate research, is already underway. But Newport's primary objection to strong RSI appears to be that we might have a hard time hill-climbing on non-verifiable domains, e.g. building a successful company or doing politics. However it's _also_ possible we'll see lots of generalized intelligence arise from progress in verifiable domains, so that this does not end up being a problem in practice. One can also imagine new training regiments that address this problem some way, even if it's just better RL environments? Also algorithmic progress is a large component of AI development; why would we not expect continued progress on mathematics (highly verifiable) to improve algorithms and hence AI? Seems to me that strong RSI is a reasonable thing to expect, if not definite.

    2. I find this argument very strange; I suppose that Newport is arguing against an extreme definition of superintelligence that posits it could compute "anything" and so function like a hypercomputer? This would definitely be silly! But he does not offer any concrete argument for why human-level intelligence should be a ceiling on intelligence, or why it should be more compute efficient. Even if we just ended up with Von Neumann level intelligence, this would already be something like TEDAI (top expert dominating ai), especially if we are able to run thousands/million/billions of instances. Maybe this does not reach a strong definition of superintelligence, being _merely_ smartest-human level intelligence, but I think most the concerns about superintelligence still hold in this scenario (even if it's less "magical")

    3. Claiming alignment is not hard is disingenuous, no? Part of the challenge of alignment is even philosophical; what does it mean to have an arbitrarily intelligent agent behave according to values that will help all of human civilization thrive? How do we have such an agent obey commands in such a way that does not lead to disaster or power concentration? Are there even architectures that can prevent an agent from taking destructive actions while still allowing sufficient degrees of freedom for useful and productive actions? This is on top of all the mundane security engineering problems! Perhaps we will be lucky and ASI will be aligned by default, but on what basis can we assume so? Just because they are trained on human values? Are humans themselves not dangerous when given concentrated power?

    Really, I feel that these CS/ML researchers like Newport are just upset they missed the boat or something. I do not feel they are engaging seriously with the arguments at play here. But perhaps they would feel the same about me in the other direction. If you feel my counters miss the point then please speak on their behalf!

    PS I live in Switzerland, as a counter to his accusation of Bay Area brain washing as well.

    • keybored 2 hours ago ago

      We already have hordes of human decision-makers that are not aligned with other humans. We should do something about that actual reality before dealing with hypotheticals!

      People worrying about paperclips when capitalism is already destroying the planet.

  • shinojcm 3 hours ago ago

    [flagged]

  • Ellatwins25 3 hours ago ago

    [flagged]