Those are amazing accomplishments but I am more interested in research developments in things like In-Memory (Analog) or other different approaches.
Companies like EnCharge, Mythic, etc.
And much more efficient devices like RRAM, MRAM, and FETs. Like FE-FETs with AlScN.
The stuff just coming out of research or still in research is more exciting in terms of the potential for truly huge efficiency and performance boosts.
Taalas is interesting also because of it's efficiency and speed. Guess it was just purchased by AMD.
I'm not sure that more efficient compute can win out, since it also has to beat out the competition on capital investment. There's an upper bound of 'just get 3 sets of GPUs and run them on solar in three different timezones' where the cost of electricity is negligible. Efficient digital architectures run about 1/5 or 1/10th the speed and require much more wafer area. Analog technologies seem unlikely to have the fidelity required for training, though inference sounds plausible. It's also not clear that analog technologies would necessarily use less power.
Transistors are most power efficient when they are either completely open (no resistance) or completely closed (no current).
If they are in some analog half-open state they have both resistance and current, which means waste heat.
Also transistors in modern chips are already so small that you can literally count the number of electrons flowing through them. At the smallest scale, nature becomes discrete. So if there are something like 128 electrons, you only get 7 bits of 'analog' to play with anyway.
Recently I started thinking about an analog optical device capable of matrix multiplication. After some online research, the Mach-Zehnder interferometer seems to have been adapted by a German company into a full addon optical based matrix multiplication system.
It is a very interesting concept and the company Q.Ant has delivered working products and software. Apparently it can greatly speed up certain training applications and future designs will solve some of the current problems.
The energy and performance advantages seem to be worth the R&D. If some form of optical memory to replace DRAM in the chain could be designed, that would solve several conversion costs between optical to digital.
Either way it is pretty cool to do math this way, basically just using light.
The assumption that optical or analog will be more efficient for a given accuracy though is not grounded. Any non-linear optical phenomenon requires multiple photons interacting in matter, or interacting sequentially with the same matter, and this requires high intensities of (almost always coherent) light in order to make the interaction happen frequently enough.
e.g for a Mach-Zehnder interferometer measuring the phase shifts requires many photons. A single photon arrives or doesn't arrive at the detector. Even if you could perfectly ensure it hits detector at the same time as the reference photon and subsequently detect with 100% quantum efficiency, that still only gives you 1 bit of phase information, not an analog value.
Digital can be done more efficiently with adiabatic or asynchronous circuits. The limiting factor then is speed and chip area rather than joules-per-and-gate.
Some technologies will definitely be much faster and much more power efficient. They haven't come out of research or if they have they have not been scaled yet, so it's not 100% proven. But it looks likely for multiple options to get there.
Agreed. One of the least efficient things to do with digital logic is floating point multiplication. The second least efficient is probably integer multiply (I'm just kidding a bit here).
Multiplication may make sense for the math, but it's just not practical in hardware. There has to be a better, much simpler operation, that can give you equivalent results. Yes, you may have to re-think and re-train everything, but eventually it should make current architectures obsolete.
Is anyone working on running neural networks on CPU architectures that are not limited by synchronous clock signals? Organic neural networks are inherently asynchronous and signals propagate through different parts of the network at their own pace.
Yes, for over a decade now and it's already available in the cloud. It's a spinn-off of human brain project. Search for spinnaker neuromorphic computing.
I had a quick look around and it seems like this is very specifically designed and used for the Human Brain Project, which ended in 2023 and at least according to wiki the most recent version of this was from 2019. Is it still being worked on, and is it actively being developed for anything outside of brain simulation?
Microsofts Bitnet is really solid on CPUs (though still better to train on a GPU for speedier training) but theres a lot of room for improvement in these spaces.
I did not spot any errors in my skim of the architectures I'm familiar with where I would expect an LLM to go wrong, and it has a nicely scoped overview of topics that people in the space should familiarize themselves with.
I don't think I've seen a better primer.
Maybe it's just me, but between the extremely thin font and layout design, I find this extremely difficult to parse. Overuse and improper use of italics is confusing as well.
Yea stresses the brain to much. I have built my own reader app, one of the simple things it does is very quickly convert article to highly readable font, with correct spacing on dark mode. Most things I read now are through that all, plus it saves it locally and remembers where I left off so useful for long form reads when I am on tube.
I have a feeling the hardware architecture for LLMs are completely wrong. There's no way hundreds of kilowatts is required for intelligence.. just in terms of the physics. Is there someone out there in the analog/neuromorphic computing world that could make these power-hungry monsters completely redundant?
It's more the economics of what sells that lets the power hungry monsters thrive. If you can tape out one chip design, and one gives you 1 TOPS and the other is 5 TOPS, the sales will go to the 5 TOPS because power is a downstream cost. That being said, it's only a matter of time before both chips are maximum TOPS and the only differentiator is power. My latest simulations are showing a competitive design around 70W, but I'm still confirming it and it would really be beneficial to do some data gathering tapeouts at larger nodes to feel like it's worth the risk of more expensive nodes.
Yes smaller nodes give lower power operation with faster transistor switching speeds. You can also underpower a node to intentionally cause slower transistor switching speeds.
You're completely wrong about hardware being the bottleneck.
The software architecture LLMs run on was chosen precisely because hardware can brute force it and brute force has become the method of choice ever since, because it is easier to scale.
And here you're saying "what if everyone was wrong and what we need is just more brute force?", without looking at the reason that makes brute force necessary to begin with.
This is also the problem, we are continuing with LLMs because they fit hardware well, instead of experimenting with trying to model knowledge (how do I know what I know) and metacognition (what do I know).
Current architecture can't tell the difference between fact and fiction, so to speak. (Which I think is a better way to say it rather than truth and falsehood; facts have provenance and context, not just "true" or "false".)
I just used it on 4 texts. The 1st version was the output from my first (chatGPT) prompt. The 4th was the last prompts of "reduce, make it less AI, etc" of the same original request/output.
In all cases it was meatbag. In the 1st case gave chatGPT 0.35. The remaining ones were much lower.
EDIT: I didn't ask just to generate some text. The original prompt was very specific, which is most of the time the way I use LLMs anyways. Probably it catches better the "hey just generate this email...".
I'm surprised by this comment. It's probably mostly AI generated, but I have no problem reading the first 10 pages, and I found them clear and correct.
Might be that we come at this page with different experience and knowledge. I'm not seeing any specifically hard to read AI here.
It’s shorthand for “it’s unreadable, but in the way an AI would write it”. I’m not OP, but that’s the issue. If it was well written (even if AI written), there would be no problem.
As it turns out, you still need to be a good writer in order to prompt an AI to write well.
def. AI-assisted, both content and style style/nav.
Content-wise, it looks legit, but with that amount of stuff, it's nearly impossible to say for sure.
It's an interesting topic, but the article is completely AI generated, which isn't surprising considering the startup, but still, if you don't even go over the output what's the point?
AI chips feel like a gimmick. An excuse to add "powered by ai chips" to the marketing material. That sort of phrase in on just about every consumer laptop now.
My brother bought one. I challenged him to find something that actually used the AI chip on the laptop. He couldn't - it always read at 0% in task manager.
As a consumer, I'd rather have that much more graphics card, hard drive space, or RAM for the price than a specialized chip thats almost never used and, when it is used, only saves you a little electricity.
Those are amazing accomplishments but I am more interested in research developments in things like In-Memory (Analog) or other different approaches.
Companies like EnCharge, Mythic, etc.
And much more efficient devices like RRAM, MRAM, and FETs. Like FE-FETs with AlScN.
The stuff just coming out of research or still in research is more exciting in terms of the potential for truly huge efficiency and performance boosts.
Taalas is interesting also because of it's efficiency and speed. Guess it was just purchased by AMD.
I'm not sure that more efficient compute can win out, since it also has to beat out the competition on capital investment. There's an upper bound of 'just get 3 sets of GPUs and run them on solar in three different timezones' where the cost of electricity is negligible. Efficient digital architectures run about 1/5 or 1/10th the speed and require much more wafer area. Analog technologies seem unlikely to have the fidelity required for training, though inference sounds plausible. It's also not clear that analog technologies would necessarily use less power.
Transistors are most power efficient when they are either completely open (no resistance) or completely closed (no current).
If they are in some analog half-open state they have both resistance and current, which means waste heat.
Also transistors in modern chips are already so small that you can literally count the number of electrons flowing through them. At the smallest scale, nature becomes discrete. So if there are something like 128 electrons, you only get 7 bits of 'analog' to play with anyway.
Recently I started thinking about an analog optical device capable of matrix multiplication. After some online research, the Mach-Zehnder interferometer seems to have been adapted by a German company into a full addon optical based matrix multiplication system.
It is a very interesting concept and the company Q.Ant has delivered working products and software. Apparently it can greatly speed up certain training applications and future designs will solve some of the current problems.
The energy and performance advantages seem to be worth the R&D. If some form of optical memory to replace DRAM in the chain could be designed, that would solve several conversion costs between optical to digital.
Either way it is pretty cool to do math this way, basically just using light.
The assumption that optical or analog will be more efficient for a given accuracy though is not grounded. Any non-linear optical phenomenon requires multiple photons interacting in matter, or interacting sequentially with the same matter, and this requires high intensities of (almost always coherent) light in order to make the interaction happen frequently enough.
e.g for a Mach-Zehnder interferometer measuring the phase shifts requires many photons. A single photon arrives or doesn't arrive at the detector. Even if you could perfectly ensure it hits detector at the same time as the reference photon and subsequently detect with 100% quantum efficiency, that still only gives you 1 bit of phase information, not an analog value.
Digital can be done more efficiently with adiabatic or asynchronous circuits. The limiting factor then is speed and chip area rather than joules-per-and-gate.
Some technologies will definitely be much faster and much more power efficient. They haven't come out of research or if they have they have not been scaled yet, so it's not 100% proven. But it looks likely for multiple options to get there.
> Mythic
Recently on HN: https://news.ycombinator.com/item?id=49352470
Agreed. One of the least efficient things to do with digital logic is floating point multiplication. The second least efficient is probably integer multiply (I'm just kidding a bit here).
Multiplication may make sense for the math, but it's just not practical in hardware. There has to be a better, much simpler operation, that can give you equivalent results. Yes, you may have to re-think and re-train everything, but eventually it should make current architectures obsolete.
memrystors for chache inside the pipeline
Is anyone working on running neural networks on CPU architectures that are not limited by synchronous clock signals? Organic neural networks are inherently asynchronous and signals propagate through different parts of the network at their own pace.
Yes, for over a decade now and it's already available in the cloud. It's a spinn-off of human brain project. Search for spinnaker neuromorphic computing.
I had a quick look around and it seems like this is very specifically designed and used for the Human Brain Project, which ended in 2023 and at least according to wiki the most recent version of this was from 2019. Is it still being worked on, and is it actively being developed for anything outside of brain simulation?
https://en.wikipedia.org/wiki/SpiNNaker
Microsofts Bitnet is really solid on CPUs (though still better to train on a GPU for speedier training) but theres a lot of room for improvement in these spaces.
I did not spot any errors in my skim of the architectures I'm familiar with where I would expect an LLM to go wrong, and it has a nicely scoped overview of topics that people in the space should familiarize themselves with. I don't think I've seen a better primer.
Maybe it's just me, but between the extremely thin font and layout design, I find this extremely difficult to parse. Overuse and improper use of italics is confusing as well.
Yea stresses the brain to much. I have built my own reader app, one of the simple things it does is very quickly convert article to highly readable font, with correct spacing on dark mode. Most things I read now are through that all, plus it saves it locally and remembers where I left off so useful for long form reads when I am on tube.
I didn't believe it till I tried to read it. Insane. Impossible to read.
I have a feeling the hardware architecture for LLMs are completely wrong. There's no way hundreds of kilowatts is required for intelligence.. just in terms of the physics. Is there someone out there in the analog/neuromorphic computing world that could make these power-hungry monsters completely redundant?
It's more the economics of what sells that lets the power hungry monsters thrive. If you can tape out one chip design, and one gives you 1 TOPS and the other is 5 TOPS, the sales will go to the 5 TOPS because power is a downstream cost. That being said, it's only a matter of time before both chips are maximum TOPS and the only differentiator is power. My latest simulations are showing a competitive design around 70W, but I'm still confirming it and it would really be beneficial to do some data gathering tapeouts at larger nodes to feel like it's worth the risk of more expensive nodes.
Systolic hardware?
Race logic with pulses, it's a descent of systolic arrays. There is less of an array shape and more about time.
Dumb question, wouldn't node size impact the logic latency itself on some level?
Yes smaller nodes give lower power operation with faster transistor switching speeds. You can also underpower a node to intentionally cause slower transistor switching speeds.
Where can I find the whitepaper?
[dead]
You're completely wrong about hardware being the bottleneck.
The software architecture LLMs run on was chosen precisely because hardware can brute force it and brute force has become the method of choice ever since, because it is easier to scale.
And here you're saying "what if everyone was wrong and what we need is just more brute force?", without looking at the reason that makes brute force necessary to begin with.
This is also the problem, we are continuing with LLMs because they fit hardware well, instead of experimenting with trying to model knowledge (how do I know what I know) and metacognition (what do I know).
Current architecture can't tell the difference between fact and fiction, so to speak. (Which I think is a better way to say it rather than truth and falsehood; facts have provenance and context, not just "true" or "false".)
human brain uses 20 watts.
I have a gut feeling the particular stochastic processes we’re currently using are the wrong ones to go beyond LLMs into actual intelligence.
This text is most likely AI-generated. Borderline unreadable.
My meatbag or robot scored it as likely a robot and most likely Claude.
https://meatbag.atomic14.com/
I just used it on 4 texts. The 1st version was the output from my first (chatGPT) prompt. The 4th was the last prompts of "reduce, make it less AI, etc" of the same original request/output.
In all cases it was meatbag. In the 1st case gave chatGPT 0.35. The remaining ones were much lower.
EDIT: I didn't ask just to generate some text. The original prompt was very specific, which is most of the time the way I use LLMs anyways. Probably it catches better the "hey just generate this email...".
There's no borderline about it.
When the density of italicized words is about 1:1, I'm out.
I'm surprised by this comment. It's probably mostly AI generated, but I have no problem reading the first 10 pages, and I found them clear and correct.
Might be that we come at this page with different experience and knowledge. I'm not seeing any specifically hard to read AI here.
Most like AI generated? If you can't tell then what's the problem?
It’s shorthand for “it’s unreadable, but in the way an AI would write it”. I’m not OP, but that’s the issue. If it was well written (even if AI written), there would be no problem.
As it turns out, you still need to be a good writer in order to prompt an AI to write well.
Wise people tend not to talk in absolutes, because they always realize they might be wrong.
Ignorant people are always certain.
This is just a rephrasing of the Dunning-Krueger effect, but quantized to a binary domain (ignorant vs wise).
Strong opinions…
always?
I am one of the ignorant ones, after all, so I'm probably wrong on this. Wait...
def. AI-assisted, both content and style style/nav. Content-wise, it looks legit, but with that amount of stuff, it's nearly impossible to say for sure.
Not slope tho!
Not too shabby, but it shouldn't have been like 3 posts
I personally found this article highly educational...
One of the best deep-dives on AI chip architectures that I've ever read.
Other posters are welcome to their opinions, but I (again, personally) thought it was great!
Can memristors help AI?
If they ever figure out how to make it in any usable scale, maybe.
It's an interesting topic, but the article is completely AI generated, which isn't surprising considering the startup, but still, if you don't even go over the output what's the point?
How do you know they didn't go over it?
[dead]
AI chips feel like a gimmick. An excuse to add "powered by ai chips" to the marketing material. That sort of phrase in on just about every consumer laptop now.
My brother bought one. I challenged him to find something that actually used the AI chip on the laptop. He couldn't - it always read at 0% in task manager.
As a consumer, I'd rather have that much more graphics card, hard drive space, or RAM for the price than a specialized chip thats almost never used and, when it is used, only saves you a little electricity.
FWIW the article is not talking about those kind of AI chips at all