Wild to think we could have implemented transformers in 1990 with consumer hardware. I guess we needed the web first for training, but small transformers are surprisingly effective. What other discoveries are right under our nose?
As someone else pointed out, training something like this requires much more hardware than running it.
It's similar with other forms of machine learning, too. A logistic regression can make calibrated decisions in complex situations with nearly no compute at all, but training it on lots of data to give it good weights is more expensive. And even if we could submit a training batch job on 1990s hardware and wait a few years for it to complete, there's a level of interactivity required at the start to explore one's way to the right parameters and tokenisation etc. that probably wouldn't have been economically feasible at the time.
In a wider sense, a general purpose computer is a general purpose computer. Hypothetically, we "could have" run transformers in the 1940s on vacuum tubes. It just would have (a) required much space, (b) produced much heat, and (c) required tube replacements all the time. We "could have" run transformers on human computers in the late 1800s or early 1900s – all the maths was there, I think? – but it would have been even slower!
The difference between 1992 hardware and 1998 was really huge. High end Sparc servers on 1998 (or double CPU based Xeons) could create something usable.
I'm not sure we would have found this impressive back then, the output is pretty nonsensical. Back then, we had other (more manual) approaches that could produce somewhat equivalent results.
It's a bit of an illusion, it looks quite impressive to us because we know how good the output from a full-sized LLM can be. We are willing to give it a lot of leeway because of its small size as long as it produces output that looks kind of like chatgpt.
Well... computational linguistics academics should have been impressed at how well it understands language with nothing more than ingesting a huge amount of untagged data.
The ancient Romans could have been the ones to discover electricity and steam engines but they didn't. Room temperature superconductors could have already been found. The right person in the right place is what's needed.
The music is great and the way it types out stuff for you is absolutely aesthetic. I'd be interested to know more about how it uses the processors to figure out what part of the network to process. I asked ChatGPT to take some guesses and all it did was find me an AI enabled CRT television on Etsy. I think it's starting to reproduce little children.
That's a great project, thanks for sharing.
Wild to think we could have implemented transformers in 1990 with consumer hardware. I guess we needed the web first for training, but small transformers are surprisingly effective. What other discoveries are right under our nose?
As someone else pointed out, training something like this requires much more hardware than running it.
It's similar with other forms of machine learning, too. A logistic regression can make calibrated decisions in complex situations with nearly no compute at all, but training it on lots of data to give it good weights is more expensive. And even if we could submit a training batch job on 1990s hardware and wait a few years for it to complete, there's a level of interactivity required at the start to explore one's way to the right parameters and tokenisation etc. that probably wouldn't have been economically feasible at the time.
In a wider sense, a general purpose computer is a general purpose computer. Hypothetically, we "could have" run transformers in the 1940s on vacuum tubes. It just would have (a) required much space, (b) produced much heat, and (c) required tube replacements all the time. We "could have" run transformers on human computers in the late 1800s or early 1900s – all the maths was there, I think? – but it would have been even slower!
The difference between 1992 hardware and 1998 was really huge. High end Sparc servers on 1998 (or double CPU based Xeons) could create something usable.
I'm not sure we would have found this impressive back then, the output is pretty nonsensical. Back then, we had other (more manual) approaches that could produce somewhat equivalent results.
It's a bit of an illusion, it looks quite impressive to us because we know how good the output from a full-sized LLM can be. We are willing to give it a lot of leeway because of its small size as long as it produces output that looks kind of like chatgpt.
Well... computational linguistics academics should have been impressed at how well it understands language with nothing more than ingesting a huge amount of untagged data.
The ancient Romans could have been the ones to discover electricity and steam engines but they didn't. Room temperature superconductors could have already been found. The right person in the right place is what's needed.
Running them isn't the same as creating them.
Sure but it could still have been done, we just didn't realize it.
By that logic we could have invented it at any point in history with a turing machine made of rocks.
And now you're re-reading the Three-body Problem.
Plenty of research papers out there that get no love.
Go through them and find the next big thing :)
this is so cute
you beat me to it
At this point, it's so small it's just a glorified Markov chain? It produces meaningless English-shaped sentences.
"How many 'r's are there in the word strawberry?"
> 1. The opposite of strawberries
AGI achieved
I asked it the answer to the question of life, the universe, and everything. It gave me a nonsensical response. I'd say it works perfectly.
Post-train it on law texts and make it fight against Nintendo's lawyers.
"1 plus 1?"
> The answer is 9.
It tried.
Nah, it just used a different meaning for the symbols involved.
The music is great and the way it types out stuff for you is absolutely aesthetic. I'd be interested to know more about how it uses the processors to figure out what part of the network to process. I asked ChatGPT to take some guesses and all it did was find me an AI enabled CRT television on Etsy. I think it's starting to reproduce little children.
I appreciate that out of a vocabulary of 8178 words, "San" and "Francisco" made the cut.
Also, it has a 128 token context window. Brutal!
Hm, I asked it questions about Metroid and the answers were gibberish. "A couch is a type of fruit" etc.
Eh?