hi hn. I don't have a powerful GPU, but I have a big desire to get into local neural nets, and instead of buying normal hardware I decided to vibecode my own schizo architecture.
The main idea is something like a hypernetwork. Like, we don't keep all layers in memory, but create a layer, apply it, throw it away, and create a second one. The idea was that it should probably be smarter in logic (like, the layers are purposefully created for the prompt) and, most importantly, require much less memory without a critical loss in speed.
In the process, the idea was supplemented, for example, by using a base layer (qwen 2.5 0.5b), generating per-head dynamic matrices (w_q, w_k, etc.), moving general knowledge into tool calls (wiki search and calculator), a stripped-down vocabulary, etc.
The result turned out surprisingly not bad. In 8 hours on an rx6600 at 36m params, it seems it already maintains a minimal dialogue, freely uses tools, etc.
hi hn. I don't have a powerful GPU, but I have a big desire to get into local neural nets, and instead of buying normal hardware I decided to vibecode my own schizo architecture.
The main idea is something like a hypernetwork. Like, we don't keep all layers in memory, but create a layer, apply it, throw it away, and create a second one. The idea was that it should probably be smarter in logic (like, the layers are purposefully created for the prompt) and, most importantly, require much less memory without a critical loss in speed.
In the process, the idea was supplemented, for example, by using a base layer (qwen 2.5 0.5b), generating per-head dynamic matrices (w_q, w_k, etc.), moving general knowledge into tool calls (wiki search and calculator), a stripped-down vocabulary, etc.
The result turned out surprisingly not bad. In 8 hours on an rx6600 at 36m params, it seems it already maintains a minimal dialogue, freely uses tools, etc.
Repo: https://github.com/RUD9/Grimoire.
It would be nice to get feedback from knowledgeable people. Open to any harsh critique, etc.