my hypothesis is they are using flash and quant'd models to keep up with demand in the hardware crunch
You see the same pattern across open weight models and homelab setups, as we try to fit the models on devices while preserving capability and getting reasonable throughput
my hypothesis is they are using flash and quant'd models to keep up with demand in the hardware crunch
You see the same pattern across open weight models and homelab setups, as we try to fit the models on devices while preserving capability and getting reasonable throughput
Trying to preserve something one cannot reliably measure. I wish you luck!
> cannot reliably measure
there are lots of things you can measure, designing tasks for evals is an art (which I'm trying to learn)
Sure but I said reliably measure.