It's not really "Flash" if it is 7 tokens per second.
Also don't bother with any GPU lower than an RTX 5080 or even a DGX Spark (it is a scam) if you are seriously considering running frontier local models.
Oh, I don't know - it could be useful for long-running and/or background tasks in some cases.
The problem is that this particular model is so cheap, compared to others. Optimizing to host it locally with relatively poor performance just doesn't make sense when yesterday I burned ~180M tokens for $2.20.
If you were wondering:
> Up to 7.4 tokens/s.
It's not really "Flash" if it is 7 tokens per second.
Also don't bother with any GPU lower than an RTX 5080 or even a DGX Spark (it is a scam) if you are seriously considering running frontier local models.
It's apparently running at ~11 tok/s on a 4090: https://github.com/Magnvs7/Promethevs/issues/7
Strata runs considerably faster than that on similar hardware, though this one doesn't seem to be a fork.
Oh, I don't know - it could be useful for long-running and/or background tasks in some cases.
The problem is that this particular model is so cheap, compared to others. Optimizing to host it locally with relatively poor performance just doesn't make sense when yesterday I burned ~180M tokens for $2.20.
Strata for running from HDD seems really popular, to get big models running.
Is that faster?