Yes, it's incredible boring to wait for the AI Agents in IDEs to finish their job. I get distracted and open YouTube. Once I gave a prompt so big and complex to Cline it spent 2 straight hours writing code.
But after these 2 hours I spent 16 more tweaking and fixing all the stuff that wasn't working. I now realize I should have done things incrementally even when I have a pretty good idea of the final picture.
I've been more and more only using the "thinking" models of o3 in ChatGPT, and Gemini / Claude in IDEs. They're slower, but usually get it right.
But at the same time I am open to the idea that speed can unlock new ways of using the tooling. It would still be awesome to basically just have a conversation with my IDE while I am manually testing the app. Or combine really fast models like this one with a "thinking background" one, that would runs for seconds/minutes but try to catch the bugs left behind.
So my personal belief is that diffusion models will enable higher degrees of accuracy. This is because unlike an auto-regressive model it can adjust a whole block of tokens when it encounters some kind of disjunction.
Think of the old example where an auto regressive model would output: "There are 2 possibilities.." before it really enumerated them. Often the model has trouble overcoming the bias and will hallucinate a response to fit the proceeding tokens.
Chain of thought and other approaches help overcome this and other issues by incentivizing validation, etc.
With diffusion however it is easier for the other generated answer to change that set of tokens to match the actual number of possibilities enumerated.
This is why I think you'll see diffusion models be able to do some more advanced problem solving with a smaller number of "thinking" tokens.
Unfortunately the intuition and the math proofs so far suggest that autoregressive training is learning the joint distribution of probabilistic streams of tokens much better than diffision models do or will ever do. My intuitive take is that the conditional probability distribtion of decoder-only autoregressive models is at just the right level of complexity for probabilistic models to learn accurately enough. Intuitively (and simplifying things at the risk of breaking rigor), the diffusion (or masked models) have to occasionally issue tokens with less information and thus higher variance than a pure autoregressive model would have to do, so the joint distribution, ie the probability of the whole sentence/answer will be lower and thus diffusion models will never get precise enough. Of course, during generation the sampling techniques influence the above simplified idea dramatically and the typical randomized sampling for next token prediction is suboptimal and could be beaten by a carefully designed block diffusion sampler in principle in some contexts though I havent seen real examples of it yet. But the key ideas of the above scribbles are still valid: autoregresive models will always be better (or at least equal) probabilistic models of sequential data than diffusion models will be. So the diffusion models mostly offer a tradeoff for performance vs quality. Sometimes there is a lot of room for that tradeoff in practice.
From the mathematical point of view the literature is about the distinction between a "filtering" distribution and a "smoothing" distribution. The smoothing distribution is strictly more powerful.
In theory intuitively the smoothing distribution has access to all the information that the filtering distribution has and some additional information therefore has a minimum lower than the filtering distribution.
In practice, because the smoothing input space is much bigger, keeping the same number of parameters we may not reach a better score because with diffusion we are tackling a much harder problem (the whole problem), whereas with autoregressive models we are taking a shortcut which happens to probably be one that humans are probably biased too (communication evolved so that it can be serialized to be exchanged orally).
I am still jetlagged and not sure what the most helpful reference would be. Maybe start from the block diffusion paper I recommended in a parallel thread and trace your way up/down from there. The logic leading to Eq 6 is a special case of such a math proof.
The human processing is still autoregressive, but using multiple parallel synchronized streams. There is no problem with such an approach and my best guess is that in the next year we will see many teams training models using such tricks for generating reasoning traces in parallel.
The main concern is taking a single probabilistic stream (eg a book) and comparing autoregressive modelling of it with a diffusive modelling of it.
Suggests an opportunity for hybrids, where the diffusion model might be responsible for large scale structure of response and the next token model for filling in details. Sort of like a multi scale model in dynamics simulations.
> it can adjust a whole block of tokens when it encounters some kind of disjunction.
This is true in principle for general diffusion models, but I don't think it's true for the noise model they use in Mercury (at least, going by a couple of academic papers authored by the Inception co-founders.) Their model generates noise by masking a token, and once it's masked, it stays masked. So the reverse-diffusion gets to decide on the contents of a masked token once, and after that it's fixed.
Thanks, yes, I was thinking specifically of "Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution". They actually consider two noise distributions: one with uniform sampling for each noised token position, and one with a terminal masking (the Q^{uniform} and Q^{absorb}.) However, the terminal-masking system is clearly superior in their benchmarks.
Once that auto-regressive model goes deep enough (or uses "reasoning"), it actually has modeled what possibilities exist by the time it's said "There are 2 possibilities.."
I think speed and convenience are essential. I use chat gpt desktop for coding. Not because it's the best but because it's fast and easy and doesn't interrupt my flow too much. I mostly stick to the 4o model. I only use the o3 model when I really have to. Because at that point getting an answer is slooooow. 4o is more than good enough most of the time.
And more importantly it's a simple option+shift+1 away. I simply type something like "fix that" and it has all the context it needs to do its thing. Because it connects to my IDE and sees my open editor and the highlighted line of code that is bothering me. If I don't like the answer, I might escalate to o3 sometimes. Other models might be better but they don't have the same UX. Claude desktop is pretty terrible, for example. I'm sure the model is great. But if I have to spoon feed it everything it's going to annoy me.
What I'd love is for smaller faster models to be used by default and for them to escalate to slower more capable models on a need to have basis only. Using something like o3 by default makes no sense. I don't want to have to think about which model is optimal for what question. The problem of figuring out what model is best to use is a much simpler one than answering my questions. And automating that decision opens the doors to having a multitude of specialized models.
You're missing that Claude desktop has MCP servers, which can extend it to do a lot more, including much better real life "out of the box" uses. You can do things like use Obsidian as a filesystem or connect to local databases to really extend the abilities. You can also read and write to github directly and bring in all sorts of other tools.
Wouldn't it be possible to trade speed back for accuracy, e.g. by asking the model to look at a problem from different angles, let it criticize its own output, etc.?
Just have it sample for longer or create a simple workflow that uses a monte carlo tree search approach. Don't see why this wont improve accuracy. I would love to see someone run tests to see how accurate the model is compared to similar parameter models in a per time block benchmark. Like if it can get same accuracy as a similar parameter autoregressive model but with half the speed, you already have a winner, besides other advantages of a diffusion based model.
> Not sure if I would tradeoff speed for accuracy.
Are you, though?
There are obvious examples of obtaining speed without losing accuracy, like using a faster processor with bigger caches, or more processors.
Or optimizing something without changing semantics, or the safety profile.
Slow can be unreliable; a 10 gigabit ethernet can be more reliable than a 110 baud acoustically-coupled modem in mean time between accidental bit flips.
Here, the technique is different, so it is apples to oranges.
Could you tune the LLM paradigm so that it gets the same speed, and how accurate would it be?
Definitely needs faster and cheaper models. Fast and cheap models could replace software in tons of situations. Imagine a vending machine or a mobile game or a word processor where basically all logic is implemented as a prompt to an llm. It would serve as the ultimate high level programming language.
I think natural language to code is the right abstraction. Easy enough barrier to entry but still debuggable. Debugging why an LLM randomly gives you Mountain Dew instead of Sprite if you have a southern accent sounds like a nightmare.
I'm not sure it would be that hard to debug. Make sure you can reproduce the llm state (by storing the random seed for the session, or something like that) and then ask it "why did you just now give that customer mountain dew when they ordered sprite?"
Of course a llm can't think. But that doesn't mean it can't answer simple questions about the output that was produced. Just try it out with chatgpt when you have time. Even if it's not perfectly accurate it's still useful for debugging.
Just think about it as a human employee. Can they always say why they did what they did? Often, but not always. Sometimes you will have to work to figure out the misunderstanding.
If you have much more speed in the available time, for an activity like coding, you could use that for iteration, writing more tests and satisfying them, especially if you can pair that with a concurrent test runner to provide feedback. I'm not sure the end result would be lower scoring/smartness than an LLM could achieve in the same duration.
I'm not sure the end result would be lower scoring/smartness than an LLM could achieve in the same duration.
It probably wouldn’t with current models. That’s exactly why I said we need smarter models - not more speed.
Unless you want to “use that for iteration, writing more tests and satisfying them, especially if you can pair that with a concurrent test runner to provide feedback.” - I personally don’t.
They can by the normal English definitions of "think" and "smart". You're just redefining those words to exclude AI because you feel threatened by it. It's tedious.
Check out RooCode if you haven’t. There’s an orchestrator mode that can start with a big model to come up with a plan and break down, then spin out small tasks to smaller models for scoped implementation.
If you’re open to a terminal-based approach, this is exactly what my project Plandex[1] focuses on—breaking up and completing large tasks step by step.
The excitement for me is the implications for lower energy models. Tech like this could thoroughly break the Nvidia stranglehold at least for some segments
I don't use the best available models for prototyping because it can be expensive or more time consuming. This innovation makes prototyping faster and practicing prompts on slightly lower accuracy models can provide more realistic expectations.
If the benchmarks aren't lying, Mercury Coder Small is as smart as 4o mini and costs the same, but is order of magnitude faster when outputting (unclear if pre-output delay is notably different). Pretty cool. However, I'm under the impression that 4o-mini was superceded by 4.1-mini and 4.1-nano for all use cases (correct me if I'm wrong). Unfortunately they didn't publish comparisons with the 4.1 line, which feels like an attempt to manipulate the optics. Or am I misreading this?
Btw, why call it "coder"? 4o-mini level of intelligence is for extracting structured data and basic summaries, definitely not for coding.
It appears to be purpose-trained for coding. They also have a generalist model, but that's not the one being compared.
I agree, the comparison is dated, cherry-picked and doesn't reference the thinking models people do use for coding.
But it's also a bit of a new architecture in early stages of development/testing. Comparing against other small non-thinking models is a good step. It demonstrates the strategy is viable and worth exploring. Time will tell its value. Perhaps a guiding LLM could lean on diffusion to speed up generation. Perhaps we'll see more mixed-architecture models. Perhaps diffusion beats out current LLMs, but from my armchair this seems unlikely.
Saw another on Twitter past few days that looked like a better contender to Mercury, doesn't look like it got posted to LocalLLaMa, and I can't find it now. Very exciting stuff
I just tried giving it a coding snippet that has a bug. ChatGPT & Claude found the bug instantly. Mercury fails to find it even after several reprompts (it's hallucinating). On the upside it is significantly faster. That's promising since the edge for ChatGPT and Claude are in the prolonged time and energy they've spent building training infrastructure, tooling, datasets, etc to pump out models with high task performance.
Keep in mind this release was never intended to prove superiority. Rather, it shows an alternative structure with some promising performance characteristics. More work needs to be done to show real application, but this very valuable learning.
That's part of the reason to compare against older, smaller models since they're at a more comparable stage of development.
Looks interesting, and my intuition is that code is a good application of diffusion LLMs, especially if they get support for "constrained generation", as there's already plenty of tooling around code (linters and so on).
Something I don't see explored in their presentation is the ability of the model to restore from errors / correct itself. SotA LLMs shine at this, a few back and forth w/ sonnet / gemini pro / etc really solves most problems nowadays.
It's nice to see a team doing something different.
The cost[1] is US$1.00 per million output tokens and US$0.25 per million input tokens. By comparison, Gemini 2.5 Flash Preview charges US$0.15 per million tokens for text input and $0.60 (non-thinking) output[2].
Hmmm... at those prices they need to focus on markets where speed is especially important, eg high-frequency trading, transcription/translation services and hardware/IoT alerting!
I would be extremely hesitant to assume a direct relationship between pricing and cost. A behemoth like Google is very willing to take significant losses for years to grow market share. Back in 2014-2015 Uber often charged less than the Boston subway, but it always cost them MUCH more under the hood. AFAIK they're still not profitable.
Chinese companies will be similarly eager for market share, but not everyone has the access to the same raw capital.
HFT is limited by time on how much processing it can do. In theory, a super-fast dLLM would enable to incorporate information sources in their decision-making that were previously too high-level. E.g., imagine using wire reports to predict an arbitrage opportunity that doesn't even exist yet (I dunno, not an HFT guy).
In practice, iiuc, HFT still happens within 10s of milliseconds, and I doubt even current dLLM is THAT fast.
It's kind of weird to think that in a coding assistant, an LLM is regularly asked to produce a valid block of code top to bottom, or repeat a section of code with changes, when that's not what we do. (There are other intuitively odd things about this, like the amount of compute spent generating 'easy' tokens, e.g. repeating unchanged code.) Some of that might be that models are just weird and intuition doesn't apply. But maybe the way we do it--jumping around, correcting as we go, etc.--is legitimately an efficient use of effort, and a model could do its job better, with less effort, or both if it too used some approach other than generating the whole sequence start-to-finish.
There's already stuff in the wild moving that direction without completely rethinking how models work. Cursor and now other tools seem to have models for 'next edit' not just 'next word typed'. Agents can edit a thing and then edit again (in response to lints or whatever else); approaches based on tools and prompting like that can be iterated on without the level of resources needed to train a model. You could also imagine post-training a model specifically to be good at producing edit sequences, so it can actually 'hit backspace' or replace part of what it's written if it becomes clear it wasn't right, or if two parts of the output 'disagree' and need to be reconciled.
> an LLM is regularly asked to produce a valid block of code top to bottom, or repeat a section of code with changes, when that's not what we do.
Eh, it's mostly what we do. We don't re-type everything every time, but we do type top-to-bottom when we type. As you later mentioned, "next edit" models really strike that balance, and they're like 50% of the value I derive from a tool like Cursor.
I'd love to see more diff-outputs instead of "retyping" everything (with a nice UI for the humans). I suspect that part of the reason we have these "inhuman" actions is because of the chat interface we've been using has lead to certain outputs being more desirable due to the medium.
Depends on the shape of the cup! You can contrive a cup shaped like an exponentially flaring horn, where adding the milk increases the volume a little, which massively increases the surface area, and so leads to faster cooling. Or you can have a cup with a converging top, like a brandy glass, where adding the milk reduces the surface area, and makes cooling even slower.
To determine which option cools coffee the most, I'll analyze the heat transfer physics involved.
The key insight is that the rate of heat loss depends on the temperature difference between the coffee and the surrounding air. When the coffee is hotter, it loses heat faster.
Option 1 (add milk first, then wait):
- Adding cold milk immediately lowers the coffee temperature right away
- The coffee then cools more slowly during the 2-minute wait because the temperature difference with the environment is smaller
Option 2 (wait first, then add milk):
- The hot coffee cools rapidly during the 2-minute wait due to the large temperature difference
- Then the cold milk is added, creating an additional temperature drop at the end
Option 2 will result in the lowest final temperature. This is because the hotter coffee in option 2 loses heat more efficiently during the waiting period (following Newton's Law of Cooling), and then gets the same cooling benefit from the milk addition at the end.
The mathematical principle behind this is that the rate of cooling is proportional to the temperature difference, so keeping the coffee hotter during the waiting period maximizes heat loss to the environment.
As apposed to humans who all derive the physics of heat transfer independently when given a question like this?
Not picking on you - this brings up something we could all get better at:
There should be a "First Rule of Critiquing Models": Define a baseline system to compare performance against. When in doubt, or for general critiques of models, compare to real world random human performance.
Without a real practical baseline to compare with, its to easy to fall into subjective or unrealistic judgements.
"Second Rule": Avoid selectively biasing judgements by down selecting performance dimensions. For instance, don't ignore difference in response times, grammatical coherence, clarity of communication, and other qualitative and quantitative differences. Lack of comprehensive performance dimension coverage is like comparing runtimes of runners, without taking into account differences in terrain, length of race, altitude, temperature, etc.
It is very easy to critique. It is harder to critique in a way that sheds light.
If an LLM has only that knowledge and nothing else (pieces of text saying that heat transfer is proportional to some function of the temp difference) such that is not trained on any texts that give problems and solutions in this area, it will not work this out, since it has nothing to generate tokens from.
Also, your knowledge doesn't come from anywhere near having scanned terabytes of text, which would take you multiple lifetimes of full time work.
Hmm a good nerd-snipe puzzle. I was never very good at physics, so hopefully someone can check my work... assuming upon mixing coffee is at Tc and milk at Tm, and simplifying to assume equivalent mass & specific temp we have (Tf - Tc) = -(Tf - Tm) => Tf = (Tc+Tm)/2 which is intuitive (upon mixing we get the average temperature).
On the assumption that the cold milk is always at a fixed temperature until it's mixed in, then the temperature of coffee at point of mixing is the main factor. Before and after, it follows newton's law of cooling. So we're comparing something like Tenv + [(Tc+Tm)/2 - Tenv]e^(-2) vs (Tenv + [Tc - Tenv]e^(-2) + Tm)/2. The latter is greater than the former only when Tm > Tenv (the milk isn't cold), or in other words it's better to let the coffee cool as much as possible before mixing assuming the milk is colder than the environment.
Another interesting twist is to consider the case where the milk isn't kept at a fixed temperature but is also subject to warming (it's taken out of the fridge). Then the former equation is unchanged but the latter becomes (Tenv + [Tc - Tenv]e^(-2) + Tenv + [Tm - Tenv]e^(-2))/2. But this is equivalent to the former equation, so in this case it doesn't matter when you mix it.
Not 100% confident in both analysis, but I wonder if there's a more intuitive way to see it. I also don't know if deviating from the assumption of equivalent mass & specific temp changes the analysis (it might lead to a small range where for the fixed case, situation 1 is better?) It's definitely not "intuitive" to me.
The puzzle assumes that the room temperature is greater than the cold milk's temperature. When I added that the room temperature is, say, -10 °C, Mercury fails to see the difference.
Under any reasonable assumptions for the size and shape of the cup, the amount of coffee, the makeup of the air, etc., the room being -10c won't change the result.
It would only matter if the air were able to cool the coffee to a temperature less than that of the milk in under 2 minutes.
Okay, let's break this down using the principle of heat transfer (specifically Newton's Law of Cooling):
Heat loss is faster when the temperature difference is greater. A very hot object loses heat to cool surroundings much faster than a warm object.
Option 1: Add milk immediately.
You instantly lower the coffee's temperature by mixing it with cold milk.
Now, this warmer (not hot) mixture sits for 2 minutes. Because the temperature difference between the mixture and the room is smaller, it cools down more slowly over those 2 minutes.
Option 2: Let it sit for 2 mins, then add milk.
The very hot coffee sits for 2 minutes. Because the temperature difference between the hot coffee and the room is large, it loses heat more quickly during these 2 minutes.
After 2 minutes of rapid cooling, you then add the cold milk, lowering the temperature further.
Conclusion:
To get the coffee to the lowest temperature, you should choose Option 2: Let it sit for 2 mins, then add the cold milk.
The rate of change of temperature is proportional to the difference in temperature. Adding the cold milk later let's you juice a larger temperature differential out of the 2-minute waiting phase. So you should add the milk later.
I think what the other person is asking about is: can you be sure that the milk is (as) cold later?
There's a lot of discussion about what happens to the temperature of the coffee during the 2 minutes. What happens to the temperature of the milk during that same time?
Where is the milk stored? Do you grab it out of the refrigerator the moment you add it to the coffee? Or the cold milk sitting out on the countertop getting warmer? If so, how rapidly?
It's a safe bet that freshly brewed coffee is much farther from room temperature than refrigerated milk is. However deriving properties related to that symmetry (or lack thereof) would make an excellent question for an exam in an introductory class.
The two options are equivalent, since the final (equilibrium) temp of an adiabatic system (coffee + Milk + room) must be the same - ie it's the total amount of heat transferred that matters, and not the rates of heat transfer.
If the system is not adiabatic, ie the room is not big enough to remain constant temp, or equilibrium is not achieved in 2 mins of cooling, then the puzzle statement must be specify all three initial temps to be well poised.
What gave you the idea that this is an adiabatic system? It's a cup of freshly brewed coffee on your kitchen counter. Equilibrium will certainly not be achieved within 2 minutes. Even if it were, different schemes would reach it at different time points.
It is fundamentally a question about rate of energy transfer.
The thing to notice about the symmetric system is that both items will experience the same rate of transfer. However there's presumably more coffee in your coffee than there is milk so it's not actually symmetric. If you adjust the parameters for volume and specific heat to make the final mixed product symmetric then it no longer matters when you do the mixing.
For me, ChatGPT (the free version, GPT-4o mini I believe?) gets it right, choosing option 2 because the coffee will cool faster due to the larger temperature difference.
Unless there's a gotcha somewhere in your prompt that I'm missing, like what if the temperature of the room is hotter than the coffee, or so cold that the coffee becomes colder than the milk, or something?
I would be suprised if any models get it wrong, since I assume it shows up in training data a bunch?
This is what I got from full-fat 4o. Maybe thinking less helps!
ChatGPT:
Option 1 — Add the cold milk immediately — will result in a lower final temperature after 2 minutes.
Why:
• Heat loss depends on the temperature difference between the coffee and the environment (usually room temperature).
• If you add the milk early, the overall temperature of the coffee-milk mixture is reduced immediately. This lowers the average temperature over the 2 minutes, so less heat is lost to the air.
• If you wait 2 minutes to add the milk, the hotter coffee loses more heat to the environment during those 2 minutes, but when you finally add the milk, it doesn’t cool it as much because the coffee’s already cooler and the temp difference between the milk and the coffee is smaller.
Summary:
• Adding milk early = cooler overall drink after 2 minutes.
• Adding milk late = higher overall temp after 2 minutes, because more heat escapes during the time the coffee is hotter.
Want me to show a simple simulation or visualisation of this?
I had it write a Python program to calculate disk usage by directory -- basically a `du` clone. It was astonishly fast (2s) and correct. I've tried other models which have got it wrong, slow, and they've ignored my instructions to use topdown=False in the call to walk().
> Mercury gets this right - while as of right now ChatGPT 4o get it wrong.
This is so common a puzzle it's discussed all over the internet.
It's in the data used to build the models.
What's so impressive about a machine that can spit out something easily found with a quick web search?
I'd hope that with diffusion, it would be able to go back and forth between parts of the output to adjust issues with part of the output which it had previously generated. This would not be possible with a purely sequential model.
However,
> Prompt: Write a sentence with ten words which has exactly as many r’s in the first five words as in the last five
Would have been nice if along to this demo video[1] comparing speed of 3 models, they would have share the artifacts as well, so we can compare quality.
Anybody able to get the "View Technical Report" button at the bottom to do anything? I was curious to glean more details but it doesn't work on either of my devices.
I'm curious what level of detail they're comfortable publishing around this, or are they going full secret mode?
I would be interested to see how people would apply this working as a coding assistant. For me, its application in solutioning seem very strong, particularly vibe coding, and potentially agentic coding. One of my main gripes with LLM-assisted coding is that for me to get the output which catches all scenarios I envision takes multiple attempts in refining my prompt requiring regeneration of the output. Iterations are slow and often painful.
With the speed this can generate its solutions, you could have it loop through attempting the solution, feeding itself the output (including any errors found), and going again until it builds the "correct" solution.
I basically did this with aider and Gemini 2.5 few days ago and was blown away. Basically talked about the project structure, let it write the final plan to file CONVENTIONS.md that gets automatically attached to the context, then kept asking "What should we do next" until tests were ready, and then I just ran a loop where it modifies the code and I press Return to run the tests and add the output to prompt and let it go again.
About 10 000 lines of code, and I only intervened a few times, to revert few commits and once to cut a big file to smaller ones so we could tackle the problems one by one.
I did not expect LLMs to be able to do this so soon. But I just commented to say about aider - the iteration loop really was mostly me pressing return. Especially in the navigator mode PR, as it automatically looked up the correct files to attach to the context
This sounds like a neat idea but it seems like bad timing. OpenAI just released token-based that beats the best diffusion image generation. If diffusion isn't even the best at generating images, I don't know if I'm going to spend a lot of time evaluating it for text.
Speed is great but it doesn't seem like other text-based model trends are going to work out of the box, like reasoning. So you have to get dLLMs up to the quality of a regular autoregressive LLM and then you need to innovate more to catch up to reasoning models, just to match the current state of the art. It's possible they'll get there, but I'm not optimistic.
I wonder if it benefits because it can attend to individual tokens of the prompt while generating, compared to typical diffusion models that just get a static vector embedding of the prompt.
I just tried it and it was able to perfectly generate a piece of code for me that i needed for generating a 12 month rolling graph based on a list of invoices and it seemed a bit easier and faster then chatgpt.
This is awesome for the future of autocomplete. Current models aren't fast enough to give useful suggestions at the speed that I type - but this certainly is.
That said, token-based models are currently fast enough for most real-time chat applications, so I wonder what other use-cases there will be where speed is greatly prioritized over smarts. Perhaps trading on Trump tweets?
There are so many models. Every single day half a dozen new models land. And even more papers.
It feels like models are becoming fungible apart from the hyperscaler frontier models from OpenAI, Google, Anthropic, et al.
I suppose VCs won't be funding many more "labs"-type companies or "we have a model" as the core value prop companies? Unless it has a tight application loop or is truly unique?
Disregarding the team composition, research background, and specific problem domain - if you were starting an AI company today, what part of the stack would you focus on? Foundation models, AI/ML infra, tooling, application layer, ...?
Where does the value accrue? What are the most important problems to work on?
Word on the street is a lot of money is going into vertical application AI companies this season. Makes sense - the bitter lesson means capturing a market and proprietary data is a good play, while frontier models keep getting better at using what you (and only you) own.
Interesting approach. However, I never thought of auto regression being _the_ current issue with language modeling. If anything it seems the community was generally surprised just how far next "token" prediction took us. Remember back when we did char generating RNNs and were impressed they could make almost coherent sentences?
Diffusion is an alternative but I am having a hard time understanding the whole "built in error correction" that sounds like marketing BS. Both approaches replicate probability distributions which will be naturally error-prone because of variance.
Consider the entropy of the distribution of token X in these examples:
"Four X"
and
"Four X and seven years ago".
In the first case X could be pretty much anything, but in the second case we both know the only likely completion.
So it seems like there would be a huge advantage in not having to run autogressively. But in practice it's less significant then you might imagine because the AR model can internally model the probability of X conditioned on the stuff it hasn't output yet, and in fact because without reinforcement the training causes it converge on the target probability of the whole output, the AR model must do some form of lookahead internally.
(That said RLHF seems to break this product of the probabilities property pretty badly, so maybe it will be the case that diffusion will suffer less intelligence loss ::shrugs::).
Diffusion models are built around this type of internal lookahead from the start (accurate near prediction, progressively less accurate far prediction, step forward, repeat). They just do it in the coarse-to-fine direction, i.e. in a different dimension, and had more thought put into shortcuts and speed-accuracy tradeoffs in this process. RL is also used with both types of models. It's not immediately obvious that one must necessarily be more efficient.
Both are conditional distributions on the context of which they were requested so like you said in the second paragraph, the difference is not significant. I see what you mean though and maybe there are use cases then where Diffusion is preferable. To me it seems the context conditional and internal model is sufficient where this problem doesn't really occur.
::nods:: in the case of diffusion though "conditional on its own (eventual) output" is more transparent and explicit.
As an example of one place that might make a difference is that some external syntax restriction in the sampler is going to enforce the next character after a space is "{".
Your normal AR LLM doesn't know about this restriction and may pick the tokens leading up to the "{" in a way which is regrettable given that there is going to be a {. The diffusion, OTOH, can avoid that error.
In the case where there isn't an artificial constraint on the sampler this doesn't come up because when its outputting the earlier tokens the AR model knows in some sense about it's own probability of outputting a { later on.
But in practice pretty much everyone engages in some amount of sampler twiddling, even if just cutting off low probability tokens.
As far as the internal model being sufficient, clearly it is or AR LLMs could hardly produce coherent English. But although it's sufficient it may not be particularly training or weight efficient.
I don't really know how these diffusion text models are trained so I can't really speculate, but it does seem to me that getting to make multiple passes might allow it less circuit depth. I think of it in terms of every AR step must expend effort predicting something about the following next few steps in order to output something sensible here, this has to be done over and over again, even though it doesn't change.
Totally separate from this line of discussion is that if you want to use an LLM for, say, copyediting it's pretty obvious to me how a diffusion model could get much better results.
Like if you take your existing document and measure the probability of your actual word vs an AR model's output, varrious words are going to show up as erroneously improbable even when the following text makes them obvious. A diffusion model should just be able to score up the entire text conditioned on the entire text rather than just the text in front of it.
The linked page only compares to very old and very small models. But the pricing is higher even than the latest Gemini Flash 2.5 model, which performs far better than anything they compare to.
Their pockets are probably not as deep as Google's in terms of willingness to burn cash for market share.
If speed is your most important metric, I could still see there being a niche for this.
From a pure VC perspective though, I wonder if they'd be better off Open Sourcing their model to get faster innovation + centralization like Llama has done. (Or Mistral with keeping some models private, some public.)
Use it as marketing, get your name out there, and have people use your API when they realize they don't want to deal with scaling AI compute themselves lol
Is cost really the main differentiator here, tho? "Solving" coding seems like the holy grail atm (and I agree, it can enable a bunch of things once that's done) and "traditional, organic, human fed code" is pretty expensive atm, so does cost really matter now?
Put another way, how much would company x be willing to spend on "here's a repo, here are the tests, here is the speed now, make this faster while still passing all the tests". If it "solves" something in cudnn that makes it 10% faster, how much would nvidia pay for this? 1m$? 10m$?
this convo has me rethinking how much speed actually matters vs just getting stuff right - you think most problems are just about better habits or purely tooling upgrades at this point?
This is genius! There are tradeoffs between diffusion and neural network models in image generation so why not use diffusion models in text generation? Excited to see where this ends up and I wouldn't be surprised if we saw some of these types of models appear in the future updates to popular families like Llama or Qwen.
>Mercury is up to 10x faster than frontier speed-optimized LLMs. Our models run at over 1000 tokens/sec on NVIDIA H100s, a speed previously possible only using custom chips.
This means on custom chips (Cerebras, Graphcore, etc...) we might see 10k-100k tokens/sec? Amazing stuff!
Also of note, funny how text generation started w/ autoregression/tokens and diffusion seems to perform better, while image generation went the opposite way.
Super happy to see something like this getting traction. As someone that is trying to reduce my carbon footprint sometimes I feel bad about asking any model to do something trivial. With something like that perhaps the guilt will lessen
If you live in the U.S., marginal electricity demand during the day is almost invariably met with solar or wind (solar typically runs at a huge surplus on sunny days). Go forth and AI in peace, marcyb5st.
Thanks! That helps somewhat. However, it feels like that's just part of the story.
If I remember correctly hyperscalers put their green agendas in stasis now that LLMs are around and that makes me believe that there is a CO2 cost associated.
Still, any improvement is a good news and if diffusion models replace autoregressive models we can invest that surplus in energy in something else useful for the environment.
This made me wonder - do any cloud compute systems have an option to time jobs or use physical resources geographically based on surplus power availability to minimise emissions?
I reckon it might incidentally happen if optimising for cost of power depending how correlated that is to carbon intensivity of power generation, which admittedly I haven't thought through.
To put this into perspective, driving for an hour in an electric car (15kW avg consumption) consumes about as much energy as 50,000 chatgpt queries [0]
Running your laptop for an hour would be around 100 queries.
Not sure if I would tradeoff speed for accuracy.
Yes, it's incredible boring to wait for the AI Agents in IDEs to finish their job. I get distracted and open YouTube. Once I gave a prompt so big and complex to Cline it spent 2 straight hours writing code.
But after these 2 hours I spent 16 more tweaking and fixing all the stuff that wasn't working. I now realize I should have done things incrementally even when I have a pretty good idea of the final picture.
I've been more and more only using the "thinking" models of o3 in ChatGPT, and Gemini / Claude in IDEs. They're slower, but usually get it right.
But at the same time I am open to the idea that speed can unlock new ways of using the tooling. It would still be awesome to basically just have a conversation with my IDE while I am manually testing the app. Or combine really fast models like this one with a "thinking background" one, that would runs for seconds/minutes but try to catch the bugs left behind.
I guess only giving a try will tell.
So my personal belief is that diffusion models will enable higher degrees of accuracy. This is because unlike an auto-regressive model it can adjust a whole block of tokens when it encounters some kind of disjunction.
Think of the old example where an auto regressive model would output: "There are 2 possibilities.." before it really enumerated them. Often the model has trouble overcoming the bias and will hallucinate a response to fit the proceeding tokens.
Chain of thought and other approaches help overcome this and other issues by incentivizing validation, etc.
With diffusion however it is easier for the other generated answer to change that set of tokens to match the actual number of possibilities enumerated.
This is why I think you'll see diffusion models be able to do some more advanced problem solving with a smaller number of "thinking" tokens.
Unfortunately the intuition and the math proofs so far suggest that autoregressive training is learning the joint distribution of probabilistic streams of tokens much better than diffision models do or will ever do. My intuitive take is that the conditional probability distribtion of decoder-only autoregressive models is at just the right level of complexity for probabilistic models to learn accurately enough. Intuitively (and simplifying things at the risk of breaking rigor), the diffusion (or masked models) have to occasionally issue tokens with less information and thus higher variance than a pure autoregressive model would have to do, so the joint distribution, ie the probability of the whole sentence/answer will be lower and thus diffusion models will never get precise enough. Of course, during generation the sampling techniques influence the above simplified idea dramatically and the typical randomized sampling for next token prediction is suboptimal and could be beaten by a carefully designed block diffusion sampler in principle in some contexts though I havent seen real examples of it yet. But the key ideas of the above scribbles are still valid: autoregresive models will always be better (or at least equal) probabilistic models of sequential data than diffusion models will be. So the diffusion models mostly offer a tradeoff for performance vs quality. Sometimes there is a lot of room for that tradeoff in practice.
This is tremendously interesting!
Could you point me to some literature? Especially regarding mathematical proofs of your intuition?
I’d like to recalibrate my priors to align better with current research results.
From the mathematical point of view the literature is about the distinction between a "filtering" distribution and a "smoothing" distribution. The smoothing distribution is strictly more powerful.
In theory intuitively the smoothing distribution has access to all the information that the filtering distribution has and some additional information therefore has a minimum lower than the filtering distribution.
In practice, because the smoothing input space is much bigger, keeping the same number of parameters we may not reach a better score because with diffusion we are tackling a much harder problem (the whole problem), whereas with autoregressive models we are taking a shortcut which happens to probably be one that humans are probably biased too (communication evolved so that it can be serialized to be exchanged orally).
I am still jetlagged and not sure what the most helpful reference would be. Maybe start from the block diffusion paper I recommended in a parallel thread and trace your way up/down from there. The logic leading to Eq 6 is a special case of such a math proof.
https://openreview.net/forum?id=tyEyYT267x
What are the barriers to mixed architecture models? Models which could seamlessly pass from autoregressive to diffusion, etc.
Humans can integrate multiple sensory processing centers and multiple modes of thought all at once. It's baked into our training process (life).
The human processing is still autoregressive, but using multiple parallel synchronized streams. There is no problem with such an approach and my best guess is that in the next year we will see many teams training models using such tricks for generating reasoning traces in parallel.
The main concern is taking a single probabilistic stream (eg a book) and comparing autoregressive modelling of it with a diffusive modelling of it.
Regarding mixing diffusion and autoregressive—I was at ICLR last week and this work is probably relevant: https://openreview.net/forum?id=tyEyYT267x
Suggests an opportunity for hybrids, where the diffusion model might be responsible for large scale structure of response and the next token model for filling in details. Sort of like a multi scale model in dynamics simulations.
> it can adjust a whole block of tokens when it encounters some kind of disjunction.
This is true in principle for general diffusion models, but I don't think it's true for the noise model they use in Mercury (at least, going by a couple of academic papers authored by the Inception co-founders.) Their model generates noise by masking a token, and once it's masked, it stays masked. So the reverse-diffusion gets to decide on the contents of a masked token once, and after that it's fixed.
Here are two papers linked from Inception's site:
1. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution - https://arxiv.org/abs/2310.16834
2. Simple and Effective Masked Diffusion Language Models - https://arxiv.org/abs/2406.07524
Thanks, yes, I was thinking specifically of "Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution". They actually consider two noise distributions: one with uniform sampling for each noised token position, and one with a terminal masking (the Q^{uniform} and Q^{absorb}.) However, the terminal-masking system is clearly superior in their benchmarks.
https://arxiv.org/pdf/2310.16834#page=6
The exact types of path dependencies in inference on text-diffusion models look like an interesting research project.
Yes, the problem is coming up with a noise model where reverse diffusion is tractable.
Thank you, I'll have to read the papers. I don't think I have read theirs.
Once that auto-regressive model goes deep enough (or uses "reasoning"), it actually has modeled what possibilities exist by the time it's said "There are 2 possibilities.."
We're long past that point of model complexity.
But as everyone knows, computer science has two hard problems: naming things, cache invalidation, and off by one errors.
You left an LLM to code for two hours and then were surprised when you had to spend a significant amount of time more cleaning up after it?
Is this really what people are doing these days?
I think speed and convenience are essential. I use chat gpt desktop for coding. Not because it's the best but because it's fast and easy and doesn't interrupt my flow too much. I mostly stick to the 4o model. I only use the o3 model when I really have to. Because at that point getting an answer is slooooow. 4o is more than good enough most of the time.
And more importantly it's a simple option+shift+1 away. I simply type something like "fix that" and it has all the context it needs to do its thing. Because it connects to my IDE and sees my open editor and the highlighted line of code that is bothering me. If I don't like the answer, I might escalate to o3 sometimes. Other models might be better but they don't have the same UX. Claude desktop is pretty terrible, for example. I'm sure the model is great. But if I have to spoon feed it everything it's going to annoy me.
What I'd love is for smaller faster models to be used by default and for them to escalate to slower more capable models on a need to have basis only. Using something like o3 by default makes no sense. I don't want to have to think about which model is optimal for what question. The problem of figuring out what model is best to use is a much simpler one than answering my questions. And automating that decision opens the doors to having a multitude of specialized models.
You're missing that Claude desktop has MCP servers, which can extend it to do a lot more, including much better real life "out of the box" uses. You can do things like use Obsidian as a filesystem or connect to local databases to really extend the abilities. You can also read and write to github directly and bring in all sorts of other tools.
Wouldn't it be possible to trade speed back for accuracy, e.g. by asking the model to look at a problem from different angles, let it criticize its own output, etc.?
Just have it sample for longer or create a simple workflow that uses a monte carlo tree search approach. Don't see why this wont improve accuracy. I would love to see someone run tests to see how accurate the model is compared to similar parameter models in a per time block benchmark. Like if it can get same accuracy as a similar parameter autoregressive model but with half the speed, you already have a winner, besides other advantages of a diffusion based model.
> Not sure if I would tradeoff speed for accuracy.
Are you, though?
There are obvious examples of obtaining speed without losing accuracy, like using a faster processor with bigger caches, or more processors.
Or optimizing something without changing semantics, or the safety profile.
Slow can be unreliable; a 10 gigabit ethernet can be more reliable than a 110 baud acoustically-coupled modem in mean time between accidental bit flips.
Here, the technique is different, so it is apples to oranges.
Could you tune the LLM paradigm so that it gets the same speed, and how accurate would it be?
AI field desperately needs smarter models - not faster models.
Definitely needs faster and cheaper models. Fast and cheap models could replace software in tons of situations. Imagine a vending machine or a mobile game or a word processor where basically all logic is implemented as a prompt to an llm. It would serve as the ultimate high level programming language.
Why on earth would you implement a vending machine using an LLM?
Because it's easy and cheap. Like how many products use a Raspberry Pi or ESP32 when an ATtiny would do.
The same reason we make the butter dish suffer from existential angst.
I think natural language to code is the right abstraction. Easy enough barrier to entry but still debuggable. Debugging why an LLM randomly gives you Mountain Dew instead of Sprite if you have a southern accent sounds like a nightmare.
I'm not sure it would be that hard to debug. Make sure you can reproduce the llm state (by storing the random seed for the session, or something like that) and then ask it "why did you just now give that customer mountain dew when they ordered sprite?"
> and then ask it "why did you just now give that customer mountain dew when they ordered sprite?"
Worse than useless for debugging.
An LLM can't think and doesn't have capabilities for self-reflection.
It will just generate a plausible stream of tokens in reply that may or may not correspond to the real reason why.
Of course a llm can't think. But that doesn't mean it can't answer simple questions about the output that was produced. Just try it out with chatgpt when you have time. Even if it's not perfectly accurate it's still useful for debugging.
Just think about it as a human employee. Can they always say why they did what they did? Often, but not always. Sometimes you will have to work to figure out the misunderstanding.
Why not just store the state in the code and debug as usual, perhaps with LLM assistance? At least that’s tractable.
Yet deepseek has shown that more dialogue increases quality. Increasing speed is therefore important if you need thinking models.
If you have much more speed in the available time, for an activity like coding, you could use that for iteration, writing more tests and satisfying them, especially if you can pair that with a concurrent test runner to provide feedback. I'm not sure the end result would be lower scoring/smartness than an LLM could achieve in the same duration.
I'm not sure the end result would be lower scoring/smartness than an LLM could achieve in the same duration.
It probably wouldn’t with current models. That’s exactly why I said we need smarter models - not more speed. Unless you want to “use that for iteration, writing more tests and satisfying them, especially if you can pair that with a concurrent test runner to provide feedback.” - I personally don’t.
LLM's can't think, so "smarter" is not possible.
They can by the normal English definitions of "think" and "smart". You're just redefining those words to exclude AI because you feel threatened by it. It's tedious.
Technically correct and completely besides the point.
People cant fly.
Check out RooCode if you haven’t. There’s an orchestrator mode that can start with a big model to come up with a plan and break down, then spin out small tasks to smaller models for scoped implementation.
If you’re open to a terminal-based approach, this is exactly what my project Plandex[1] focuses on—breaking up and completing large tasks step by step.
1 - https://github.com/plandex-ai/plandex
The excitement for me is the implications for lower energy models. Tech like this could thoroughly break the Nvidia stranglehold at least for some segments
> I now realize I should have done things incrementally even when I have a pretty good idea of the final picture.
Or just save yourself the time and money and code it yourself like it's 2020.
(Unless it's your employer paying for this waste, in which case go for it, I guess.)
I don't use the best available models for prototyping because it can be expensive or more time consuming. This innovation makes prototyping faster and practicing prompts on slightly lower accuracy models can provide more realistic expectations.
If the benchmarks aren't lying, Mercury Coder Small is as smart as 4o mini and costs the same, but is order of magnitude faster when outputting (unclear if pre-output delay is notably different). Pretty cool. However, I'm under the impression that 4o-mini was superceded by 4.1-mini and 4.1-nano for all use cases (correct me if I'm wrong). Unfortunately they didn't publish comparisons with the 4.1 line, which feels like an attempt to manipulate the optics. Or am I misreading this?
Btw, why call it "coder"? 4o-mini level of intelligence is for extracting structured data and basic summaries, definitely not for coding.
It appears to be purpose-trained for coding. They also have a generalist model, but that's not the one being compared.
I agree, the comparison is dated, cherry-picked and doesn't reference the thinking models people do use for coding.
But it's also a bit of a new architecture in early stages of development/testing. Comparing against other small non-thinking models is a good step. It demonstrates the strategy is viable and worth exploring. Time will tell its value. Perhaps a guiding LLM could lean on diffusion to speed up generation. Perhaps we'll see more mixed-architecture models. Perhaps diffusion beats out current LLMs, but from my armchair this seems unlikely.
There are some open weight attempts at this around too: https://old.reddit.com/r/LocalLLaMA/search?q=diffusion&restr...
Saw another on Twitter past few days that looked like a better contender to Mercury, doesn't look like it got posted to LocalLLaMa, and I can't find it now. Very exciting stuff
this video showing how diffusion models generate text is mesmerizing to look at! (comment in top thread linked in your search results)
https://www.reddit.com/media?url=https://i.redd.it/xci0dlo7h...
That seems fake - diffusion models should evolve details over time, right? This one just feels in the blanks gradually, like an old progressive jpeg.
EDIT: This video in TFA was actually a much cooler demonstration - https://framerusercontent.com/assets/YURlGaqdh4MqvUPfSmGIcao...
I just tried giving it a coding snippet that has a bug. ChatGPT & Claude found the bug instantly. Mercury fails to find it even after several reprompts (it's hallucinating). On the upside it is significantly faster. That's promising since the edge for ChatGPT and Claude are in the prolonged time and energy they've spent building training infrastructure, tooling, datasets, etc to pump out models with high task performance.
Keep in mind this release was never intended to prove superiority. Rather, it shows an alternative structure with some promising performance characteristics. More work needs to be done to show real application, but this very valuable learning.
That's part of the reason to compare against older, smaller models since they're at a more comparable stage of development.
Looks interesting, and my intuition is that code is a good application of diffusion LLMs, especially if they get support for "constrained generation", as there's already plenty of tooling around code (linters and so on).
Something I don't see explored in their presentation is the ability of the model to restore from errors / correct itself. SotA LLMs shine at this, a few back and forth w/ sonnet / gemini pro / etc really solves most problems nowadays.
It's nice to see a team doing something different.
The cost[1] is US$1.00 per million output tokens and US$0.25 per million input tokens. By comparison, Gemini 2.5 Flash Preview charges US$0.15 per million tokens for text input and $0.60 (non-thinking) output[2].
Hmmm... at those prices they need to focus on markets where speed is especially important, eg high-frequency trading, transcription/translation services and hardware/IoT alerting!
1. https://files.littlebird.com.au/Screenshot-2025-05-01-at-9.3...
2. https://files.littlebird.com.au/pb-IQYUdv6nQo.png
I would be extremely hesitant to assume a direct relationship between pricing and cost. A behemoth like Google is very willing to take significant losses for years to grow market share. Back in 2014-2015 Uber often charged less than the Boston subway, but it always cost them MUCH more under the hood. AFAIK they're still not profitable.
Chinese companies will be similarly eager for market share, but not everyone has the access to the same raw capital.
Not sure how HFTs are relevant here
HFT is limited by time on how much processing it can do. In theory, a super-fast dLLM would enable to incorporate information sources in their decision-making that were previously too high-level. E.g., imagine using wire reports to predict an arbitrage opportunity that doesn't even exist yet (I dunno, not an HFT guy).
In practice, iiuc, HFT still happens within 10s of milliseconds, and I doubt even current dLLM is THAT fast.
What is the price on Mercury Mini?
It fails the MU Puzzle¹ by violating rules:
To transform the string "AB" to "AC" using the given rules, follow these steps:
1. *Apply Rule 1*: Add "C" to the end of "AB" (since it ends in "B"). - Result: "ABC"
2. *Apply Rule 4*: Remove the substring "CC" from "ABC". - Result: "AC"
Thus, the series of transformations is: - "AB" → "ABC" (Rule 1) - "ABC" → "AC" (Rule 4)
This sequence successfully transforms "AB" to "AC".
¹ https://matthodges.com/posts/2025-04-21-openai-o4-mini-high-...
It's kind of weird to think that in a coding assistant, an LLM is regularly asked to produce a valid block of code top to bottom, or repeat a section of code with changes, when that's not what we do. (There are other intuitively odd things about this, like the amount of compute spent generating 'easy' tokens, e.g. repeating unchanged code.) Some of that might be that models are just weird and intuition doesn't apply. But maybe the way we do it--jumping around, correcting as we go, etc.--is legitimately an efficient use of effort, and a model could do its job better, with less effort, or both if it too used some approach other than generating the whole sequence start-to-finish.
There's already stuff in the wild moving that direction without completely rethinking how models work. Cursor and now other tools seem to have models for 'next edit' not just 'next word typed'. Agents can edit a thing and then edit again (in response to lints or whatever else); approaches based on tools and prompting like that can be iterated on without the level of resources needed to train a model. You could also imagine post-training a model specifically to be good at producing edit sequences, so it can actually 'hit backspace' or replace part of what it's written if it becomes clear it wasn't right, or if two parts of the output 'disagree' and need to be reconciled.
From a quick search it looks like https://arxiv.org/abs/2306.05426 in 2023 discussed backtracking LLMs and https://arxiv.org/html/2410.02749v3 / https://github.com/upiterbarg/lintseq trained models on synthetic edit sequences. There is probably more out there with some digging. (Not the same topic, but the search also turned up https://arxiv.org/html/2504.20196 from this Monday(!) about automatic prompt improvement for an internal code-editing tool at Google.)
> an LLM is regularly asked to produce a valid block of code top to bottom, or repeat a section of code with changes, when that's not what we do.
Eh, it's mostly what we do. We don't re-type everything every time, but we do type top-to-bottom when we type. As you later mentioned, "next edit" models really strike that balance, and they're like 50% of the value I derive from a tool like Cursor.
I'd love to see more diff-outputs instead of "retyping" everything (with a nice UI for the humans). I suspect that part of the reason we have these "inhuman" actions is because of the chat interface we've been using has lead to certain outputs being more desirable due to the medium.
Ok. My go to puzzle is this:
You have 2 minutes to cool down a cup of coffee to the lowest temp you can
You have two options:
1. Add cold milk immediately, then let it sit for 2 mins.
2. Let it sit for 2 mins, then add the cold milk.
Which one cools the coffee to the lowest temperature and why?
And Mercury gets this right - while as of right now ChatGPT 4o get it wrong.
So that’s pretty impressive.
Depends on the shape of the cup! You can contrive a cup shaped like an exponentially flaring horn, where adding the milk increases the volume a little, which massively increases the surface area, and so leads to faster cooling. Or you can have a cup with a converging top, like a brandy glass, where adding the milk reduces the surface area, and makes cooling even slower.
Clever, but it's implicitly a normal cylindrical cup.
And add the twist: what if the cup is made from a highly conductive material?
Claude 3.7 gets it exactly right:
To determine which option cools coffee the most, I'll analyze the heat transfer physics involved. The key insight is that the rate of heat loss depends on the temperature difference between the coffee and the surrounding air. When the coffee is hotter, it loses heat faster. Option 1 (add milk first, then wait):
- Adding cold milk immediately lowers the coffee temperature right away
- The coffee then cools more slowly during the 2-minute wait because the temperature difference with the environment is smaller
Option 2 (wait first, then add milk):
- The hot coffee cools rapidly during the 2-minute wait due to the large temperature difference
- Then the cold milk is added, creating an additional temperature drop at the end
Option 2 will result in the lowest final temperature. This is because the hotter coffee in option 2 loses heat more efficiently during the waiting period (following Newton's Law of Cooling), and then gets the same cooling benefit from the milk addition at the end. The mathematical principle behind this is that the rate of cooling is proportional to the temperature difference, so keeping the coffee hotter during the waiting period maximizes heat loss to the environment.
That's totally cribbed from some discussion hat occurred in its training.
As apposed to humans who all derive the physics of heat transfer independently when given a question like this?
Not picking on you - this brings up something we could all get better at:
There should be a "First Rule of Critiquing Models": Define a baseline system to compare performance against. When in doubt, or for general critiques of models, compare to real world random human performance.
Without a real practical baseline to compare with, its to easy to fall into subjective or unrealistic judgements.
"Second Rule": Avoid selectively biasing judgements by down selecting performance dimensions. For instance, don't ignore difference in response times, grammatical coherence, clarity of communication, and other qualitative and quantitative differences. Lack of comprehensive performance dimension coverage is like comparing runtimes of runners, without taking into account differences in terrain, length of race, altitude, temperature, etc.
It is very easy to critique. It is harder to critique in a way that sheds light.
This exact problem was in Martin Gardner's column for Scientific American in the 1970s. There are surely references all over the internet.
So is my knowledge of newtons law of cooling
If an LLM has only that knowledge and nothing else (pieces of text saying that heat transfer is proportional to some function of the temp difference) such that is not trained on any texts that give problems and solutions in this area, it will not work this out, since it has nothing to generate tokens from.
Also, your knowledge doesn't come from anywhere near having scanned terabytes of text, which would take you multiple lifetimes of full time work.
We get way more info than llms do, just not solely from text
You have not read every accessible piece of text in existence.
If it was just ‘in the training data’ they’d all get it right.
But they don’t.
Hmm a good nerd-snipe puzzle. I was never very good at physics, so hopefully someone can check my work... assuming upon mixing coffee is at Tc and milk at Tm, and simplifying to assume equivalent mass & specific temp we have (Tf - Tc) = -(Tf - Tm) => Tf = (Tc+Tm)/2 which is intuitive (upon mixing we get the average temperature).
On the assumption that the cold milk is always at a fixed temperature until it's mixed in, then the temperature of coffee at point of mixing is the main factor. Before and after, it follows newton's law of cooling. So we're comparing something like Tenv + [(Tc+Tm)/2 - Tenv]e^(-2) vs (Tenv + [Tc - Tenv]e^(-2) + Tm)/2. The latter is greater than the former only when Tm > Tenv (the milk isn't cold), or in other words it's better to let the coffee cool as much as possible before mixing assuming the milk is colder than the environment.
Another interesting twist is to consider the case where the milk isn't kept at a fixed temperature but is also subject to warming (it's taken out of the fridge). Then the former equation is unchanged but the latter becomes (Tenv + [Tc - Tenv]e^(-2) + Tenv + [Tm - Tenv]e^(-2))/2. But this is equivalent to the former equation, so in this case it doesn't matter when you mix it.
Not 100% confident in both analysis, but I wonder if there's a more intuitive way to see it. I also don't know if deviating from the assumption of equivalent mass & specific temp changes the analysis (it might lead to a small range where for the fixed case, situation 1 is better?) It's definitely not "intuitive" to me.
There is a more intuitive way: Adding the milk later has a chance to get the cup below room temperature, whereas adding it early never will.
That example is probably in the training data?
The puzzle assumes that the room temperature is greater than the cold milk's temperature. When I added that the room temperature is, say, -10 °C, Mercury fails to see the difference.
Under any reasonable assumptions for the size and shape of the cup, the amount of coffee, the makeup of the air, etc., the room being -10c won't change the result.
It would only matter if the air were able to cool the coffee to a temperature less than that of the milk in under 2 minutes.
Token-based Gemini is 4 seconds of thinking:
Okay, let's break this down using the principle of heat transfer (specifically Newton's Law of Cooling):
Conclusion:To get the coffee to the lowest temperature, you should choose Option 2: Let it sit for 2 mins, then add the cold milk.
Is there a trick question in here, where the cold milk is trending closer to room temperature over the two minutes waited?
The rate of change of temperature is proportional to the difference in temperature. Adding the cold milk later let's you juice a larger temperature differential out of the 2-minute waiting phase. So you should add the milk later.
> the cold milk later
I think what the other person is asking about is: can you be sure that the milk is (as) cold later?
There's a lot of discussion about what happens to the temperature of the coffee during the 2 minutes. What happens to the temperature of the milk during that same time?
Where is the milk stored? Do you grab it out of the refrigerator the moment you add it to the coffee? Or the cold milk sitting out on the countertop getting warmer? If so, how rapidly?
It's a safe bet that freshly brewed coffee is much farther from room temperature than refrigerated milk is. However deriving properties related to that symmetry (or lack thereof) would make an excellent question for an exam in an introductory class.
The two options are equivalent, since the final (equilibrium) temp of an adiabatic system (coffee + Milk + room) must be the same - ie it's the total amount of heat transferred that matters, and not the rates of heat transfer.
If the system is not adiabatic, ie the room is not big enough to remain constant temp, or equilibrium is not achieved in 2 mins of cooling, then the puzzle statement must be specify all three initial temps to be well poised.
What gave you the idea that this is an adiabatic system? It's a cup of freshly brewed coffee on your kitchen counter. Equilibrium will certainly not be achieved within 2 minutes. Even if it were, different schemes would reach it at different time points.
It is fundamentally a question about rate of energy transfer.
The thing to notice about the symmetric system is that both items will experience the same rate of transfer. However there's presumably more coffee in your coffee than there is milk so it's not actually symmetric. If you adjust the parameters for volume and specific heat to make the final mixed product symmetric then it no longer matters when you do the mixing.
For me, ChatGPT (the free version, GPT-4o mini I believe?) gets it right, choosing option 2 because the coffee will cool faster due to the larger temperature difference.
Unless there's a gotcha somewhere in your prompt that I'm missing, like what if the temperature of the room is hotter than the coffee, or so cold that the coffee becomes colder than the milk, or something?
I would be suprised if any models get it wrong, since I assume it shows up in training data a bunch?
This is what I got from full-fat 4o. Maybe thinking less helps!
ChatGPT:
Option 1 — Add the cold milk immediately — will result in a lower final temperature after 2 minutes.
Why: • Heat loss depends on the temperature difference between the coffee and the environment (usually room temperature). • If you add the milk early, the overall temperature of the coffee-milk mixture is reduced immediately. This lowers the average temperature over the 2 minutes, so less heat is lost to the air. • If you wait 2 minutes to add the milk, the hotter coffee loses more heat to the environment during those 2 minutes, but when you finally add the milk, it doesn’t cool it as much because the coffee’s already cooler and the temp difference between the milk and the coffee is smaller.
Summary: • Adding milk early = cooler overall drink after 2 minutes. • Adding milk late = higher overall temp after 2 minutes, because more heat escapes during the time the coffee is hotter.
Want me to show a simple simulation or visualisation of this?
I had it write a Python program to calculate disk usage by directory -- basically a `du` clone. It was astonishly fast (2s) and correct. I've tried other models which have got it wrong, slow, and they've ignored my instructions to use topdown=False in the call to walk().
Qwen3 30B A3B: https://pastebin.com/kdJZvqVb
If you let it sit for 2 minutes your time is up and you don't have time to add the cold milk
By this logic you can't let it sit for 2 mins after you add the cold milk either, so both options are invalid.
In math/science questions some things are assumed to be (practically impossibly) instant.
> My go to puzzle is this:
> Mercury gets this right - while as of right now ChatGPT 4o get it wrong.
This is so common a puzzle it's discussed all over the internet. It's in the data used to build the models. What's so impressive about a machine that can spit out something easily found with a quick web search?
Just that what I thought would be better models don’t do it right.
I was expecting this model to be no-where near chatGPT
Although someone above is saying 4o-mini got it right so maybe it’s meaningless. Or maybe thinking less helps…
I'd hope that with diffusion, it would be able to go back and forth between parts of the output to adjust issues with part of the output which it had previously generated. This would not be possible with a purely sequential model.
However,
> Prompt: Write a sentence with ten words which has exactly as many r’s in the first five words as in the last five
>
> Response: Rapidly running, rats rush, racing, racing.
Why not possible with autoregressive model?
o4 mini
https://chatgpt.com/share/681315c2-aa90-800d-b02d-c3ba653281...
Thanks! I didn't know o4 was autoregressive!
Would have been nice if along to this demo video[1] comparing speed of 3 models, they would have share the artifacts as well, so we can compare quality.
[1] https://framerusercontent.com/assets/cWawWRJn8gJqqCGDsGb2gN0...
Anybody able to get the "View Technical Report" button at the bottom to do anything? I was curious to glean more details but it doesn't work on either of my devices.
I'm curious what level of detail they're comfortable publishing around this, or are they going full secret mode?
I would be interested to see how people would apply this working as a coding assistant. For me, its application in solutioning seem very strong, particularly vibe coding, and potentially agentic coding. One of my main gripes with LLM-assisted coding is that for me to get the output which catches all scenarios I envision takes multiple attempts in refining my prompt requiring regeneration of the output. Iterations are slow and often painful.
With the speed this can generate its solutions, you could have it loop through attempting the solution, feeding itself the output (including any errors found), and going again until it builds the "correct" solution.
I basically did this with aider and Gemini 2.5 few days ago and was blown away. Basically talked about the project structure, let it write the final plan to file CONVENTIONS.md that gets automatically attached to the context, then kept asking "What should we do next" until tests were ready, and then I just ran a loop where it modifies the code and I press Return to run the tests and add the output to prompt and let it go again.
About 10 000 lines of code, and I only intervened a few times, to revert few commits and once to cut a big file to smaller ones so we could tackle the problems one by one.
I did not expect LLMs to be able to do this so soon. But I just commented to say about aider - the iteration loop really was mostly me pressing return. Especially in the navigator mode PR, as it automatically looked up the correct files to attach to the context
Unfortunately a 4o mini level of intelligence just isn't enough to make this work, no matter how many iterations you let it try.
Everyone will just switch to LibreOffice and Hetzner.
High tech US service industry exports are cooked.
This sounds like a neat idea but it seems like bad timing. OpenAI just released token-based that beats the best diffusion image generation. If diffusion isn't even the best at generating images, I don't know if I'm going to spend a lot of time evaluating it for text.
Speed is great but it doesn't seem like other text-based model trends are going to work out of the box, like reasoning. So you have to get dLLMs up to the quality of a regular autoregressive LLM and then you need to innovate more to catch up to reasoning models, just to match the current state of the art. It's possible they'll get there, but I'm not optimistic.
The reason image-1 is so good is because it’s the same model doing the talking and the image making.
I wonder if the same would be true for a multi-modal diffusion model that can now also speak?
Facebook has their Chameleon model from 2023 that was in this space. Ancient now.
There is also this GitHub project that I played with a while ago that's trying to do this. https://github.com/GAIR-NLP/anole
Are there any OSS models that follow this approach today? Or are we waiting for somebody to hack that together?
Does it beat them because it's a transformer, or because it's a much larger end-to-end model with higher quality multimodal training?
I wonder if it benefits because it can attend to individual tokens of the prompt while generating, compared to typical diffusion models that just get a static vector embedding of the prompt.
I just tried it and it was able to perfectly generate a piece of code for me that i needed for generating a 12 month rolling graph based on a list of invoices and it seemed a bit easier and faster then chatgpt.
Look into groq.com guys. some good models at similar speed to inception labs
It's faster inference because of the Hardware (LPUs), here the question is about architectures (AR or Diffusions)
SRAM doesn't scale with advanced semiconductor node.
Groq is heading to a dead end.
This is awesome for the future of autocomplete. Current models aren't fast enough to give useful suggestions at the speed that I type - but this certainly is.
That said, token-based models are currently fast enough for most real-time chat applications, so I wonder what other use-cases there will be where speed is greatly prioritized over smarts. Perhaps trading on Trump tweets?
It seems that with this technique you could not possibly do "chain of thought." That technique seems unique to auto-regressive architecture. Right?
1000+ tokens/sec on H100s, a 5–10x speedup over typical autoregressive models — and without needing exotic hardware like Groq or Cerebras - impressive
I actually just tried it. And I’m very impressed. Or at least it’s reasonable code to start with for nontrivial systems.
There are so many models. Every single day half a dozen new models land. And even more papers.
It feels like models are becoming fungible apart from the hyperscaler frontier models from OpenAI, Google, Anthropic, et al.
I suppose VCs won't be funding many more "labs"-type companies or "we have a model" as the core value prop companies? Unless it has a tight application loop or is truly unique?
Disregarding the team composition, research background, and specific problem domain - if you were starting an AI company today, what part of the stack would you focus on? Foundation models, AI/ML infra, tooling, application layer, ...?
Where does the value accrue? What are the most important problems to work on?
Word on the street is a lot of money is going into vertical application AI companies this season. Makes sense - the bitter lesson means capturing a market and proprietary data is a good play, while frontier models keep getting better at using what you (and only you) own.
Speed is great, but you have to set the bar a little higher than last year's tiny models
Interesting approach. However, I never thought of auto regression being _the_ current issue with language modeling. If anything it seems the community was generally surprised just how far next "token" prediction took us. Remember back when we did char generating RNNs and were impressed they could make almost coherent sentences?
Diffusion is an alternative but I am having a hard time understanding the whole "built in error correction" that sounds like marketing BS. Both approaches replicate probability distributions which will be naturally error-prone because of variance.
Consider the entropy of the distribution of token X in these examples:
"Four X"
and
"Four X and seven years ago".
In the first case X could be pretty much anything, but in the second case we both know the only likely completion.
So it seems like there would be a huge advantage in not having to run autogressively. But in practice it's less significant then you might imagine because the AR model can internally model the probability of X conditioned on the stuff it hasn't output yet, and in fact because without reinforcement the training causes it converge on the target probability of the whole output, the AR model must do some form of lookahead internally.
(That said RLHF seems to break this product of the probabilities property pretty badly, so maybe it will be the case that diffusion will suffer less intelligence loss ::shrugs::).
> in the second case we both know the only likely completion.
You two may, but I don't. 'Decades'? 'Months'? 'Wives'? 'Jobs'? 'Conservative PMs'?
Diffusion models are built around this type of internal lookahead from the start (accurate near prediction, progressively less accurate far prediction, step forward, repeat). They just do it in the coarse-to-fine direction, i.e. in a different dimension, and had more thought put into shortcuts and speed-accuracy tradeoffs in this process. RL is also used with both types of models. It's not immediately obvious that one must necessarily be more efficient.
Both are conditional distributions on the context of which they were requested so like you said in the second paragraph, the difference is not significant. I see what you mean though and maybe there are use cases then where Diffusion is preferable. To me it seems the context conditional and internal model is sufficient where this problem doesn't really occur.
::nods:: in the case of diffusion though "conditional on its own (eventual) output" is more transparent and explicit.
As an example of one place that might make a difference is that some external syntax restriction in the sampler is going to enforce the next character after a space is "{".
Your normal AR LLM doesn't know about this restriction and may pick the tokens leading up to the "{" in a way which is regrettable given that there is going to be a {. The diffusion, OTOH, can avoid that error.
In the case where there isn't an artificial constraint on the sampler this doesn't come up because when its outputting the earlier tokens the AR model knows in some sense about it's own probability of outputting a { later on.
But in practice pretty much everyone engages in some amount of sampler twiddling, even if just cutting off low probability tokens.
As far as the internal model being sufficient, clearly it is or AR LLMs could hardly produce coherent English. But although it's sufficient it may not be particularly training or weight efficient.
I don't really know how these diffusion text models are trained so I can't really speculate, but it does seem to me that getting to make multiple passes might allow it less circuit depth. I think of it in terms of every AR step must expend effort predicting something about the following next few steps in order to output something sensible here, this has to be done over and over again, even though it doesn't change.
Totally separate from this line of discussion is that if you want to use an LLM for, say, copyediting it's pretty obvious to me how a diffusion model could get much better results.
Like if you take your existing document and measure the probability of your actual word vs an AR model's output, varrious words are going to show up as erroneously improbable even when the following text makes them obvious. A diffusion model should just be able to score up the entire text conditioned on the entire text rather than just the text in front of it.
The linked page only compares to very old and very small models. But the pricing is higher even than the latest Gemini Flash 2.5 model, which performs far better than anything they compare to.
Their pockets are probably not as deep as Google's in terms of willingness to burn cash for market share.
If speed is your most important metric, I could still see there being a niche for this.
From a pure VC perspective though, I wonder if they'd be better off Open Sourcing their model to get faster innovation + centralization like Llama has done. (Or Mistral with keeping some models private, some public.)
Use it as marketing, get your name out there, and have people use your API when they realize they don't want to deal with scaling AI compute themselves lol
> The linked page only compares to very old and very small models.
They're comparing against the fastest models. That's why smaller models are shown.
Sort of. The benchmarks showing Flash 2.5 doing really well are benchmarking its thinking mode, which is 4x more expensive than Mercury here
Is cost really the main differentiator here, tho? "Solving" coding seems like the holy grail atm (and I agree, it can enable a bunch of things once that's done) and "traditional, organic, human fed code" is pretty expensive atm, so does cost really matter now?
Put another way, how much would company x be willing to spend on "here's a repo, here are the tests, here is the speed now, make this faster while still passing all the tests". If it "solves" something in cudnn that makes it 10% faster, how much would nvidia pay for this? 1m$? 10m$?
Flash 2.5 without thinking mode is also exceptionally good fwiw.
this convo has me rethinking how much speed actually matters vs just getting stuff right - you think most problems are just about better habits or purely tooling upgrades at this point?
This is genius! There are tradeoffs between diffusion and neural network models in image generation so why not use diffusion models in text generation? Excited to see where this ends up and I wouldn't be surprised if we saw some of these types of models appear in the future updates to popular families like Llama or Qwen.
>Mercury is up to 10x faster than frontier speed-optimized LLMs. Our models run at over 1000 tokens/sec on NVIDIA H100s, a speed previously possible only using custom chips.
This means on custom chips (Cerebras, Graphcore, etc...) we might see 10k-100k tokens/sec? Amazing stuff!
Also of note, funny how text generation started w/ autoregression/tokens and diffusion seems to perform better, while image generation went the opposite way.
Nice
Can't use the Mercury name Sir. It's a bank!
Super happy to see something like this getting traction. As someone that is trying to reduce my carbon footprint sometimes I feel bad about asking any model to do something trivial. With something like that perhaps the guilt will lessen
If you live in the U.S., marginal electricity demand during the day is almost invariably met with solar or wind (solar typically runs at a huge surplus on sunny days). Go forth and AI in peace, marcyb5st.
Thanks! That helps somewhat. However, it feels like that's just part of the story.
If I remember correctly hyperscalers put their green agendas in stasis now that LLMs are around and that makes me believe that there is a CO2 cost associated.
Still, any improvement is a good news and if diffusion models replace autoregressive models we can invest that surplus in energy in something else useful for the environment.
This made me wonder - do any cloud compute systems have an option to time jobs or use physical resources geographically based on surplus power availability to minimise emissions?
I reckon it might incidentally happen if optimising for cost of power depending how correlated that is to carbon intensivity of power generation, which admittedly I haven't thought through.
To put this into perspective, driving for an hour in an electric car (15kW avg consumption) consumes about as much energy as 50,000 chatgpt queries [0] Running your laptop for an hour would be around 100 queries.
[0] https://epoch.ai/gradient-updates/how-much-energy-does-chatg...
Please see yesterday's https://simonwillison.net/2025/Apr/29/chatgpt-is-not-bad-for... instead of propagating the hand-wringing.