I'm sort of surprised it took Jev for everyone to realize "oh yeah, this is a useful thing that we should have in our product portfolio".
But maybe this is one of those things that's only obvious in retrospect? I'm sure people were trying something similar years ago with grammar-constrained outputs (for example to only output a simple json structure), but you can't get probabilities out of the model that way. Which makes me wonder: how do these models calculate probabilities?
A lot of AI stuff is like that, actually. GPT 2 and 3 were mainly curiosities that would get articles on tech news sites but didn’t attract much attention from the general public. But after RLHF the UX improved dramatically and ChatGPT became mainstream. A lot of MLEs underestimate how much UX matters, and the same is true for Jev.
> how do these models calculate probabilities?
The short answer is: more and better data.
The long answer is: probably the same way any other classifier does. Take your ground truth probabilities and your predicted probabilities. The gap between those is called cross entropy, which is used to push the model towards the correct probabilities.
> which is used to push the model towards the correct probabilities.
The definition of "correct probability" here is usually "p=1 for the ground truth label, p=0 for all others." Which seems like an intuitive approach but if you actually think about it doesn't trivially imply the model would learn sensible distributions under uncertainty.
For a single prediction, sure. But the model doesn't make a single prediction and then stop existing. It makes an unbounded number of future predictions and under that assumption it is reasonable to expect the model to behave optimally under Bayes rule - which has a unique global minimizer at E[(Y-F(X))^2].
So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
That's how the original Transformer architecture (LaMDA) should have been released back in the day, but Google thought it would cannibalize their search revenue.
I think its more like how LLMs caught everyone by surprise. A reasonable person would not think such a simple system would have been this effective, and yet here we are. Distilling that system down even further into a decision model again does not seem like it should work. But it seems to.
The whole system is foreign to us, and makes us rethink what "intelligence" actually is.
OpenAI was already working on a Decisions API, and Jev launched theirs quickly enough to have 1 week in the spotlight before OpenAI put theirs in beta (which works great btw).
When trying it out, OpenAI's Decisions was ~2x slower, worse accuracy and more expensive. Unless used with image-related tasks, Jev is clearly superior.
We were doing zero and few-shot classification with LLMs since before they could reliably complete whole sentences. It’s an easier task than generating coherent text. The model needed to be tuned for classification (not generation) and for the domain, but it would often succeed at new classification tasks with a simple description.
I guess given the huge advances in LLM since then, these models probably achieve much better accuracy on a wide range of domains with lower effort compared to what we were doing, which makes it compelling as a product. And yeah, the models probably were already good enough a few years ago too. Other factors include how easy it is to release products at high speed now, and the desire in the industry to reduce token spend.
You can do it with LLMs by looking at the logprobs of ('yes', 'Yes', 'YES'...) vs. ('no'...). This works for bool or whatever they call it in jev space nool I think?
You can do it manually or use ready made libs. I know there's one in python on top of vllm that can do something like this.
LLM token probabilities are really poorly calibrated for decision making.
Every time I see someone saying something like "isn't this just structured output?", it's because they don't understand that probability calibration is one of the most important parts of the product, especially to understand uncertainty.
I think Jev did a pretty good job on calibration. I wonder how the fast-follow competition from OpenAI and Gemini do.
Not reliably - look at microsoft's guidance project that did some of this.
Once a model goes through instruction fine tuning and RLVR/RLHF the log probs decorrelate from what's ground truth (given the propensity of the data seen during training)
But I assume it wouldn't take much more training to make them good again - just a matter of picking the right loss function, since this sort of stuff is probably all encoded in the last layer or two anyway.
I've found that they work great as gates ("does this require a human review?") or filters ("is this suspicious?") and occasionally as judges (based on this context, is the expected pain score for this user above 7.5/10?).
Interesting how do you know if it does a good job at these tasks? It seems that thinking in LLM's has been one of the things that has been so important to making sure it's a good response.
I'm using them to label training data for model development. You can do something like label a bunch of records with Jev, then train the model, then check results with opus and a hold-out set.
Yes, literally built my own version of /goal in Claude/codex today for pi using jev. My agent was stopping so jev is called each turn to see if it’s done or not - similar to how it works in Claude, according to Claude…
We're about at the point where the AI is now pretty insanely cheap. Passing 100K tokens of context then asking hundreds of questions of 100 tokens each would cost about as much as a single Google search API call.
Out of the hundreds of questions, most of them could even be different variations of the same question to capture more nuance than what is normally possible in a single multiple-choice.
When I reviewed our internal AI workflows i was shocked at how often a decision API was a good fit or could streamline the process.
Being able to get a fast, high quality answer to the question "does this warrant interrupting a human?" or "does this look suspicious?" was a game changer for us.
I was also shocked to see that after a week of heavily experimenting I did not even burn through 5 USD!
The best decision model I have used (besides Jev itself) is respan/span-02, which is based on (finetuned from?) Gemma4 26A4. That makes me optimistic for this release, despite the high cost.
Image input is a primary reason I've never moved off my "home rolled" version of Jev (that Claude wrote for me over a year ago) for my file organizer/indexer. Takes images resizes them to 1k, then asks a handful of questions about the image for tagging purposes. It is able to process around 3 images a second based on Qwen 3 VL, but distilled and optimized for 1k images and up to 2048 token inputs (after the system prompt, then the questions are asked one after each other reusing the cache). Documents that are longer than that end up going through a summarizer that runs on my other GPU (Qwen 3.6, I think). And all of that gets fed into a vector search engine, for easy lookup later.
I'm not even sure what all is going on inside it anymore, because I just add new features when they popup, and tell Claude to do it, and it let's me know when it is done, and every new "input feature" can run against the entire catalogue of files.
Google ensuring “share of wallet” from their existing customer base with a solid but not great “we do it too” product, priced to capitalise on risk and change aversion.
That's always been the enterprise tech way. Using the product that comes with the platform the company is already signed up to pay for is easier than requesting a payment to a brand new service.
They don't need to be the best, they just need to not force a customer to go elsewhere.
What got me into Claude Code started with Google Antigravity. Somewhere along the way Antigravity took a dive. The rollout was great, then it started feeling like other G-products, the dreaded tail of lackluster developer energy after a peak.
We have Antigravity at work. It's become a lot better, especially since the Gemini 3.8 Flash and 4 Argon versions. The harnesses themselves (GUI and CLI) are updated quite regularly.
I don't really miss Claude when I'm using it for coding tasks. Also, it doesn't speak Claudish, which is a plus.
Startup / Founder focus vs Large Organizations with no vision.
With AI, we know distribution moat really doesn't matter and Google doesn't have a great Developer distribution moat anyway
Meanwhile, they are suffocating Jules today (https://killedbygoogle.com/) with a metaphorical pillow to the face and no migration path mapped out for casual users apart from "use antigravity" in a 1 line high banner; after about a year of existence.
It seems unlikely this will be around in 6 months at this rate.
That doesn't make any sense, this is a Jev competitor and a model release, like their Carbon and Argon models. It won't simply be 'killed' like apps might without something to succeed it.
And why not, exactly? OpenAI was more than fine killing Sora with no replacement. I don't see why Google would be any different, especially considering they're notorious for this behavior.
Gemini Decisions API is likely to be used by enterprise customers in real products. Sora was a toy that only seemed to exist to infringe on IP OpenAI didn’t control. Apples and oranges.
It took OpenAI 2 weeks after Jev to release a decision end point, and now google does it 2 months later. Why such a difference? Bureaucracy? Quality control?
I'm sort of surprised it took Jev for everyone to realize "oh yeah, this is a useful thing that we should have in our product portfolio".
But maybe this is one of those things that's only obvious in retrospect? I'm sure people were trying something similar years ago with grammar-constrained outputs (for example to only output a simple json structure), but you can't get probabilities out of the model that way. Which makes me wonder: how do these models calculate probabilities?
A lot of AI stuff is like that, actually. GPT 2 and 3 were mainly curiosities that would get articles on tech news sites but didn’t attract much attention from the general public. But after RLHF the UX improved dramatically and ChatGPT became mainstream. A lot of MLEs underestimate how much UX matters, and the same is true for Jev.
> how do these models calculate probabilities?
The short answer is: more and better data.
The long answer is: probably the same way any other classifier does. Take your ground truth probabilities and your predicted probabilities. The gap between those is called cross entropy, which is used to push the model towards the correct probabilities.
> which is used to push the model towards the correct probabilities.
The definition of "correct probability" here is usually "p=1 for the ground truth label, p=0 for all others." Which seems like an intuitive approach but if you actually think about it doesn't trivially imply the model would learn sensible distributions under uncertainty.
For a single prediction, sure. But the model doesn't make a single prediction and then stop existing. It makes an unbounded number of future predictions and under that assumption it is reasonable to expect the model to behave optimally under Bayes rule - which has a unique global minimizer at E[(Y-F(X))^2].
So counter to your claim that "It doesn't trivially imply that the model would learn sensible distributions under uncertainty" I think you need to consider what it means to optimally update a posterior under the global minimizer of the true conditional distribution.
That's how the original Transformer architecture (LaMDA) should have been released back in the day, but Google thought it would cannibalize their search revenue.
I think its more like how LLMs caught everyone by surprise. A reasonable person would not think such a simple system would have been this effective, and yet here we are. Distilling that system down even further into a decision model again does not seem like it should work. But it seems to.
The whole system is foreign to us, and makes us rethink what "intelligence" actually is.
OpenAI was already working on a Decisions API, and Jev launched theirs quickly enough to have 1 week in the spotlight before OpenAI put theirs in beta (which works great btw).
When trying it out, OpenAI's Decisions was ~2x slower, worse accuracy and more expensive. Unless used with image-related tasks, Jev is clearly superior.
There were classifier models on hugging face for years though…
We were doing zero and few-shot classification with LLMs since before they could reliably complete whole sentences. It’s an easier task than generating coherent text. The model needed to be tuned for classification (not generation) and for the domain, but it would often succeed at new classification tasks with a simple description.
I guess given the huge advances in LLM since then, these models probably achieve much better accuracy on a wide range of domains with lower effort compared to what we were doing, which makes it compelling as a product. And yeah, the models probably were already good enough a few years ago too. Other factors include how easy it is to release products at high speed now, and the desire in the industry to reduce token spend.
Zero-shot encoders even pre-date LLMs. People who worked in NLP in the transformers "BERT-era" may be familiar with natural language inference (NLI).
Decision models are really not much different than that, but (usually) scaled up to bigger causal models.
If you are looking to do Jev-like things with smaller models, check into NLI.
> I'm sort of surprised it took Jev for everyone to realize "oh yeah, this is a useful thing that we should have in our product portfolio".
Because it's especially parsimonious with token use.
You can do it with LLMs by looking at the logprobs of ('yes', 'Yes', 'YES'...) vs. ('no'...). This works for bool or whatever they call it in jev space nool I think?
You can do it manually or use ready made libs. I know there's one in python on top of vllm that can do something like this.
LLM token probabilities are really poorly calibrated for decision making.
Every time I see someone saying something like "isn't this just structured output?", it's because they don't understand that probability calibration is one of the most important parts of the product, especially to understand uncertainty.
I think Jev did a pretty good job on calibration. I wonder how the fast-follow competition from OpenAI and Gemini do.
Not reliably - look at microsoft's guidance project that did some of this.
Once a model goes through instruction fine tuning and RLVR/RLHF the log probs decorrelate from what's ground truth (given the propensity of the data seen during training)
But I assume it wouldn't take much more training to make them good again - just a matter of picking the right loss function, since this sort of stuff is probably all encoded in the last layer or two anyway.
I'm pretty sure these models also went through RL post-training, the question you ask still needs to be interpreted as an instruction.
[delayed]
I have still not used a decision model, How are people using them at the moment? is any one using them in a coding harness like Pi?
I've found that they work great as gates ("does this require a human review?") or filters ("is this suspicious?") and occasionally as judges (based on this context, is the expected pain score for this user above 7.5/10?).
Interesting how do you know if it does a good job at these tasks? It seems that thinking in LLM's has been one of the things that has been so important to making sure it's a good response.
I'm using them to label training data for model development. You can do something like label a bunch of records with Jev, then train the model, then check results with opus and a hold-out set.
Email filtering, ticket triage/categorization/etc.
It would be awesome to be able to create mail filters using natural-ish language instead of grammar
Yes, literally built my own version of /goal in Claude/codex today for pi using jev. My agent was stopping so jev is called each turn to see if it’s done or not - similar to how it works in Claude, according to Claude…
We're about at the point where the AI is now pretty insanely cheap. Passing 100K tokens of context then asking hundreds of questions of 100 tokens each would cost about as much as a single Google search API call.
Out of the hundreds of questions, most of them could even be different variations of the same question to capture more nuance than what is normally possible in a single multiple-choice.
When I reviewed our internal AI workflows i was shocked at how often a decision API was a good fit or could streamline the process.
Being able to get a fast, high quality answer to the question "does this warrant interrupting a human?" or "does this look suspicious?" was a game changer for us.
I was also shocked to see that after a week of heavily experimenting I did not even burn through 5 USD!
The best decision model I have used (besides Jev itself) is respan/span-02, which is based on (finetuned from?) Gemma4 26A4. That makes me optimistic for this release, despite the high cost.
Its bit more expensive than other other Decision Models in the same lane.
However, it can accept Audio and Video as input as well and for those specific cases it can be extremely useful.
Image input is a primary reason I've never moved off my "home rolled" version of Jev (that Claude wrote for me over a year ago) for my file organizer/indexer. Takes images resizes them to 1k, then asks a handful of questions about the image for tagging purposes. It is able to process around 3 images a second based on Qwen 3 VL, but distilled and optimized for 1k images and up to 2048 token inputs (after the system prompt, then the questions are asked one after each other reusing the cache). Documents that are longer than that end up going through a summarizer that runs on my other GPU (Qwen 3.6, I think). And all of that gets fed into a vector search engine, for easy lookup later.
I'm not even sure what all is going on inside it anymore, because I just add new features when they popup, and tell Claude to do it, and it let's me know when it is done, and every new "input feature" can run against the entire catalogue of files.
For anyone wanting something local, private, faster, which can also run and train on CPU, checkout https://http://jeffyclassify.com/
It comes with a bunch of pre trained classifiers, and you can use the tooling to train your own
There are also integrations with workflow tools like n8n
API docs: https://aistudio.google.com/docs/decisions-api
There’s also this guide by google to use embedding gemma 2 as a decision maker. Wonder how these compare.
https://developers.google.com/edge/mediapipe/solutions/decis...
Right. I also wonder how it compares to Strands Decisions with E4B or whatever.
Presuming this started when Jev was released... If so, thats pretty quick time to release for Google.
Anyone hear anything around when it might land in Vertex? As far as I can tell it's just in AI Studio for now.
Google shipped a "Decision Maker" API a while ago, see https://developers.google.com/edge/mediapipe/solutions/decis...
Apparently the two teams aren't talking to each other.
Google ensuring “share of wallet” from their existing customer base with a solid but not great “we do it too” product, priced to capitalise on risk and change aversion.
Standard practice by the big G.
That's always been the enterprise tech way. Using the product that comes with the platform the company is already signed up to pay for is easier than requesting a payment to a brand new service.
They don't need to be the best, they just need to not force a customer to go elsewhere.
What got me into Claude Code started with Google Antigravity. Somewhere along the way Antigravity took a dive. The rollout was great, then it started feeling like other G-products, the dreaded tail of lackluster developer energy after a peak.
We have Antigravity at work. It's become a lot better, especially since the Gemini 3.8 Flash and 4 Argon versions. The harnesses themselves (GUI and CLI) are updated quite regularly.
I don't really miss Claude when I'm using it for coding tasks. Also, it doesn't speak Claudish, which is a plus.
So is it over for Jev now that the big boys are in the game
It’s been a little while since I’ve looked, but when the first round of jev competitors came out, jev was still significantly faster.
Getting a response in 250ms opens up a lot possibilities over 750ms.
They just got a huge round, almost a billion in financing - that was after OAI Decisions API was released. I really wonder what their next move is.
Take the money and go home.
Startup / Founder focus vs Large Organizations with no vision. With AI, we know distribution moat really doesn't matter and Google doesn't have a great Developer distribution moat anyway
Surprised how close all these models are on the benchmarks
That would be just a sign that we need better benchmarks
Or that everyone is just copying each other
Why’s the probability type called “noul”?
Because they model this on Jev's API instead of OpenAI's version.
Was my question too. Apparently it is derived from Bernoulli (Can't find the source now).
Jev's founder mentioned the background on the latent space podcast.
https://youtu.be/cFx9Z3ZXca0?t=3340
Meanwhile, they are suffocating Jules today (https://killedbygoogle.com/) with a metaphorical pillow to the face and no migration path mapped out for casual users apart from "use antigravity" in a 1 line high banner; after about a year of existence.
It seems unlikely this will be around in 6 months at this rate.
That doesn't make any sense, this is a Jev competitor and a model release, like their Carbon and Argon models. It won't simply be 'killed' like apps might without something to succeed it.
And why not, exactly? OpenAI was more than fine killing Sora with no replacement. I don't see why Google would be any different, especially considering they're notorious for this behavior.
Because Jules was explicitly an experiment, not guaranteed to "graduate".
Google is maintaining the AI models and APIs for much longer, even if many people don't know about them unless they use them to build things
Gemini Decisions API is likely to be used by enterprise customers in real products. Sora was a toy that only seemed to exist to infringe on IP OpenAI didn’t control. Apples and oranges.
Antigravity is far superior to Jules.
I say plant a thousand flowers and cull the ones that don't bloom.
Barring Search, Gmail, and YouTube I am really wary of using anything from Google now.
The difference is this is not a consumer-focused product.
Google kills consumer products because most consumers don't pay for them directly, so they often end up not making enough money.
With enterprise products, they have actual paying customers, so the situation is different.
Looks like Google announced their own Jev inspired API and model
It was a hot topic 2 months ago.
I still don't really understand where and why these decisions or Jev is useful or interesting? Is there a clear demonstration or tldr anywhere?
Routing, classification, replacement for LLM-as-judge
It took OpenAI 2 weeks after Jev to release a decision end point, and now google does it 2 months later. Why such a difference? Bureaucracy? Quality control?
Looking at their benchmarks, it's actually a better model than OpenAI and Jev (which in the same evals is worst than Jev?).
Looking at anyone's benchmarks, they're all better than each other.
Why is that important to you? Seems a bit twitchy.