Hacker News

The whole point is the quantified output. If you just ask an LLM to type out its confidence "manually", it'll make up some nonsense. The logprob numbers are more reliable.

I got this technique to work extremely reliably last year. However there were a bunch of caveats: 1) Firstly, you must institute a check that the multiple choice tokens dominate the output distribution. They should sum to 95% or more, ideally 99%, or the LLM is not following instructions properly. This is also the problem with constrained decoding - if the LLM really doesn't want to output a valid answer, the one you extract will not be high quality. 2) You need to ask it multiple times, permuting which option corresponds to which letter, and average the results. LLMs are surprisingly biased towards picking "A", especially if they're otherwise not sure. 3) For the same reason, performance improves if you frame the prompt as if it were the middle of a quiz. "Question 1" carries baggage that "Question 12" doesn't. 4) You must be exceedingly careful with tokenization.

But when all was said and done, I got a general purpose A/B classifier that gave high resolution quantitative output for the cost of a couple dozen tokens ingested and a couple inference passes.

In my experience as well using logprobs to try to quantify uncertainty, LLMs are a poor fit. Neural nets in general struggle with 'calibration' --- ie. if a prediction is truly 50/50, neural nets are often prone to predicting overconfidently [0].

I ran some tests using GPT-4 to do some basic classification a couple years ago. On ambiguous options which had to be escalated to a human, the LLM would regularly output something like a 99.8% probability, compared to 99.99% for a correct answer.

0: https://arxiv.org/pdf/1706.04599

+1, llama.cpp has a --grammar parameter which you can pass a BNF style grammar file to constrain generation. It can be used in Python llama.cpp wrapper

https://til.simonwillison.net/llms/llama-cpp-python-grammars

If we're talking about running it locally, what about passing a partial response as part of the input?

Prompt part: "What is better, toast or bread?"

Incomplete answer part: "The answer to this question is "

and then have the LLM finish the answer. I did this with subtitle translation using llama.cpp (with Python) and had great success. Just past 5 already translated subtitles as the incomplete answer, and the LLM infallibly just continues to translate. No markdown, and usually no talkback if the subtitles contain nasty subjects like bioweapons or nuclear stuff. It just works.

Agreed, it's a real issue, but it can probably be vastly reduced by having the schema in the system prompt and by giving the model an expectation of a fixed value: no decent modern would pick a prose ligament over a provided value.

To completely squash the issue, a few cheap LoRa iterations will do the trick just fine.

Seems like all normal english words could risk the same, so would using short but random strings be even better?

Actually to me it sounds it could be benchmarked if this kind of effect exists in the first place.

"Reply with just the letter A, B, or C."

There, I fixed your problem.

A fundamental benefit of LLMs over Jev is that you can use test-time compute to improve the accuracy. Jev might eventually evolve to use test-time compute, but the formulation seems to more elusive to me than for LLMs.

Isn't that effectively the same as the blog post? You are just pushing the token filter to the sampling step.

That's the approach that daseinlabs/open-jev takes, in contrast to the above, which is what TheoLeeCJ/openjev and ekzhang/openjev-sglang do

https://sgnt.ai/p/jev/

Wow! TIL! I've been running a for loop around the two ordering variations to catch the winner of each turn and the difference is quite noticeable. In the options-after-body case in 47 of 100 attempts it classifies as phishing, whereas in the options-before-body case it classifies clearly as rickroll (94 out of 100 attempts)

Payroll sends you an email with a link to a Youtube video that plays a song.

Options after body:

    Average probabilities:
    Rickroll   0.5158 ( 51 wins)
    Phishing   0.4561 ( 47 wins)
    Spam       0.0281 (  2 wins)
    Joke       0.0000 (  0 wins)
    Legitimate 0.0000 (  0 wins)

    Average probabilities:
    Rickroll   0.9293 ( 94 wins)
    Joke       0.0549 (  5 wins)
    Phishing   0.0140 (  1 wins)
    Spam       0.0018 (  0 wins)
    Legitimate 0.0000 (  0 wins)

EDIT

Gemma4-12B-it-NVFP4 seems way less sensitive to option/body ordering:

Options after body:

    Average probabilities:
    Rickroll   0.9867 ( 99 wins)
    Phishing   0.0133 (  1 wins)
    Joke       0.0000 (  0 wins)
    Spam       0.0000 (  0 wins)
    Legitimate 0.0000 (  0 wins)

    Average probabilities:
    Rickroll   0.9401 ( 93 wins)
    Phishing   0.0336 (  3 wins)
    Spam       0.0250 (  4 wins)
    Joke       0.0010 (  0 wins)
    Legitimate 0.0002 (  0 wins)

What a time to be alive, repeating questions to a model twice to increase accuracy.

Prompt Repetition Improves Non-Reasoning LLMs: https://arxiv.org/abs/2512.14982

The 25 lines is the only thing that makes Jev different compared to Solution apparently.

>we didn't call an api

>calls an api

ok

latency and compute comparisons highly depends on your local setup.

you can swith to a better model for lower error rate.

Yeah it says it's a parody, but then in the same sentence it refers to the other "OpenJev" implementations, which are basically the same thing with marginally more effort. And it doesn't imply that those things are parodies too (and I don't think they are parodies).

Somehow the HN crowd has a bunch of "professionals" who don't care about error rates and think that a Qwen model running on a potato is frontier intelligence.

> nothing about its...

Non deterministic systems have furthered the "brain rot" in our industry.

Lots of people were happy to ignore the code in their "supply chain" before LLM's - but suddenly not reading the LLM's output is a problem. I get they are different but we're in the same realm.

The lack of real data on performance of what ever application that one is trying to pitch is getting appalling. It's a lot of "trust me bro" this works better hand waving. And it's getting gross.

And how do we even measure nondeterministic systems? Because if I told you that Anthropic was spending millions of dollars having 1000's of agents "pre solve" benchmarks to build into their next version of the system you would scream they were cheating. Every one is focused on the "hacking" in the hugging face incident and no one is looking why they were even playing with those benchmarks in the first place.

"Trust me Bro"...

after looking closer at the typesafe's jev, i want to point out that i misunderstood the significance of this jev model

it is the latency that makes it significant

What about the probabilities?

>they're trying to create that "System One" narrative

I think you answered your own question. Executives are going to ask two questions, 1) how is this different/why does it matter and 2) how will i use it to make money?

Leya came to market more than a year before Jev, and failed because nobody understood how to use it, and he was unable to market it properly. Jev used this strategy and did not fail.

> In real life, a human doesn't do classification tasks with the System One part of their brain, they use System Two.

I'd argue that most human classification is pre-conscious / System One. You see a table, you recognize it as a table without asking yourself "is this a table?"

I guess their marketing implies that it moves classification into system one response time.

I assumed they call it System One just because it's fast and there's no chain of thought/reasoning.

Either way it's an analogy that's bound to be loose as Kahneman's modes are about humans.

It's just marketing. More specifically, it's an answer for why their model can't answer questions that require reasoning.

> In real life, a human doesn't do classification tasks with the System One part of their brain, they use System Two.

Huh? I guess that depends on the exact definition of "classification", but I think the bulk of basic classification tasks we make every day to make sense of our surroundings, such as object recognition is definitely done using system 1. So is higher-level "stereotyping" or anything you could described with "I know it when I see it".

Because those responses can be incorrect or even harmful, you would sometimes make use of system 2 to correct them - but that doesn't change that the initial response is from system 1.

Oh, I thought it was some weird branding thing. So then I looked it up.

"System One" and "System Two" were coined in some pop science book...so back to its usage being a marketing ploy.

Uninformed hype for their startup. And they did a great job.

By design it can't be significantly slower than Jev: the prompt processing (AKA PP) is exactly the same on both and will take most of the time. Then you can process every single "question" in parallel, just predicting one or two tokens (if an answer is ambiguous with a single token) per each question, again in a single batch.

So, fast in the LLM space and comparable with Jev.

Off the top of my head it's 3 things it advertises:

- By not being a optimised for chat, it can deliver confidence for answer and not for how an answer should be phrased

- Speed. It can take seconds for OpenAI to compile schemas, jev can respond before openAI has even begun thinking

- Token efficiency and price. I think its the output token they don't even charge for because they are negligible, and the tokens they do charge for are at a fraction of a comparable model.

If you are using structured output, I think those 3 together is a really big deal.

>But their example is classification but that would also be possible and faster with a classic BERT model.

I believe the things you can classify with ChatGPT without any tuning or training is way beyond what BERT can do.

I think this discourse is still in the "figuring it out" phase. But here's where my thoughts are currently:

If you accept the premise that there are use cases where you might ask a frontier model a classification-shaped question and expect an ok enough answer, rather than creating a purpose specific classifier on some dataset that you have, then it follows that this is quite an inefficient thing to do, because you're doing extra work to turn the output tokens into a structured output and mostly throwing them away. So then if you could instead train a frontier level model that skips the output tokens and directly returns the structured classification information, that would be more efficient, and that's what jev seems to be.

But a lot rides on that initial premise of whether this is a use case that makes sense. But if you find yourself asking a model like Opus arbitrary yes/no questions and then maybe you switch to a faster and cheaper model because it's too slow and expensive, it seems like jev might be a great replacement for that.

You need to train data for a BERT-based classifier, and then there's a risk that it will pick up specific biases from the data instead of what you want.

As far as I understand, the idea of Jev is zero-shot or few-shot classifier: it learns a lot of stuff at pre-training, but unlike a classic LLM it doesn't need to learn how to chat, so it can be much smarter at a particular size

I am in no way trying to sell Jev here as some panacea of the modern world; I'm only responding to your questions:

> But their example is classification but that would also be possible and faster with a classic BERT model.

With BERT, you need a large, labeled dataset, and you have to train/fine-tune the model. Jev is pitched as a zero- or 'few-shot' model. You define the schema in code, give it instructions, and it works without a traditional training pipeline.

> So their pitch is a task specific smaller model or am I completely misunderstanding the whole thing?

Yup; that about sums it up: it is more or less an optimized, task-specific small model with the flexible understanding of a traditional LLM.

> is already fully solvable with structured output.

Not particularly. There is still the problem of hallucinations and varying results across runs.

That's more of what type-safety means for their team. Every run gives the same results. It's type-safe

One thing I would like to know is how fast it is when it's being presented with a 8000 ctx prompt? 16k? 32k?

no one knows but everyone pretends so go along with it.

Replace 'explain' with 'sell'. Don't forget that it's a gold rush. There's no reason to sympathize with corporations in their rush for the slice of the pie.

IT's the infamous "OneDrive in 10 lines of code (SFTP)"

While technically correct, it's not the same thing

Interesting. Have you repeated these experiments with recent models? I'm thinking frontier models APIs have tools/MCPs for math stuff but curious about recent Qwen models, etc.

If you believe the marketing, constraining the output this way can make the model much faster and much more type-safe (the model didn’t give you a fifth choice not present in the choices).

The argument the article makes is it's not that different. Jev's argument is that they have trained the model to better output probabilities (which is not necessarily a training object of LLMs but we don't actually know that)

Ultimately Jev claims to have a data advantage which is likely where the future lies. They'll have a unique edge in improving general purpose classification / decisioning.

I was thinking the same. Haven't tried it out.

You don’t that’s a python f string

name = “qurren” print(f”hello {name}”)

[dead]

not sure if true, but if you look at laya they use BERT type models. If jev is also using a BERT-type model it is autoregressive and therefore can't reason in the way that GPT-type models can. However, you get the advantage of being able to attend in both directions.

You beat me to it!

Is benchmarking Jev still a ToS violation?

It wasn't

> I guarantee existing labs will have this figured out in a matter of days.

Exactly.

Jev has no moat, and the incumbents will devour their lunch if Jev actually starts gaining traction.

127ms latency on 26B model, here you go: https://gambler-relay-us-west1.leo-fish.ts.net/demo

Running on old home hardware, Jev is probably running on a very powerful cluster.

How it's done: https://news.ycombinator.com/item?id=49813610

Nothing has malformed output if you coerce it's output into a statically defined set of options

> <200ms for 45 questions at once

Considering your own question length: ~120 characters x 45 divided by 4.1 ~= 1317 tokens.

So question processing at 5.5k PP(around the actual PP speed of GPT5.6 Sol) it would take around ~0.24 seconds + the context processing.

Computing the output should be around ~20ms (at 50 tok/s), computing 45 tokens in parallel.

> have 0% malformed output

Pretty trivial; only the allowed output is selectable :)

So, I keep repeating myself: Jev was a low-hanging fruit all along; no one cared, and probably no one will in a few weeks?

> note: this is a parody blog post

Should they be writing quicksort in assembly as a first step? I think its legitimate in this case given that Jev is likely using the same tools as the example. Showing how easily the core is created using those tools helps to dispel some of the mystery and hype.

Example why its legit:

I just invented a new "Regression Estimate Validator" aka Rev. It takes hundreds of input dimensions, then outputs an interpretable score. Its very fast and statistically robust. Response: Ok but you could just use `pytorch.nn.Linear(d_in, 1)`? True, it is equivalent, but that's concealing millions of lines of hand-tuned math libs, CUDA, python, and other stuff.

The fact that there are many lines of code underpinning the target functionality doesn't make it any harder to use, and doesn't increase the value of the sales pitch for the "new shiny thing" using those few lines of code.

However, I do sympathize with your frustration that people can just say "its 1 line of code" when that line is "invoke API" which is really millions of lines / databases, etc. as a way to dismiss legitimate work without understanding its implications.

I wonder if Qwen 3.0 0.6B q8 would have noticed

> note: this is a parody blog post