Hacker News

That sounds like a lot of fun, love the creativity. Do you have a repo you can share? It would be nice to run it, and if you wanted, maybe I could add it to the demos on https://playground.jeffyclassify.com

Ha. I did something similar but made a Jeopardy judge for my family to play jeo-paw-dy hosted by the family dog. It uses our phones as buzzers and speech to text to let people just speak the answer

ha i made one to play the Oregon trail

I think part of what made Jev catch on is that the API is like an if(...) or switch statement

People are just so used to the chat style APIs that they didn't even consider doing things like sending a bunch of emojis to a chat model and then asking for the optimal one in this context etc. Also chat models are pricier for the same behavior and can also output something random like a refusal

But yeah ironically I think in the initial breakthrough LLM paper on GPT-3 in 2020 some of the multiple choice questions were answered by comparing token probabilities of specific continuations rather than fill in the blank

> But then again plain LLMs have been perfectly cheap and serviceable as zero-shot classifiers for quite some time now, so again I'm back to my internal screaming.

It's the scale of "perfectly cheap", jev (specifically) is so dirt cheap and fast that you can throw it at things that should not be justifiable in the past and you barely have to do any work other then quick testing.

on the bright side, maybe this zero-shot classifier wave could be the back bone for more specialized classifiers (with more mindful selection of data and training)

What are some pre-Jev classifier and ranker approaches? And how do you train them?

Jev is classification for normies. Look ma, no training.

I want to share the rage. Can you expound on what makes you scream?

> But then again plain LLMs have been perfectly cheap and serviceable as zero-shot classifiers for quite some time now

I'm no prompt engineer, but I've found them to be too slow any time I wanted to use them that way. I have never tried Jev, but apparently it is supposed to be fast, so it seems like, according to the marketing, it could become usable where LLMs haven't been.

Maybe instead of screaming you can take this as a chance to level up your engineering.

Good engineers don't treat approach as A == B or even A like B, when extremely integral parts of their applications differ.

Zero-shot isn't just "more convenient", in a low data regime: it's the only workable solution, and 100x so if your plan involves the acornym "BERT" (because even the largest of those models has the world knowledge of a fart to draw priors from)

Better ergonomics while being faster and cheaper as the existing things really is enough to justify callling what you've done a new thing, in a world of finite resources and time. It's actually making me scream how many people don't get that.

> As an MLE

Yea but those models you were building were lame and inaccessible to play with for common devs.

Just because they have the same api doesnt mean you were building the same thing.

You can easily get under 100ms per decision in browser with a Qwen based decison model: https://alxnahas.github.io/strands-decider-web/?backend=engi... There is nothing about Qwen or Laya that make one or the other better at running in the browser. It is really just: how large is your model and how does KV cache expand as input length increases

Why would a Qwen 1.7b be too slow? You should get well under 200ms there.

Laya is just ModernBert fine-tuned. It's still a language model (just a bidirectional one). Let's call it what it is instead of this weird Decision Model mysticism.

decades ago you would train a classifier over a fixed set of classes. decision models are flexible regarding the possible outputs they can produce. also, you get text and images in input

If you think “bullet” is the cool new name for “arrow” then sure.

Decision models can perform zero-shot classification over almost unlimited text-input domains. This was science fiction decades ago.

The difference is in the training, or more accurately what data the model is trained on.

Cool project too. If you are looking for a 500MB instead of gigabytes, with evals on Jevbench that you can run fast on CPU check out gutsy.

https://github.com/kouhxp/gutsy

The point is there is no right answer until we want it to be accurate in the domain we're working in. Like a sporting goods service would definitely want the baseball answer.

This started as a fun experiment, but I now use Jev on my mac as an advanced auto-correct. Much better than any other option I tried. https://levmiseri.com/nospace (the no space being more of a gimmick, but the autocorrection is good)

I am working on a game https://imperiaquiz.com/en that needs over 20,000 questions. I use jev to classify 'has statement' and 'can be a question' paragraphs/chunks taken from cli script to reduce token usage. Meaning, an LLM only starts work once I've chunked text and marked it as 'to be reviewed' instead of parsing the full content. This reduces tokens usage at least 10x on average

I see Jev as a major step towards commoditizing current LLM paradigm. One thing would be to further optimize this particular route to work purely on CPU. This will grant an option to embed this feature into any application, from MS Office to games. The other is integrating this into agent workflow to vastly minimize token consumption.

Not in a serious manner but I created a testing harness for a Nintendo 3DS game I'm making that uses the OpenAI Decisions API. The main advantage is the speed (~2-300ms per input) which I really need for this purpose.

Not with Jev, but you can use classifiers for a lot of use cases, here’s a few: https://playground.jeffyclassify.com/

Also like to see a "layman harness" like LM Studio integrate an open decision model in its workflow.

use it to classify a bunch of research papers (by feeding it section by section, or summaries of sections if too long)

works like a charm

(not a product, so not much to share)

I use Jev in a Claude code hooks to detect dangerous commands.

if it works it works! as long as the test set is reasonably large and diverse its better than nothing. You could characterize how robust it is by throwing dozens of different types of work at it and see how much the confidence varies

https://en.wikipedia.org/wiki/Platt_scaling

The memory limit for Safari on iOS is less than 400 MB so it's probably because of that

Take ModernBert, add a fully connected layer with 255 outputs, take a bunch of classification datasets from Huggingface, write the code to have the datasets fit the jev format on these 255 outputs, do supervised fine tuning on the datasets with that format. then use a confidence loss of some kind

Indeed. Token masking just limits the costs associated with using an LLM. It therefore also limits the accuracy by limiting the amount of compute available.

But LLMs can mimick decision models, and I wouldn't be surprized if some labs are doing it this way at the moment.

it doesnt generate text though, only probabilities