NitpickLawyer9 hours ago
> But I also think the demand for "fast/cheap/good-enough" models is just about to take off.
There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on.
Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do.
I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.
swatcoder8 hours ago
Yes.
The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words.
The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of restrained but efficient model+harness-tuples that have been distilled, finetuned, and rigged to deliver on narrowly scoped but idiosyncratically-shaped tasks with incredible efficiency and erogonomics.
theendisney2 hours ago
At work i only had early copilot which was hysterically bad at everything. As i wanted it to do the same task repeatedly and could spot wrong results instantly i kept evolving a prompt that attempted to correct all ways it found to do it wrong. It kept inventing new ways to get it wrong until it eventually got it right 90% of the time. My theory is that an avanced model that has no issues with a task could do the prompt enginering much better than i ever could. You could for example run x different queries that all do the same thing and compare the results y times. If there are >1 correct results and the wrong versions are all unique you should be able to drill down to a valid result with even a truly shit model running on a potato. Basically what humans do.
jimmaswell7 hours ago
This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always performed the best at all tasks. Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Recent comment touching on this in relation to LLM's in more depth: https://news.ycombinator.com/item?id=49322695#49323341
nickysielicki6 hours ago
The Bitter Lesson is very popular right now. It seems true right now. It’s having its moment right now. That doesn’t actually mean it’s axiomatically true.
Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LLMs could eventually outperform stockfish. It’s not true today and it remains to be seen whether it will ever be true. For now, specialized models are absolutely better at specialized tasks.
Evidlo6 hours ago
This seems really backwards. The Bitter Lesson is all about large data-based approaches vs hand-crafted ones, it doesn't say anything about language models not trained specifically for chess.
I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.
nickysielicki4 hours ago
The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.
nlan hour ago
> a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games
Not really, if anything it's closer to the opposite. The Bitter Lesson essay literally has this as an example:
> These researchers wanted methods based on human input to win and were disappointed when they did not.[1]
and
> Enormous initial efforts went into avoiding search by taking advantage of human knowledge, or of the special features of the game, but all those efforts proved irrelevant, or worse, once search was applied effectively at scale[1]
The actual bitter lesson is this:
> breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning. The eventual success is tinged with bitterness, and often incompletely digested, because it is success over a favored, human-centric approach.[1]
Applying to the "LLMs-for-chess" example the bitter lesson approach would be to put many, many more games into the LLM.
Does this work? People have trained fairly small LLMs that are competitive Stockfish at the ELO 1500-2000 level, eg: https://github.com/kinggongzilla/chess-bot-3000
This seems to be evidence that large LLMs probably don't have as much chess training data as Stockfish does.
[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html
camuel36 minutes ago
It's the exact opposite.
The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks.
Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson.
Computer chess is the canonical example of this.
inigyou4 minutes ago
Denser sampling only seems useful if the problem domain is in some way smooth - interpolatable. If you run it on a fractal problem domain you just learn more special cases. Chess is fractal.
klipt3 minutes ago
But the harness still matters.
In the case of stockfish, the harness is a tree search around the neural network evaluations.
iainmerrick3 hours ago
I think you have it backwards.
The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data.
Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than human intelligence. The LLM is based on a massive data corpus, sure, but the amount of data specifically about chess in there pales in comparison to just playing billions of games of chess.
TwelveEyes2 hours ago
Also, training it on chess books is literally training it on human knowledge, and not the actual game, which is exactly what the bitter lesson says not to do.
Dylan16807an hour ago
> I think you have it backwards.
> maybe if we use a blend of raw data and hand-crafted heuristics
I don't follow. They're suggesting giving raw chess data to the LLM, no heuristics involved.
antihipocratan hour ago
Maybe a future frontier LLM could approach the problem by first building its own stockfish, then applying the subsequent results
kmeisthax4 hours ago
The Bitter Lesson says that the only things that scale are search and learning.
Stockfish is the best chess search engine we've got, and you can learn some good heuristics for chess search policy that will make time-limited chess search a lot more powerful. That's perfectly in line with the Bitter Lesson.
In contrast, LLMs playing chess are relying solely on learned behavior. The inference harnesses surrounding them aren't designed to do chess things, they're designed to do autoregressive token decoding, which isn't a search process. Reasoning traces can resemble a search process, but they're far less efficient - the LLM would have to work out each legal move, test each one, calculate a score, and simulate minimax over all of that. Assuming the LLM is smart enough to even do all that.
A hand-crafted approach can absolutely beat data if your approach unlocks more search and/or learning than the general solution.
Dylan16807an hour ago
> A hand-crafted approach can absolutely beat data if your approach unlocks more search and/or learning than the general solution.
Now let's look at the bitter lesson again. It says that general methods that leverage computation are ultimately the most effective, and by a large margin.
That's different from just saying to leverage computation (which is how I would interpret "unlocks more search/learning"). If the lesson is "more computation wins, when sufficiently channeled" you're basically looking at a truism. Of course more computation beats less when it's used right. The bitter lesson is about abandoning specialization in order to get more computation, and while there's a couple ways where that helps with chess, there's a lot more ways where it's counterproductive. It looks like it's more true for Go than it is for chess, and that it's not universally true. It probably correlates with the state space.
mikepurvis4 hours ago
But isn't that really just about giving "front end" models more access to specialized tool libraries, which include models tuned to specific tasks? Like the first model says ah, we're being asked to code something, oh and we've been provided with some example code, let me invoke a tool call to my model the recognizes many languages, that model says that we're looking at ocaml. Okay, I better pass this off to my ocaml model which will decipher the supplied code and make a plan for what we do about the user's intent. The ocaml model recognizes that there are tests in the supplied code, let's have the special testing model have a look at the testing strategy and see how that fits in with what we just implemented, etc etc.
And perhaps at the end it all gets a single pass by a god-tier model for overall sanity and congruence, but the actual work, planning, coordination, and even user interaction was done by cheaper and faster agents of much more limited capability.
edotan hour ago
Yes, but an LLM will just call stockfish if it needs to play chess … sure if you arbitrarily constrain an LLM to use no tools it’ll suck at chess. But no one is using LLMs in isolation. Even consumer-grade, bone-stock ChatGPT has tools.
fragmedean hour ago
ChatGPT does not have stockfish as a tool it can call.
nlan hour ago
edot38 minutes ago
Yeah but it can just install it. It writes arbitrary code. It can do whatever you want it to do.
Animats5 hours ago
Good point.
Dumb AIs are needed for customer service. Most of that industry is still at "press 1 for sales, 2 for billing..." and needs something that will run locally on a 1U server.
cjkaminski4 hours ago
Yes, and the technology to improve the interface you described is already available to run hundreds of concurrent instances on a 1U server. The barrier to entry is getting the people who manage those systems to care enough to implement something better.
MrDrMcCoyan hour ago
Fact. My company's largest partner is CoreWeave, and convincing leadership that we could run it ourselves on partner discounted hardware for a lot less money has gone nowhere.
inigyoua minute ago
Maybe you need to walk into the office with a $1000 server running a hundred instances of whatever your code does.
[deleted]3 hours agocollapsed
PEe9bB7D6 hours ago
Maybe depends on how you ask it? Directly, or let it write a chess program? I think the latter can yield way better results.
srcreigh6 hours ago
No. The bitter lesson is about capabilities. GP is talking about efficiency.
GP isn’t suggesting that focused narrow model(s) will be more capable than large model, but that many small focused models can have sufficient capability while being more optimal.
Also, the bitter lesson is just wrong. The bitter lesson is about hand tuned AI vs computational general methods. However in truth today’s AI uses both. We have general compute heavy models which require narrow expert instructions (eg tools internet docs).
LLMs would not be as good without expertly written context, and expert context without LLMs aren’t as good either.
anon3738393 hours ago
> The bitter lesson is about hand tuned AI vs computational general methods. However in truth today’s AI uses both. We have general compute heavy models which require narrow expert instructions
The models are not even really trained bitter lesson-style anymore. That concept peaked during the era of pre-train scaling, back when it was thought that making a bigger and bigger GPT-3 would automatically solve all problems through prompting. In 2026, the most important part of training is post-training, which uses vast quantities of niche, hand-curated data to fit the models for specific tasks in domains like tax law.
ZainRiz6 hours ago
I'd respectfully push back on the framing here.
If you look at value as purely the LLM output, then there's a valid argument that the best frontier models will always be better than fine tuned specialists. (I'm not convinced personally, but it's a defensible claim)
But that misses two dimensions: 1. The cost of acquiring that output 2. What is actually "good enough" for that specialist domain
Not every output needs to be the best to produce value.
And as specialist models increase in cost, their cost/value proposition goes down.
At some point, there's a threshold where cheaper, fine tuned models are "good enough" at the task and also substantially cheaper than the expert models.
That's where fine tuning helps.
Personally, I became a believer in fine tuning after fine tuning a 1B Qwen model as a second pass over my local voice transcription app, achieving excellent accuracy at ~zero token cost and waaaay lower latency than if I'd invoked my Claude subscription under the hood.
DrewADesignan hour ago
At some point, the idea of cost/benefit analysis in the software business turned into the benefit analysis. The amount of money going into the frontier LLM model game is fantastically ridiculous. Being much better than the free resources doesn’t even touch how much better they will have to be to justify the expense of creating them, let alone continually maintaining these services.
joefourier5 hours ago
> It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always performed the best at all tasks. Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.
Absolutely false. At least when it comes to multimodal inputs, even a simple classifier will outperform the largest LLMs who still hallucinate details or don’t describe audio and images accurately.
And there’s also the issue of cost/inference speed. Running a trillion parameter model for all tasks will be incredibly costly, require a cloud API, while a tiny CNN can be run locally or at a cost multiple orders of magnitude lower.
applfanboysbgon7 hours ago
This idea has not failed to pan out at all. I work for a startup that is exactly what GP described, and am set for life because of how wildly successful it is. Notably, we are successful, in a genuine sense of the word: we bootstrapped from running tiny models to larger and larger models on our own slowly improving fleet of GPUs, and now have millions in revenue without a single dime of outside investment. Conversely, you cannot call taking on ~1 trillion in debt and purchase commitments to scale "success". OpenAI and Anthropic are underwater financially. To be precise, they're in the Mariana Trench.
wild_egg7 hours ago
Wait, you actually found a viable counter to The Bitter Lesson? Please say more
klipt7 hours ago
Perhaps an analogy to Moore's law?
Bitter lesson #1: don't waste time optimizing code when a faster processor is around the corner.
What countered it: Moore's law stopped working.
Bitter lesson #2 similarly relies on scaling laws that might have diminishing returns wrt model runtime vs intelligence. Runtime matters for turnaround on the problem you're solving.
wild_egg6 hours ago
Moore's Law has nothing to do with processors getting faster. Dennard scaling stopped working but Moore just slowed somewhat, not stopped.
bsderan hour ago
You are technically correct. The best kind of correct.
However, what most people think of as Moore's Law--CPU speed doubles every 18 months--broke somewhere between 90nm and 22nm.
And even the actual Moore's Law--2x the transistors every 18 months--doesn't hold for all types of chips anymore. Memory only gained 2x density over 10 years.
applfanboysbgon7 hours ago
This is a misunderstanding of either the bitter lesson or what was being claimed, on multiple accounts. Firstly, the bitter lesson is merely about human expertise-tuned algorithms vs. throwing raw compute at a domain. But, notably, it is still domain-specific. No matter how much compute you throw at training an LLM, it is never going to beat a Chess engine at Chess. If you give a Chess engine 1,000,000 compute units and a general-purpose LLM 1,000,000 compute units, the Chess engine is obviously superior at Chess; ergo, there is value in throwing compute units into training models for specific tasks. This is true for within several orders of magnitude of compute, in fact. It's also true that if you give the Chess engine 1000 compute units it'll still beat the all-purpose model with 1,000,000 units, so actually there's a lot of value in training for specific tasks.
Secondly, the bitter lesson is predicated on compute being cheap. There was a period where a hand-tuned algorithm informed by human expertise would outperform a raw alpha-beta search at Chess. Then compute got cheaper, and DeepBlue ascended to the top. Compute is now expensive again relative to the tasks being performed. We are absolutely still in a period where human expertise in training LLMs will outperform a naive approach with more raw compute.
CamperBob26 hours ago
I don't know much about chess engines; do they still use hand-tuned algorithms, or are they more like AlphaZero, where they learn through self-play to beat any/all possible human contenders? I don't believe DeepBlue was automated to that extent, but it may have been.
In the latter case, the chess example would tend to support the Bitter Lesson, rather than refute it.
I would also be VERY slow to claim that general-purpose models will never be competitive at chess. It wasn't so long ago that transformers couldn't add two-digit numbers reliably without resorting to tool use. They are now as good at "mental arithmetic" as any human savant. It wouldn't surprise me at all to see someone come up with a model that just happens to be really, really good at leveraging the portions of its general training data having to do with chess.
In fact you could argue that AGI demands such a model, if we are to assume that LLMs are a guidepost in that direction.
dmoy5 hours ago
I don't know anything about the last 8 years of chess engines, but yea maybe 8-10 years ago AlphaZero shit all over e.g. stockfish.
brucehoultan hour ago
I guess you missed Leela then.
And in 2020 Stockfish 12 adding some NN evaluation. And then in 2023 Stockfish 16 entirely removing the classical position evaluation code.
applfanboysbgon4 hours ago
DeepBlue beat Kasparov with essentially raw compute thrown at alpha-beta search. That does support the premise of the bitter lesson in general. But that does not mean the bitter lesson is correctly being applied here. The point is that even if throwing raw compute at a task is better than careful human-crafted algorithms, it's still task-dependent. The current trend with the people blowing hundreds of billions of dollars is developing an all-purpose model that is everything to everyone, but you don't need hundreds of billions of dollars to create a task-specific model that outperforms their model at a given task.
> I would also be VERY slow to claim that general-purpose models will never be competitive at chess.
This is not the claim. The claim is that for the same amount of compute, a general-purpose language model will never beat a Chess model. I'm dubious, but allow for the possibility that a language model could eventually compete at a top level against humans with enough compute. However, it will never compete with a dedicated Chess model with similar resources. Training a model for a specific task with the same amount of compute will outperform training a general-purpose model with the same amount of compute. This should be common sense, right? The bitter lesson was only about compute over human algorithms, not at throwing compute at a generalised domain over throwing compute at a specific domain.
You made arguments against two claims that I did not make (that I was trying to refute the bitter lesson or that I claimed that LLMs could never be competitive against humans at Chess), so I'd like to ask you read my statements a little more carefully this time.
joe_the_user3 hours ago
I think the gp found a viable counter to the ggp's version of the bitter lesson, which seems so extreme as to certainly exceptions. IE, they seem to say nothing but the latest frontier model is ultimately viable as an AI business.
The actual argument of the Bitter Lesson essay is pretty limited but people's interpretation of it has gradually drifted until it's seen as prediction that current LLM will reach AGI at a large enough scale.
HDThoreaun7 hours ago
The issue is that GP is misusing the bitter lesson. Yes, search + learn tends to be more effective than human rules based strategies, but that's not what's being considered here. The original claim is effectively that AGI isn't needed for most tasks and more value can be created by using search + learn to solve specific problems instead of applying general models to every problem. Then GP commented a non sequitur
shimman2 hours ago
Are you willing to say the same of the startup or are you still stealth?
z3t47 hours ago
Do you have a website?
CamperBob26 hours ago
Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.
VibeThinker 3B constitutes extraordinary evidence, IMO. The first such evidence I've seen myself. Very small model, very low literacy, almost no world knowledge, but it is as good at math and logical reasoning as models a hundred times larger.
The Bitter Lesson is a valid and trenchant observation about how about we got here, but I think it's a mistake to assume it tells us very much about where we're going. Too much has changed recently and is still doing so.
spockz6 hours ago
So theoretically, if you give that model the means to find information, ascertain the quality of said information, it could still reason its way to an proper answer?
Is this whole thing than maybe a read vs write optimisation again? Spent more time and effort training more knowledge into the model upfront and get it out in a single question instead of training a small model and needing more steps to answer the same question?
algo_trader6 hours ago
> VibeThinker 3B constitutes extraordinary evidence.. math and logical reasoning
Any similar model aimed at coding?
A >10B model for mass spawning/swarming and reporting back to a larger model
CamperBob25 hours ago
To some extent VT 3B is intended as a coding model (see https://old.reddit.com/r/LocalLLaMA/comments/1u7dzdr/scaling... ). It reportedly does well on leetcode-type problems, although I didn't check that myself.
I wouldn't use it for anything important without heavy supervision, as it's very weak outside its specialty. Not ideal for instruction-following tasks.
HoldOnAMinute7 hours ago
Someone will eventually figure out how to package it all into a single, cheap chip
bmitc6 hours ago
That you can then write text to program and make applications with.
apatheticonion3 hours ago
Agreed. I've been "guide coding" in my editor (Zed / VSCode) for a while now and it's really enjoyable.
DeepSeek v4 flash has been dirt cheap and so fast that my development loop is;
- small prompt
- review
- small prompt
- review
I build software with the same quality I normally would but it's way faster to produce and I think more about architecture and flows than I do about implementation details. The small diffs let me accept / modify / veto diffs and if the model struggles, I just write it by hand. It prevents compounding defects from leading the model astray (like you see in vibe coding).
In some cases vibe coding is useful, like when the complete specification is available (e.g. creating a JavaScript engine that implements the standard) - but anything that requires iterative development sees vibe coding break down pretty quickly (you could argue that is the case for a JavaScript engine).
I feel energised by AI assisted coding rather than drained, as it's a force multiplier for my skills and it lets me build more than I could by myself.
That said, most of my team vibe codes and reviewing their work is like pulling teeth.
ianmarcinkowskian hour ago
Basically how I feel, but I use the low-powered API models like Sonnet. I feel more energized and focused on the bigger picture than getting stuck on stupid implementation of micro-problems.
I think the muscle memory of doing those tiny problems is good for our minds, but solving larger-scale issues is also challenging.
I'm on vacation right now and getting claude to build a mostly-throwaway e2e testing harness (admittedly not small-prompt-review-repeat) for a backend API to speed up our existing e2e test suites which do click-ops to set up tests 8-10 years ago, we had a team who spent 3-4 months every year maintaining our E2E suite and people would do rotations on there to spread the knowledge.
I basically want an industry standard practice implemented on my team of 4 devs who are too busy doing other things.
jermaustin19 hours ago
To me, most local models work just fine for anything you can be patient for. If I want something quicker, I will go to a SOTA model via API, but with multiple 3090s, I have never really needed a hosted model for a lot of my experiments.
For code, they are great, but for creativity for NPC controllers, they leave something to be desired, but work well enough for testing, so I don't burn tokens until I'm actually playing my games.
But nothing one-shots a prototype better than Fable 5. I can have a prototype built in 30 minutes, hooked up to my local LLMs and Claude Code is very good at testing the interactions and even tuning the prompts of the NPCs for better experiences.
__float9 hours ago
"with multiple 3090s" is quite a bit of burying the lede for "most local models work just fine", don't you think?
jermaustin19 hours ago
Having multiple 6 year old cards doesn't seem like it's that big of burden for local LLMs.
I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060.
My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that price tag seem a lot better than it was, but it was an investment in all things GPU that were happening in 2020 (crypto, blender, image gen), then LLMs exploded.
thayne9 hours ago
A single, used 3090 costs more than I have ever spent on a computer.
9cb14c1ec08 hours ago
Yes, the tunnel vision around local models on this site is crazy. The percentage of people in the world who can afford the hardware is extremely low.
layer88 hours ago
It seems roughly similar to the pricing level of personal computers in the early eighties (i.e. IBM PC and Apple Macintosh). I’d expect prices to come down significantly over the next few years. Not so much in the next year or two, but after that.
oblio7 hours ago
Just like for warships, the complexity and cost of building cutting edge hardware has grown exponentially up to a point where a significant chunk of the world's computing is dependent on 2 companies: ASML, TSMC. We shouldn't extrapolate linearly from examples from the 80s.
layer85 hours ago
No, but I wouldn’t expect it to stagnate like with Intel in the 2010s either. Maybe the biggest caveat is that most people will be fine with using cloud providers, so the market for non-server hardware won’t be subject to as much competition.
regularfry6 hours ago
There's a non-small contingent who lucked into the periodic games machine upgrade at the right time to snag a {3,4,5}090 rig just before everything exploded. It's a small contingent now but it was less so then. And now those people can add a second card for roughly what that whole system would have cost new originally.
hparadiz6 hours ago
The current supply chain problems will eventually pass.
jermaustin17 hours ago
I don't think there is tunnel vision. I'm just saying that I have a couple 3090s I invested in a handful of years ago, and they are still going strong today as multiple GPU-needing technologies emerged.
I'm not saying everyone has to run local LLMs, because the APIs are in a race to the bottom, and my $10 of OpenRouter credits I bought months ago is down to $8.94 because most models give you MILLIONS of tokens for a US Quarter.
9cb14c1ec02 hours ago
> I'm just saying that I have a couple 3090s
This is tunnel vision. The percentage of people who could afford the hardware you could at the time you back it so vanishingly small. I do not know a single non-tech person who has multiple graphics cards in a single computer.
BoxOfRain7 hours ago
It's a decreasing pool as well I'd say, the dev machine I built last summer would make less financial sense to me now for example.
zahlman7 hours ago
I mean, I'm not rushing out to buy that kind of hardware myself, but it is a matter of perspective. People commonly spend an order of magnitude more on a car, and that's just the sticker price.
Karrot_Kream4 hours ago
I keep coming back to this: why do I need to run a local model on my own GPU? Open models can run in dedicated clouds and while, yeah, they may be more expensive per token than my own GPU, when accounting for depreciation, energy usage, and opportunity cost (money not spent on my GPU will instead sit in my portfolio appreciating with its particular blend of returns), I'm pretty sure I break even or even net lose money with a GPU.
Don't get me wrong, there are advantages to a fully local model in that, I can have agents looping 24/7 even when my internet is not working. But this is niche enough that if I had to price the advantages they don't seem worth it.
If I'm willing to pay the Openrouter tax, I can fire up Openrouter today and just get access to whatever model I want, and still pay a fraction for tokens as what I'm paying with the big guys.
wafflemaker8 hours ago
My single 3080 runs so hot I don't need to warm my room in winter, and have to play games in my underwear in summer.
throwaway2194504 hours ago
Unless you value privacy, pay for openrouter. You still get the benefits of cheap tokens and programmatic usage.
3090 pricing is something of a wild card. Since the only big-mem consume cards are the xx90s, and a 5090 is pushing $5000, resale value has gone way up. The bottom hit ~$700 last year. It's still a very good GPU, if power hungry.
zamadatix8 hours ago
I got a great deal on ~72 TB of NVMe right before storage prices shot up, doesn't make it any less ridiculous that I have it or any more relevant to people talking about building a NAS now. 99% of people, even in tech, do not have the stupid amounts of hardware people like us hobby on.
oceanplexian7 hours ago
Most people in the US have a car, and the average new car is $40,000. Hell where I live a middle class consumer will spend double that on a Boat or an RV and think nothing of it. These aren’t elite tech workers.
It’s not unfathomable that if a personal, generally intelligent local AI provides enough utility and doesn’t require you to tweak CLI flags millions of Americans would want one.
spockz5 hours ago
Spending that kind of moment on a product that gives you personal happiness for years up to decades and then will still have residual worth, which people save up for ages for, is an entirely different proposition than buying a product that may make you faster professionally, but which in the short time can also be achieved by a few dollars worth of subscriptions to a hosted model for even greater effect.
kevin_thibedeauan hour ago
> people save up for ages for
Americans by and large don't do that. Much of the population engages in discretionary spending with debt instruments. Combined with mass innumeracy, they're all oblivious to the true cost of their purchases because they only think of the monthly payment.
shimman2 hours ago
There are many payday loan operators and those willing to sell predatory loans to those workers you mention buying boats or RVs. I've yet to see a payday loan open up in SF to help tech workers buy hardware.
ninglor5 hours ago
Most people in the US don't drive a new car, and used cars can be had for far less than $40k. An $80k purchase would be just shy of the median annual household income -- anyone who thinks nothing of that has financial resources far above typical. You are in a bubble.
vel0city4 hours ago
An $80k purchase is far more affordable when you're looking at an 84 month loan. You trade in your current $20k truck with $30k in debt on it for your $80,000 car, get a couple grand in incentives and a $10k down payment, and boom you're only looking at a bit under $1,200/mo in payments. The median household is bringing home ~$84k before taxes, hypothetical person lives in a no income tax state, they take home ~$5k/mo. Easy peasy, its not like you were planning on taking any vacations anyway since you're always working.
What matters is you've got the Duramax HD King Ranch TRD Big-Boy machine. Doesn't matter the cost. You can tow anything, drive anywhere, do anything, and do it all in comfort. Other than parking in a normal parking spot comfortably. Or even park it in your own garage at home.
I've seen this exact scenario many times personally.
sroussey8 hours ago
where? i would love that.
zamadatix4 hours ago
"Where'd I buy it" or "where is it now" ;)?
It was a 96 core gen 4 epyc+supermicro board build with consumer NVMe drives on 1x16->4x4 "dumb" bifurcation cards. I had to get a few MCIO-> PCIe adapters as well to get the full lane coverage. Mounted in a standard EATX compatible consumer case with a consumer PSU and a lot of Noctua fans - surprisingly cool and quiet for what it is.
Motherboard+CPU I got from Ebay. Rest from the best MicroCenter/Amazon/Walmart deal of that day. Bought juuuuust before the AI pricing apocalypse, largely by pure chance.
xnx7 hours ago
> 2x 3090s
You could sell those and have enough money to pay for hosted inference for years.
jermaustin17 hours ago
They cost more to run than hosted anyway. But that isn't the point of having them. They are a playground, a backup when the internet is down, or claude is down. They can render Blender scenes pretty well. They play any game I want.
You can do each of those at various hosts and own nothing. Or own a couple "over priced" cards and do it all at home on battery power for a few hours while the power is out.
robotresearcher7 hours ago
For me it’s more that you can show them your financial and medical data without BigCo looking over your shoulder.
Gecko40727 hours ago
But after all those years you’d still have 2 3090s, which are now about 6 years old and still holding value.
irishcoffee7 hours ago
I keep seeing this comment. This is _hacker news_ where, back in the day, people just hacked on things, because it was a hobby. They weren't "moneymaxxing" or desperately trying to be as insanely efficient as possible. They hacked on stuff with a can of surge at 3am because it was fun.
Your comment is like a meta comment of "LLMs are generating everything, after a while the ouroboros will eat itself. (Which I agree with)" If people aren't hacking on this shit just because, you have completely conceded control of software to a handful of sociopaths, and open source software is dead.
vel0city4 hours ago
[dead]
bitexploder5 hours ago
Not really. 2 years ago that was a pretty normal amount of GPU hardware for a hacker or gamer. It's all relative. They are not accessible to most people yet, but for someone that cares and is a technologist? Likely accessible.
sroussey8 hours ago
I have trouble getting simple extraction to work sometimes. I have a block of text describing people and their roles at a company and their ages, and i asked for structured results of an array of these things with the text span that it appears in and all i can say is: nope.
quotescoreai3 hours ago
[flagged]
keeda6 hours ago
Yep, I've been having excellent experiences with the models even from the 2023 era. They required a lot of "holding it right" (mostly: being very precise in what went into the context) but their raw coding capabilities were astonishingly good even then.
However, back then I was getting the AI to write individual functions or classes or a test suite. I was decomposing the larger task into smaller tasks, delegating some of them to the AI, reviewing the results and composing the codebase from those. I was also essentially the harness.
Today the models can write and test and deploy an entire project. In terms of the code quality, I actually don't think today's frontier models would have written it much better than the 2023 models did. So in terms of raw coding capabilities i.e. converting a high-level specification into working code, I think we hit the peak way back in 2024 itself.
What has changed is the AI has learned how to do the task I was doing (besides being the "harness"!), which was the mid-to-higher level "engineering" aspects like decomposing a task, specifying it to a reasonable level, reviewing the outputs, and course correcting as needed.
I'm not sure if that is something the AI labs explicitly focused on during training (which may be why Meta is having its highly paid engineers do annotation work), or an emergent property of "better reasoning" (which I believe Dario implied in a podcast), or some mix of both.
But the fact remains that even the weaker models are more capable than we realize, and many being open weights, are here to stay.
mw88813 minutes ago
It does have to be said that if LLMs keep becoming better coders at some point the bottleneck on quality is prompting. Good ideas have many hidden assumptions you think are procedural but often are pivotal to your broader vision.
I find that when I give an LLM my full handcrafted codebase, it does very well. It follows my conventions, sees the intent and can coherently build within its scope. It writes much better code than a 'vibe' prompt.
It is always tempting and I myself will continue pushing the boundaries, but when you keep an LLM in reasonable scope (that may be one line, function, file at a time, depending on your idea of reasonable), you, by definition, can get sound utility out of them.
ksec8 hours ago
While they are improving rapidly, or as you say even if they don't. The next stage is for hardware companies ( cough Apple cough ) to ship these Local Model ready hardware in their products.
It will be interesting to track the improvements of these 7B model over time.
There will be a turning point in the next few years where it attract enough consumer attention to create yet another Smartphone and PC super cycle.
nowittyusername6 hours ago
There's A LOT low hanging fruit still out there for sure. And with antigenic systems being able to do the boring repetitive work of looking for that low hanging fruit I think we will see interesting things indeed. Also I think heuristics is where its at for such things. Once you describe some good heutistical structures for the research models to always follow related to "creativity" and such things, thats where we will see biggest difference. The agentic systems know the scientific method well and can follow it they just need the ability to be "creative" so their sampling becomes less rigid.
gozzoo4 hours ago
> We have, right now, access to things that 10-20 years ago would be considered magic
These things would be considered magic even 4 years ago!
eqmvii7 hours ago
I see it in a slightly opposite way: even the good models are relatively cheap, and so I worry what we might miss by spending too much time playing with the Sonnets of the world when the Opuses are still objectively a bargain for the power they bring.
zahlman7 hours ago
> when the Opuses are still objectively a bargain for the power they bring.
The cost isn't just what you're billed. There are security, privacy etc. concerns.
Foobar85687 hours ago
I know companies that are using github, even using public repo, and request their teams to not use SOTA models, but are ok with local models. Just stupid policy.
avadodin3 hours ago
If Orang mane bans GitHub they've got their local clones and can whip out a local server and a CI solution.
If Orang mane bans Claude, they've got their local models.
The latter has already happened too so I'd say their risk modeling is spot on.
riazrizvi8 hours ago
I think there's something subtle about language and ambiguity that means they aren't designed to become superintelligent autonomous machines. They're value is as information repositories that actual intelligent autonomous machines (us) mine and string together.
dgellow8 hours ago
Yes LLMs are a beautiful way to compact knowledge. It would be such a cool technology to develop and worked with if it wasn’t linked to such a toxic industry
riazrizvi7 hours ago
I think you're just observing ppl in one of these rare instances where enough of them come together because they are motivated. 'Toxic' is the clamoring sound of a crowded room where what gets through to your ears are just the most annoying snippets of incomplete conversations. I dare you to hang out with any actual people here, understand their viewpoint and listen to what they actually have to say in person, within the context of watching them do it.
dgellow5 hours ago
I know those people. Lots of them are fantastic humans. That doesn’t change the fact the AI industry is extremely toxic
LoveMistral9 hours ago
Same. Mistral 7b has been more than I ever needed for text for years now.
Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid.
Mistral 7b can do anything, and it’s basically instant even on an M3
frigidwalnut9 hours ago
Sounds interesting. Can you give more details on your workflow and what tasks you use it for?
LoveMistral8 hours ago
Code, creative writing, email summaries, automated email replies, and I prefill my invoice notes and daily updates for work.
Actually built a full invoicing product for that, using it too.
I use Mistral 7b and LlamaIndexTS on Node, I run it on a MacBook M3 and on a Linux server with only 8GB VRAM (old gaming PC).
Basically flawless, runs very fast and I don’t even know what paying for “tokens” is :)
jgthvxevbc8 hours ago
[flagged]
Barbing7 hours ago
This is how your comment displays on my screen, perhaps a typo:
>”@dang I really need an IP &/or account ban”
Almondsetat9 hours ago
What kind of work are you doing? For example, if I have some code in the hot path and I want to do all the usual tricks to help the compiler vectorize it, such a small model is not able to do much.
LoveMistral8 hours ago
RAG is your friend (or any vector db). No model can vectorize an entire codebase in context.
Even a big mainstream product (like Gemini) cannot handle more than ~1k lines without missing details and making mistakes. And about every 1k lines, it seems to forget the previous 1k, doesn’t it? So you can never hold more than a file or 2 (or 3) in context at a time without losing details.
What you find is that the big models like Gemini are doing vector storage and retrieval too, and breaking prompts down into chunks for various models to handle to assemble a thorough response.
If you want that kind of control in your outputs, and be able to hold a lot in your inputs, I don’t see any other way regardless of which model you use.
usef-5 hours ago
Out of interest, have you tried the newer models? You are not describing my experience recently.
LoveMistral4 hours ago
Yes - you are experiencing a mix of context caching and db retrieval from these mainstream model experiences.
Even the best models available lose a ton of detail over time if you were to paste in tens of thousands of lines of code.
The only way to hold huge amounts of context with a high degree of accuracy is to store it using various mechanisms (one of which is RAG).
On “effectiveness”, I mean end use case effectiveness in the tasks at hand, not whatever benchmark the model developer or vendor themselves come up with - which may or may not be useful to the work I’m doing.
casper148 hours ago
What are some limitations you have found with using a smaller model like that?
[deleted]9 hours agocollapsed
QuercusMax4 hours ago
Just being able to instantly generate a complicated query expression to pull specific bits out of a JSON blob sold me. It's awesome that I can ask Claude to build a whole feature and it will often one-shot it for me, but generating utility bash / python scripts or little throwaway utility webapps is what really excites me.
viscousviolin8 hours ago
If someone has an old GPU laying around, say a GTX 1080 with 8 GB of memory, would that be enough to get a (small?) local model running?
bityard5 hours ago
A small model, yes! But not necessarily a good model.
With the additional caveat that I don't know whether that specific card is supported by modern drivers.
You'd be looking at one in the 6B or 7B parameters range at FP8. Or smaller. It's been quite some time since a recognizable company in the AI space released a model that small. You can try larger model that has been quantized down to that size, but they don't always fare well with that.
Modern text-to-speech and speech-to-text models also fit well into modest amounts of VRAM.
avadodin3 hours ago
You're arguing for a very specific range of weights but many slightly smaller and slightly larger models have been released including QAT and MoE versions.
An old nVidia brand card with 8GB is more than enough to see those models running at usable speeds and accuracy.
Der_Einzige7 hours ago
BTW structured/constrained generation has so many places to trivially enable jailbreaking/alignment/safety problems that closed source models heavily limit the full expresivity of grammars and capabilities, particular of on-the-fly dynamic grammar construction/reconstruction.
dominotw7 hours ago
ppl keep talking about the supposed unexplored and untapped "model overhang" but very few things in the world are where you can write elaborate test criteria to before using ai.
A sales person sending a prospect email doesnt have a way to write a test harness for it. Yet these tasks dominate what humans do compared to writing a crud app . otherwise anthropic wouldnt have trillions dollar valuation
cyanydeez8 hours ago
I've amassed access to 4 different GPU rigs with 128GB to 72GB; I didn't this before I event touched an agentic engineering harness. It was sometime in February/March when I set them to first tackle small problems, and now with deer-flow, they're scaffolding full project/scope implementation and I'm finishing off the fine details around the problematic edges.
NickNaraghi9 hours ago
> Across his various startups, Peter has seen two kinds of work:
> 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of.
> 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts.
Interesting comp to pg's Maker's Schedule, Manager's Schedule https://www.paulgraham.com/makersschedule.html
I'm curious about not only which of these roles models will fill, but also how they will empower us to be in the mode we prefer.
kridsdale13 hours ago
I conceive of this as Protoss vs Zerg. I’ve had a lot of success in my career following a Zerg strategy.
michael0church9 hours ago
It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model.
There are many applications where world knowledge is unnecessary or even a negative, and in which only a small amount of language skill is necessary, and there we can expect small models more intelligently used to beat large ones naively used.
LPisGood9 hours ago
Small amounts of world knowledge seems like it would inherently be tied to more hallucinations.
TJTorola9 hours ago
Perhaps we'll get to a point where believing any un-sourced information from an LLM will feel crazy. I don't want my model to know more than it needs to perform logic and use tools. Once it is capable of using tools I would much rather it looked up information or sourced it from existing context rather than just divine it from it's weights.
pinkmuffinere3 hours ago
I empathize, and I have the same preference, but I wonder how this interacts with other people (many of them being our coworkers) using LLMs. There is no authoritative source for the models to pull info from, so either people will have to exercise good judgement and double check important claims, or they will trust too blindly and fall close to the level of whatever LLM they use. In that case, I prefer my coworkers to use an LLM that does have world-knowledge -- I will still hear them spout ridiculous claims, but at least it should be less frequent. It strikes me there's a sort of prisoners dilemma here, where if nobody trusts others to critically evaluate info, it's in our interest to make the tooling do it instead, to whatever degree that is possible. Maybe I'm too cynical about working with others though.
DennisP8 hours ago
Only if we require the knowledge to be built into the weights. Give it access to a search engine and a big library of ebooks, and it might do better.
jbstack4 hours ago
Doesn't matter if you aren't asking the type of questions where hallucinations are relevant e.g. you're seeking pure reasoning rather than factual information.
Zambyte9 hours ago
Probably. You can solve it with either some grounding context, or spending hundreds or thousands a month extra on a model that has more knowledge baked in. With modern harnesses, the choices is obvious.
giraffe_lady9 hours ago
Everyone wants this to be it but over and over we discover that the bigger a model is the better it is at all tasks, even ones far outside the domain it was optimized for. IE claude fable is better at writing both code and prose than smaller code- and prose-specific models.
The way vision and language models converge into the same geometric space should be extremely alarming for the "you don't need global knowledge for local tasks" type dreams.
And to be clear I'm not saying that smaller models don't or can't work well, or that we shouldn't be heading in this direction. And it's not quite the case that broad knowledge is strictly necessary. But it never seems to be negative! And so far it is the best way we've found to do... everything. Small models are good to the extent they are like big models, not to the extent that they are small.
janalsncm6 hours ago
On narrow domains, it is very common for small models to match or outperform larger ones at a fraction of the parameter count.
For example in language, this is called the “curse of multilinguality”. Small models that handle a single translation direction can easily outperform big ones that try to handle them all.
https://arxiv.org/pdf/2311.09205
In any case, for most tasks the question is not “how many tasks can this model kind of do well” but “given time/cost constraints, what is the maximum level of quality we can achieve”. And for that, small models are usually very competitive.
wredcoll8 hours ago
I think the context here is that small models run locally, not rented from a cloud.
giraffe_lady7 hours ago
Yes small models are and will be useful for lots of stuff for several reasons.
But the idea they’d be better than a bigger model is cope, you’re pretty much always better off running the biggest one you can bring to bear within your constraints.
michael0church2 hours ago
Fable 5 is actually a lousy writer. Opus 4.6 is the best for writing and prose assessment. Gemini 3 is smarter at reading comprehension but tends to be more unstable in judgement.
cpill6 hours ago
yeah, I think they will get smaller so they can be run everywhere, and really just be an interface to various non AI systems.
cyanydeez8 hours ago
[flagged]
andsoitis32 minutes ago
> One thing a few investors I've talked with have mentioned: "It's weird we're not seeing more consumer AI companies. Why is that?"
What would consumer AI company even be? The frontier labs have declared they will eat everything and they have a head start.
Best bet would to be a contrarian and build products and services that people actually want or need. Fine to be AI powered or augmented, but consumer companies do the hard part of understanding specific consumer needs and wants and pursuing that.
swiftcoder10 hours ago
I find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back
SomeonesAccount10 hours ago
Exactly! Composer 2/2.5 were amazing, cheap, and fast. Everyone else was Gaga about GPT 5.5 and such, while we were over here doing the work with less cost and more speed
sickcodebruh4 hours ago
Composer 2.5 is phenomenal for so many tasks!
jbjbjbjb9 hours ago
I’ve been playing around with Luna, Terra and Sol and for the type of work I’ve been doing lately I actually think Sol is just a likely to trip up as Luna. Examples were Sol over assuming, persisting in the wrong direction, over engineering a little script to do some exploration of api. They can all be fixed but it’s a waste of tokens, I rather have Luna do it because course correction on small pieces of work is cheaper.
scoring17749 hours ago
I've found the distinction to be in how much I care about how the final product looks. If I want high-quality code I typically find a smaller model with a well-designed spec to do better, if I want it to just run and produce something close to my vague description typically Sol does better. For most actual business use-cases I think the first is likely better but the experimentation speed up with the frontier is very nice.
ZeWaka5 hours ago
I've found Luna to be quite capable.
kccqzy9 hours ago
> for most tasks
The word “most” is doing a lot of work here. On a percentage basis perhaps most tasks a typical SWE needs to do when they aren’t in meetings or writing docs are just glorified autocomplete. But that’s boring and that’s why people don’t usually talk about it.
People are addicted to chasing frontier models because they all have memories of spending a week on a deeply challenging algorithm problem or even have crazy complicated algorithms they cannot implement themselves and want to have the models achieve this technical breakthrough. It’s the kind of productivity boost from spending one week on a problem to spending one hour. In contrast the productivity boost from spending ten minutes to spending one minute just doesn’t occupy people’s mind.
swiftcoder7 hours ago
> crazy complicated algorithms they cannot implement themselves
I'm not sure I know very many engineers who would fall in this bucket. Or do you mean the business types who suddenly think AI can replace all the engineers?
kccqzy7 hours ago
It probably depends on the background and the company. For example if one works at a startup that happens to use technology, it’s unlikely to happen because SWEs just translate business rules to code. But if one works at the place where the technology itself is the focus, then yes most people will fall in that bucket.
In fact I noticed that this is the one place where people discussing AI on HN tend to talk past each other. On the one hand people are talking about supreme intelligence like designing new algorithms (on the same vein as finding counter examples for the Jacobian conjecture) and on the other hand people are just satisfied using AI to automate a few quotidian tasks that hitherto couldn’t be automated.
jlkuester710 hours ago
Exactly. Even 32b parameter models you can run locally on consumer hardware are "good enough" at this point for some workflows!
dominotw7 hours ago
no they are not good enough for "most" tasks
teiferer5 hours ago
A friend of mine told me earlier today that they had a discussion at work (a coding shop) about "downgrading" to luna from sol for cost reasons and that many were quite unhappy about this because they didn't want inferior tech to be forced upon them. Do they have a point? Is sol actually worth the extra cost? Especially if you ramp up the effort level?
bunderbunder4 hours ago
I don’t love the “forced upon them” framing; if that’s really how people are thinking about it then maybe they should pause and reflect for a moment yhat it isn’t their money being spent. Amd the default isn’t always having the latest and greatest, it’s not paying for anything at all.
Now, if the debate is really about which option is more cost effective, then we could easily run an A/B test to find out. Though TBH my instinct is that that experiment is likely to cost more than the potential cost savings.
What I will say is that my own sense from experimenting around in a non-rigorous way is that the answer depends on how you use the tool. For actual vibecoding you should always go for the SOTA model because it will need less oversight. It’s also less likely to get stuck in a vicious loop that fruitlessly wastes tokens. But for a more hands-on approach where you move in small, carefully planned increments that you review and test in human-comprehensible chunks, smaller models may be preferable. SOTA ones don’t do that much better when working that way, and the slower inference adds a detrimental amount of friction to the work cycle.
pseudosavant5 hours ago
It is a good question. Luna is definitely a very capable model. Much more capable than the top SOTA models from 12 months ago. It definitely isn't at the same level as Sol, but you get 20x the tokens for the cost, and it has a much faster tokens/second rate.
If this is a cost conscious company where I'm going to get a fairly limited amount of Sol, or a nearly unlimited amount of Luna, I'm probably choosing Luna.
shepherdjerredan hour ago
I choose to use Luna for most tasks because it is cost efficient, even though I get a pretty generous budget from my company.
Sometimes I will use Fable or Sol for large features/projects, or research/exploration.
I would not be at all happy if I were forced to use Luna, though. I’d probably start looking to leave. I don’t want to work somewhere where I don’t have choice over my tools.
usef-5 hours ago
Luna as a doer, with a smarter model planning, can be a good compromise. Using sol for everything can be expensive without much gain, as a lot of steps don't need that sort of intelligence.
praveer134 hours ago
Luna is great at doing targeted smaller work, I use sol max for creating a plan and targeted /goal prompts after I finalize the design. Or Claude with ultracode for design and planning and adversarial review by sol max and then delegate to Luna for smaller goal prompts
azuanrb4 hours ago
I’m building an internal tool for our company, basically an agent to help with on-call and alerts via Slack. I have evals running across a few scenarios, and my favorite models so far are Sol medium and Luna xhigh.
Sol medium has been a nice balance between intelligence and response time. Luna xhigh can achieve similar scores on the evals, but it takes noticeably longer. My impression is that the higher reasoning effort helps compensate for the lower base intelligence.
Cost is definitely a big factor, but latency and intelligence matter too. If I had the budget, I’d take Sol medium over Luna xhigh.
From using both on real scenarios, Sol is noticeably better at navigating around issues, exploring alternatives, and being creative when the obvious approach doesn’t work. That matters quite a bit when you’re investigating live alerts, where the path to the root cause isn’t always straightforward.
brikym4 hours ago
It's silly to discuss it. Just do the evals.
kingstnap4 hours ago
Smaller models + more effort has strong diminishing returns, especially if your goal is to save money.
Sol already lacks judgement. It will absolutely add idiotic tests and comments. Luna is that but worse so if you account for things like going down wrong paths, producing bad results, overthinking then it could easily cost you more to get less.
dude2507115 hours ago
Yes. If they don't like the cost then they should fire the "leader" who introduced the AI there to begin with.
noodletheworld3 hours ago
Is sol better?
Yes. Categorically. Anyone who tells you otherwise and that luna is “just as good” does not know what they are talking about.
Going from sol to luna is a downgrade.
It is not a question, it is a fact.
> Is sol actually worth the extra cost?
Is a question only you can answer, because it has no generic answer.
Right now, for me, being able to use sol is worth the cost, but using it all the time is not.
I’m sure going from using it to using luna feels rubbish; but there are realities about costs you have to face sooner or later.
Maybe like… give your team credits and make them pick the right tool for the job; and if they burn their credits on sol in 20 minutes, well, tough luck buddy, looks like you're coding by hand for the rest of the month.
Team will quickly shift. People hate losing access to ai.
vatsachakan hour ago
Idk... Luna is great if you generate specs before implementation.
Sure a Lexus is better than a used Prius, until you include price
throwaway634679 hours ago
I’m kind of cautiously excited for the next five to ten years, with these AI chips becoming incredibly fast and RAM capacities ramping up its in the cards that we’ll have chips like today’s ATMEL microprocessors that fit on a single board computer and can run small models locally, then all our gizmos can have local AI and I can have a truly intelligent home. Of course there will be a huge push to put all of it in the cloud but maybe we have a chance to take this technology home for good as it’s hard to imagine people will submit to this kind of surveillance required for AI home automation 24/7 (then again I might be wrong). Exciting times.
mathgenius5 hours ago
Can we stick one of these in something that looks and sounds like HAL 9000 ?
verdverm4 hours ago
with wireless tech, you can embody Ai in just about anything, waiting for that hacker post about turning their toaster into a thinking machine, I have a Anki vector I've been meaning to do this with (has camera, speakers, microphone, and screen built in)
dgunay3 hours ago
Luna max is suitable for like 90% of the kinds of code changes I want to make. I only find myself actually reaching for a Sol or Fable tier model if the problem is very complex. If you're willing to build the guardrails and do some extra planning, Luna is very capable.
a13n3 hours ago
Regarding the Pareto frontier and related benchmarks, I have a hard time taking anything seriously that claims that Opus is anywhere near the intelligence of Fable. Are there any benchmarks that haven't just been benchmaxxed that more accurately represent actual usage?
kakugawa2 hours ago
FrontierCode is prob the closest. [1] It's closed source (so no direct benchmaxxing), and it was calibrated by 20+ open source maintainers. It shows Opus 5 (medium), beating out the other reasoning levels by a large margin. i.e. Opus 5 w/ higher reasoning levels actually reduces performance. [2]
However, you'll have to gauge for yourself how closely their tasks resemble your tasks.
glimshe10 hours ago
> There's obviously a lot we can optimize here, but if you're charging what the WSJ or The Economist charges, you'd better be delivering similar value.
Gosh, watching paint dry has been a better value than reading The Economist in the last 5 years or so.
That aside, I had good results with Luna. I'd be interested in hearing about a comparison that takes into consideration response time (not TPS), cost and performance of the popular models at different settings. That chart has some of that. For instance, is Luna Max a better value than Terra Medium?
yousif_12312310 hours ago
[flagged]
[deleted]7 hours agocollapsed
[deleted]9 hours agocollapsed
my-next-account8 hours ago
Are ya kidding me
2001zhaozhao6 hours ago
A dream of mine is to be able to host a LLM-powered video game that I can host on a home server running a decent mid-range GPU like the RTX 5060, and the LLM is fast and intelligent enough to make for a fun game experience for a few dozen concurrent players. People can ask for features and they just get made and added to the game on the fly for the lobby to enjoy. The hosting costs would be manageable enough that I don't have to charge anything for the game.
I think with one more year or so of small model progress, that might just be possible to accomplish.
civvv4 hours ago
Lol
yipinwong8 hours ago
"Small models" nowadays work like someone who has IQ 100+ while SOTA ones are like 150, "relatively".
Given sheer number of turns I can make with small models, I can do a lotta stufff
- cheaper, and faster
Harness makes differences: There have been many HN posts about how one made tiny models work better at certain tasks using harnesses.
These "small" models with right context, and guidance, they work wonders.
---
I've been saying Luna has been my go-to AI in previous comments and why Luna is still more compelling than GLM-5.3-flash.
anuptalwalkar4 hours ago
I kind of agree with your assessment. Running models not just on local, but cheap and lightweight frameworks will drive the next phase.
Not trying to plug, but I do't know any other way. I wrote a piece couple of days ago on small models and memory usage on the edge devices- https://polign.com/blog-edge-agent-memory and https://news.ycombinator.com/item?id=49450816 closing on the same problem.
zmmmmm40 minutes ago
The "good enough" concept is interesting because of how systematically people over estimate it. So often, things that are lower quality but thought to be "good enough" turn out to be either not good enough or not worth it compared to just using the higher quality "thing".
I will believe that smaller models have hit that bar empirically when I see them in production. At the moment, even frontier models are stuck in most of the scenarios I am seeing for high value tasks at the "not good enough" gate - so small models are not even close to being on the scene there yet.
dev_awesome20 minutes ago
how effective are the small model?
pranav_tech268 hours ago
Running small models locally beats wrestling with API latencies and rate limits. The compute trade-off is 100% worth the privacy and DX gains.
bartleeanderson5 hours ago
Nobody is questioning what is meant by small? When you started mentioning frontier models that don't run locally, I just go TB;DR "Too big, didn't read"
weinzierl9 hours ago
Small is relative. I'm looking for models that I can with run around 100 MiB mark (RAM just for the weights) to demo what you can do with this little memory.
I know of SmolLM 2 which in Q4 is borderline regarding the size and rather dated. There is also TinyStories, which is also old and also focussed on children's stories.
Is there anything newer in this category? Or should I try to distill something down to this size?
laruss58 hours ago
[dead]
nullbio4 hours ago
The future is self-hosted models trained on your own reasoning traces autonomously, as you sleep, using QLoRA and whatever else.
1saadcodes3 hours ago
The cost difference is pretty dang nice. Going from roughly a dollar to $0.10 for the same kind of task makes it so that products that didn't make financial sense before become possible
highfrequency8 hours ago
> the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts... ~95% of the work he does falls into bucket 2. It's hopping on calls. Nudging people. Blocking and tackling.
This is a good insight broadly!
caust1c9 hours ago
IMO big models are not a product in and of themselves. Inference is just a new type of compute. I'm confident that in two or three years, every product will have inference capabilities integrated into the experience, and models will become less and less distinctive from one another.
What most products need from a model is a pretty short list: the ability to make tool calls well, accurate recall, and the ability to follow directions without wavering (whether or not those directions are baked into the weights or provided in a system prompt). That covers 95% of inference utility in products.
We're nearly there, and I believe these capabilities will fit on small models.
Because of this though, I predict hardware demand will stay high despite demand for "hosted" inference dropping. Unless there's some regulatory shenanigans that step in to say otherwise.
low_tech_punk9 hours ago
The tokens per second speed measurement is highly inflated nowadays because most of the tokens went into thinking. I wonder if there is a more realistic measurement for "effective speed", which accounts for thinking efficiency.
ak_t9 hours ago
Many benchmarks now measure the total cost or energy usage per completed task.
[deleted]6 hours agocollapsed
wxw9 hours ago
100% agreed. Small, cheap, and hosted models. Luna (and open weight models and others) is ridiculously cheap @ $0.2/$1.2, easily accessible, and more than good enough for basic use cases (e.g. summarization, simple tool calling, etc.).
marius_6 hours ago
I wouldn't call $0.2/$1.2 "ridiculously cheap"
embedding-shape8 hours ago
I love how "Small Models" apparently is "Model of unknown size but probably smaller than another model that we also don't know the size of".
zatkin9 hours ago
Maybe I'm being super reductive here, but operating small models at the core of your business kind of moves the needle from making external API calls (against frontier models) to running internal API calls (against your locally-run models). It seems like if we want local models to take off, it will need to become easier to run local models for cheap. I'm thinking like reducing the barrier of entry for running "local models" in the cloud providers like DigitalOcean, AWS, etc.
malfist9 hours ago
You should be glad to know digital ocean already offers this
regularfry6 hours ago
In theory so does AWS, but the Bedrock model selection is badly in need of a refresh.
spl7579 hours ago
I only run local models and I don't give them access to much externally. I don't do anything serious with it, but it comes in handy and I know that they can do so much more. I'm on a meager RTX 3060 12GB and a GTX 1660 Ti with 6GB for some extra vram space. When I first started playing with local models, I was really impressed with what I was able to achieve locally.
That's great, but the thing that worries me is that many companies have billions invested in the AI bubble. It's around 1.5 trillion last time I looked. It's all circular spending between the companies building out the infrastructure, and the models. None of it is profitable. They will want to recoup that 1.5 trillion from consumers, which means using online-only pay-as-you-go cloud models. They will inevitably see that people using capable local AI are "lost customers" and they will try to kill the ability to locally host AI or somehow enshitify it enough to make paying a subscription more palatable.
I'm not saying I believe that will happen, I'm just worried that it will. Is anyone else worried about that as well?
mlnj8 hours ago
I am very excited that more makers will come up with fast memory for consumers rather than enterprise. Companies can only pre order so much RAM.
At some point there will be a surplus of fast memory and even in a crash the current generation of SLMs are bounced to be plenty to build a lot of intelligence at home.
mattmaroon9 hours ago
The demand for fast, cheap, good enough models has always been borderline infinite, it’s the supply that’s going to take off.
ittsel6 hours ago
Watch reasoning tokens though. We tried a small reasoning model that burned ~2800 thinking tokens per call, 3x the cost of a cheaper non-reasoning one despite a better price sheet.
jmtulloss9 hours ago
I forked my Big Serious Harness™ that models construction projects into a harness for building a vibe coded family assistant. I couldn't figure out how to make the toy operate at toy prices until Luna. Now you can vibe code all the little apps you might want for your fam for like $5 and operate it day to day for a few cents.
possibilistic9 hours ago
> Peter runs multiple companies. Beyond Segment, he's raised $100m+ for Charm Industrial, and just recently closed a Series A for Revoy. He's incredibly organized and efficient with his time.
You can do this before an exit? Build and fundraise for multiple (3?) companies at the same time?
zachthewf8 hours ago
Segment had a $3B+ exit to Twilio back in 2020.
caruasdo3 hours ago
That's why the market has our solutions but economics.
fitsumbelay5 hours ago
using small local models - with a little bit of extra work - for the first time over the past few days was _really_ illuminating and inspired similar thoughts about how far you can practically get with so little. column of zap emojis, mane ...
verdverm4 hours ago
I only use open models now, I really think the era of open models is upon us, big or small, but I also agree small models reached the point where you don't have hand hold them with qwen 3.8 27B
oybng7 hours ago
An absolutely nothing post at #2 on the frontpage
toshop10 hours ago
I think we'll see more of this soon
replit is already leading the way with free luna usage
Zigurd7 hours ago
I recently had some relevant experience: for a couple of months now I've been experimenting with on device models to summarize feeds in a Bluesky client I am developing. The feature extracts topic areas, categorizes posts, and creates a summary under each topic.
At first the results were hot garbage, and progress was slow. I hooked up the settings to download models from Hugging Face conveniently, so I could run experiments faster, and I massaged the prompts a bit. Last week this feature made a qualitative jump from science experiment to something I'd actually use.
The fact that all runs on the device means I've got no variable costs associated with adding this to what will be, at best, a pretty low revenue product. I've tested it on trailing edge devices like an M1 Mac and a Pixel 8, and performance is very tolerable.
The key is I'm not asking for open ended answers to open ended problems. When it proves to be useful it's not going to get less useful or more expensive.
There are vast domains of uses for LLM models with similar characteristics and likely similar results.
dzonga8 hours ago
small models + a good application layer - are more than enough, good for routine business tasks.
the application Layer i.e having a good graph RAG & connecting it up together is the missing piece for most.
sroerick8 hours ago
Can you elaborate on this?
lantry7 hours ago
The model doesn't have to be smart if all it's doing is pushing a few different buttons.
I don't have to be an automotive engineer to start my car and put it in drive.
hartator9 hours ago
I have trouble seeing the points of using less capable models.
I just want the smartest, best, and most capable models. It feels smaller models for speed and cost are just transitions towards better hardware allowing the very best model.
krisoft9 hours ago
And that is why i always carry my groceries with an Antonov An-225 Mriya. Is it really needed? No, but i refuse to compromise on what is(was/will be) the best.
arjie8 hours ago
My experience has been that responsiveness is value. For tasks where you need steering, responsiveness allows for better steering. For tasks which you want unattended, better models are just better.
There are still tasks that even Fable is bad at doing. And many are just mundane things. Because of the fact that you have to steer it on those tasks, you might as well steer an 80% model that is 5x faster. And those do exist.
Naturally there’s a bit of a gap because the faster models need steering on tasks the slower models don’t so there’s no smooth transition but I find it worth it. Especially if you want to stay in flow.
Ironically this sometimes means starting a plan with a great model, planning with a worse model, iterating, then submitting it to a better model for review, and then having the better model do the implementation.
trvz9 hours ago
First, smaller models are fun for hackers: you can run them locally, or run them faster.
Second, when cloud models become unavailable or otherwise deteriorate, these will be all you have. May as well prepare.
breezybottom8 hours ago
If you're hacking a US-based entity, using a high-performance Chinese model through a VPN is probably safe enough. I doubt a local model is going to be sufficiently smart to hack any major company.
trvz8 hours ago
You misunderstood what I was referring to by “hacker” there.
ebiester9 hours ago
It depends on what you're trying to do. For non-coding tasks luna is quite often enough. Flash models are more than enough for summarizing a text, for example, or whipping up a small script to save me fifteen minutes. If you're on a 200/month plan, I see your point. If you're on a dollar limit - or worse, paying per token out of your pocket - you look to be more efficient.
polotics9 hours ago
Can you define your use of the word 'smartest' here just in case some of us don't quite know what you mean?
shafyy9 hours ago
Some reasons: - Smaller models will always be cheaper - Smaller models will always use less energy, therefore better for the environment
It's a bit like saying you always want the fastest and best car; Sure, you can have it if you keep paying for it. But a small car will also get you from A to B, will use less gas and will be much cheaper.
tartuffe789 hours ago
Cost is the point
0xbadcafebee7 hours ago
There's a difference between want and need. I want a 650hp V8 supercar. I need a 150hp I4 toyota corolla. Why choose a less capable car? Because I don't want to spend 10x as much money to get groceries.
agcat9 hours ago
I like the analogy on ways to make small model useful.
[deleted]8 hours agocollapsed
hnrprtlpdb9 hours ago
Well said
retinaros5 hours ago
somehow I cant understand how luna is a step up. to me it feel dumber than 4.1. slower too
bsramin6 hours ago
[flagged]
stas40009 hours ago
[flagged]
kaiji11269 hours ago
[flagged]
RanginGfx7 hours ago
[dead]