dwa35923 hours ago
I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware.
I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat
This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
smeeth2 hours ago
Probably won't have to wait that long. Prism released Bonsai 27B (https://huggingface.co/prism-ml/Ternary-Bonsai-27B-mlx-2bit) as a ternary model a few days ago, its just ~7GB and runs at 44+ t/sec on an m4 max laptop. That's already in the ballpark of active parameter count of most 200B+ models, so we will get a model like this whenever Prism feels like releasing one.
It is debatable if we will actually need that many parameters though, since recursive nets like HRM (https://huggingface.co/sapientinc/HRM-Text-1B) don't need to parametrize as heavily.
dwa35922 hours ago
agreed!! in my heart i really wanted to say by the end of 2026 but wanted to add some wiggle room in case they start to ban open source AI development.
notnullorvoid2 hours ago
I mostly agree with the prediction though maybe a bit more pessimistic about the timeline. Also I'm not sure our current usage of parameter count would make sense in this scenario, such a feat would require compressing current parameters in a manner much different then something producing a bit count per parameter. A hypothetical example would be using a single seed parameter per layer which then passed into a noise function produces the functional weights for that layer, able to reduce per weight size to sub bit levels (256 bit seed, producing 16K weights).
reidrac2 hours ago
> on a decent speed
But you said 7-9 tokens/second, that's not a decent speed. I'm not an expert by all means but in my local experiments, less than 12 to 16 tps is too slow.
LeifCarrotson2 hours ago
I think that any workflow that requires the user to stare at the tokens being generated live is using it wrong. Delegate, don't stare!
https://mikeveerman.github.io/tokenspeed/?rate=10&mode=text
You think of an idea that you want to have the LLM process, queue it up, and go back to what you were doing. Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete. It's kind of like using a 3D printer: It doesn't matter if a print takes 10 hours, because when you come back in the morning it will be done.
Yes, with top-tier GPU farms you can hit hundreds of tokens per second. But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
RussianCow2 hours ago
Once you've used a model that runs at hundreds of TPS, it's hard to go back. Everything completes so quickly that you can iterate without breaking out of flow state. My biggest gripe with slow (<50tps) LLMs is that I've lost all the mental context I built up by the time it's done, which makes it extremely difficult to explore or iterate on solutions.
allknowingfrog2 hours ago
We clearly have different goals. I want an LLM to review my code, not the other way around.
nolok2 hours ago
It's still the same thing, you can ask it to do a full on report give explanation and details be thorough and then go do something else, another task a lunch break whatever and it will be done when you're back
allknowingfrog34 minutes ago
How do you maintain a flow state during a lunch break? I'm looping with Claude on a scale of minutes. While you're waiting, I'm iterating.
vitally3643a few seconds ago
This is like comparing a hammer to a screwdriver and feeling smug because you can hammer nails faster than someone else can drive screws.
These are fundamentally different tools for entirely different applications. They only look similar to people who don't understand the tools or their purpose.
[deleted]28 minutes agocollapsed
Aurornisan hour ago
> Once you've finished reading the next article on HN about a 5 tps Xeon, your task will be complete.
If I spend 10 minutes reading an article, that would only generate 3000 tokens.
That’s not counting the prompt processing time.
We have very different expectations for LLMs if your tasks only take a couple thousand tokens and you’re happy waiting 10 minutes for it.
> Yes, with top-tier GPU farms you can hit hundreds of tokens per second
My 5090 gets hundreds of tokens per second with this model. No farm needed. I’d have to double check but I think even a $1000 Intel B70 might break 100 tokens per second.
> But if the old Xeon in the closet can get useful work done at 5 tokens per second, there are lots of people and lots of use cases where a free, unlimited 5 TPS stream is worth more than paying a dollars per day to get access to a 500 TPS source.
If that old Xeon pulls 200W from the wall and you pay national average electricity costs, it’s going to cost $0.90 per day to run it.
I would rather pay a dollar per day, get my answers 100X faster, and not have an old Xeon heating up my house.
zoobab37 minutes ago
You could suspend it to ram, and only wake it up on request, it takes 2 seconds on my box.
Aurornis24 minutes ago
It’s not a cost savings relative to paying API prices even if you’re suspending it.
This is an option if you must run local inference, you’re not sensitive to speed, and the budget is low.
It’s not going to be cheaper than paying API prices for the model though.
adastra222 hours ago
We aren’t there yet. Not for frontier development work at least.
bitpush2 hours ago
is there a good tool to manage these workloads? batch process a bunch, handle failures, retry things etc?
ctoth2 hours ago
[dead]
aziis982 hours ago
For me, at least for agentic use, you need at least ~40tps. Less might be good only for tasks you could run in the background (like at night maybe).
Instead of coding agents like tool for local models I would like to see more "docker ps" like tools where you can queue up tasks that get processes incrementally (maybe when the pc i idle) and are specialized for doing retries and caching as much work as possible to work decently with slow local models.
dwa35922 hours ago
i am working on making it faster but to me 7-9 tokens/sec feels very good. it was 0 tokens/sec a year ago.
jckahn2 hours ago
Ignore the haters. What you've done is incredible!
Aurornisan hour ago
If you’re interested in these projects you should check out the project this was based on: https://github.com/JustVugg/colibri
It says so right in the readme. They’re not hiding anything.
bigyabai2 hours ago
Yep, and we don't even know how long they spent on prefill. A typical 50-100k token session could take 10-20 minutes to prefill on a Mac.
Pending any new hardware or radically different LLM architectures, we're going to be waiting a lot longer than 2027 for 200B models on local hardware. SOC platforms like Apple Silicon are hamstrung by their obsession over raster performance, the hardware lacks the fundamental GPGPU hardware to be a replacement for real-world inference.
TYPE_FASTER21 minutes ago
Downloading now just 'cause the repo name
typon30 minutes ago
By early 2028, major players like Intel, AMD, QC will ship accelerators in consumer laptops capable of running ~1T MoE models at ~100 tok/s
haute_cuisine2 hours ago
How are the thermals? I noticed that running any serious workload locally heats system fast.
dwa35922 hours ago
i have been optimizing for that. for now samosa is capped at using half of the avaiable cores and switching between them, which keeps the system 'less hot' as it would have been. i will also release better thermal control in the next release. at this point its basically sacrificing about 20% of the speed to keep the hardware less stressed (and hot).
Aurornisan hour ago
> I have a prediction. By the mid of 2027, we will have >200B MoE models running on basic consumer hardware.
This prediction alone isn’t useful at all without a bound on speed and maybe quantization.
You can already run >200B MoE models on basic consumer hardware by picking a low bpw quantization and then streaming the experts from SSD. There have been a lot of proof of concept demos, but nobody uses them because they’re so slow and the quality is so degraded.
If you’re saying that hardware will catch up by mid-2027, I disagree. The limitation is fast memory and that’s going to be expensive for a while. I have a 128GB unified memory machine that can technically run 200B MoE models with enough quantization, but it’s so slow that there’s no reason to do it. It’s going to be a few years before we have enough RAM and processing power in basic consumer hardware without spending as much as a used car to get it.
> This is a GPT4 level model running locally with a decent speed on a 16gb ram macbook air.
Sorry, but 9 tokens per second with a slow prompt processing speed is not decent for anything other than getting short chat responses.
You’re also not running the full GPT4 quality model. I’m very familiar with that model from some other work and the 4-bit quants are just not as good as all of those KL divergence plots would have you believe.
You also have very short context. It’s basically useless for anything more than short chats where you’re okay watching the output come back at reading speed, skipping the reasoning part (which is important for calling it GPT4 level quality), and waiting a long time for the first token.
Yes, it’s technically running, but not in a way that would be useful by normal LLM standards.
Capricorn24812 hours ago
I tried Qwen3.6-35B-A3B, but it couldn't generate a 50-100 line Clojure file without having broken parens mismatches. I know Clojure isn't super popular, but the syntax is pretty simple and the frontier models do fine with it.
segmondy19 minutes ago
try q8, check your parameters. qwen3.6-35b-a3b should definitely be able to do so with no issues at all.
Azantys2 hours ago
You are comparing a 35B models to a 635B+ frontier model, of course thats not even close
Capricorn248142 minutes ago
I'm not lamenting that they aren't close, I'm saying Qwen will frequently output code that isn't even syntactically correct, even when the syntax is simple. Which makes it unusable for coding.
IsTom29 minutes ago
To be fair, they don't have the text editor highlighting all the matching parens. I'd be lost too.
[deleted]2 hours agocollapsed
rvz2 hours ago
> I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second.
That is no where near decent at all.
dwa35922 hours ago
it's a 16GB machine. i am proud of this machine so far.
embedding-shape3 hours ago
> I am running Qwen3.6-35B-A3B locally on my 16GB mac with 7-9 tokens/second. Link - https://github.com/deepanwadhwa/samosa-chat
It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that, seems there was another collaborator ;)
Show something you've built with the model+tooling instead, truly dogfood it. I'm sure you'll discover things along the way too!
smith70182 hours ago
Nothing says they're using Qwen for local development. They could be using it to for conversations, knowledge, or "creative writing."
dwa35922 hours ago
>>It's quite telling you didn't use Qwen3.6-35B-A3B locally to build that
that would have run into a race condition unfortunately ;)
but there is a sample landing page + a python function on the repo which shows what the model produced. my goal is to integrate the local model in my workflow so that claude/OAI can call this model for basic stuff.
hagen82 hours ago
Some ppl don't like to hear it. But I would assume that token costs when using an inference provider are cheaper than electricity of using locally.
If we just take into account output token generation for simplicity. With 5tps u get 18k tokens an hour. That would costs around 0.005USD from an inference provider.
I estimate that the server consumes probably around 500W during inference.
In Germany where 1kwh cost around 0.3USD, 18k tokens inferred locally would therefore cost 0.15USD which is 30x the costs of using an inference provider.
But for ppl who worry about their data, running locally might still be good. However, they should be aware, that it is much less efficient than using an inference provider.
The efficiency gap will also significantly increase as new GPUs will make inference much more efficient.
EDIT: I first thought it'd be 180k token, but thanks to someone mentioning in the comments, it is 18k. I guess with that, it will be tough unless u got electricity almost for free. Also, the inference providers are probably still using H200/H100 for those small models. Once they use GB300 or next year the new Ruby GPUs, inference will be cheaper by a factor of 30. By then, running local models will mostly be about privacy.
bischofs2 hours ago
I don't pay anywhere near 0.30usd in the US - I pay half that off peak and can buy 1000$ worth of batteries to load up on super off peak (0.11usd). Also the inference providers are fighting over market share with huge debt loads so they are definitely going to go up in price.
hagen829 minutes ago
Inference costs will go down massively once they use the upcoming GPUs. I estimated that a model like GLM5.2 will be around 0.03USD/M output tokens in 2 years when the Feynman GPUs will be available in 2028. And this did not even consider architectural efficiency improvements. In mid 2027 we will already see a 10x reduction once everyone has switched to the Ruby architecture.
It will be feasible for everyone to have 20 different agents running at all times. A new world is coming
amanaplanacanal2 hours ago
Yeah I had to check, I'm paying 0.08usd per kwh. This is in the US PNW with quite a bit of local hydro power.
brewtide2 hours ago
It's all relative. On the opposite coast, Maine it is ~ $0.28 cents kwh including getting it there. (~ 50% energy, 50% delivery). It's too darn expensive here.
sosodev2 hours ago
Of course efficiency matters, but a lot of people either have cheap electricity or efficient hardware. My AMD strix halo home server can serve Gemma4-26B at like 70 TPS (rough estimate, I don’t remember the exact speed buts its fast af) while only using 100W.
segmondy18 minutes ago
don't care, and yeah i don't like to hear it. we don't run local because it's cheaper money wise. we do it for freedom, for privacy and having option makes it cheaper in the long run. if there was no local options, your cloud model would cost much more!
albrewer2 hours ago
Not OP, but your math is a bit off - I have solar panels :)
apitmanan hour ago
It would be more efficient if you had multiple users (or agents) making parallel requests to take advantage of batching, right?
kylemaxwell2 hours ago
You pay 3x as much for electricity as I do, so the math here is going to work out very differently depending on a lot of factors.
Waterluvian2 hours ago
Locally I’m looking at about CAD $0.05 per kWh when off peak.
newusertoday2 hours ago
its 18k not 180k
bigyabai2 hours ago
It's the "Race-to-Idle" situation all over again. It consumes less power to complete a task faster, whereas using "low power" hardware that draws max TDP for 30 minutes isn't very power efficient.
The privacy nuts have a better leg to stand on, but even then it's hard to believe that they're using on-prem AI to replace SOTA model inference. As cool as local LLMs are, a lot of the stuff people run is a novelty.
hparadiz3 hours ago
Here's my report running several different models on a dual Xeon with 256 GB of DDR4 and no GPU.
https://gist.github.com/hparadiz/f3596d00a62d8ebb2dadcc46ee5...
puzzlingcaptcha3 hours ago
Have you tried with a single CPU to get rid of the NUMA penalty? I understand this likely means halving the memory but I am interested in how much of a difference it makes
trollbridge3 hours ago
I have (192GB machine with two CPUs), pretty much does the trick. It just runs some small models used for embedding, etc. and has those on one CPU / memory node and all the Docker containers on the other one.c
ryandrake2 hours ago
I have a dual xeon also, same as OP: Ivy Bridge + 128GB DRAM, and was never really able to get decent LLM performance out of it. So I ended up biting the bullet and adding a "budget tier" A4000 20GB GPU. Too bad all my DRAM is wasted now--not sure if there is a way to take advantage of lots of DRAM once you move over to having inference happening on the GPU.
neomindryanop3 hours ago
Thank you for sharing!
throwaway20274 hours ago
That's quite slow I'm getting 8-12 t/s on a 13 year old CPU. (Speed varies by context size and other settings who knows)
neomindryanop3 hours ago
Thank you for sharing / linking!
tuwtuwtuwtuw2 hours ago
But OP is using Q8 and you're using Q4?
neomindryanop5 hours ago
Author here, it looks like my original comment was flagged for some reason. The fix is open upstream as PR #2138 (https://github.com/ikawrakow/ik_llama.cpp/pull/2138)
Aurornisan hour ago
A dual Xeon of this era is probably pulling 300W or more when loaded.
At national average electricity prices, that’s $1.35 per day. More during the summer if you have to cool the space.
If you run it 24/7 and ignore prompt processing time (not a good assumption at all) it would get around 400,000 tokens in a day.
That’s about $0.30 per million output tokens.
Coincidentally, that’s the same price for this model on OpenRouter right now, but OpenRouter token gen will be 8X faster.
There are a lot of good reasons to experiment with running LLMs locally, like if you don’t want any data leaving your house.
Don’t think that you’re going to come out ahead monetarily. I say this as someone with a lot more money invested in local inference hardware at home. It’s fun, but it’s not a way to save money.
bitexploder11 minutes ago
It gets better in the cooler months when heating is running in a home :)
brailsafean hour ago
Reasonable analysis, especially because this person seems to have an actual house. In my case, I rent and don't pay for electricity directly, so the cost effectiveness threshold is whenever the landlord starts complaining
twstedan hour ago
I think, may be actually wrong, that most of us do not consider running a model locally a way to save money. It is a way not to spread personal info around.
Aurornis37 minutes ago
Anyone running LLMs at home will come to that realization quickly, if they’re looking at their power bills. Even feeling the heat output of a computer running at 100% in your office makes it clear.
I was responding to a lot of the comments saying this was a reasonable way to avoid paying for tokens or subscriptions. I don’t want anyone getting the wrong idea that this is a way to save money if that’s their priority.
rhema38 minutes ago
I love my little dual core X99 board with Xeon E5 2673 V3. It's not power efficient, but I just leave it in my basement for local Jupyter Notebook stuff. Much faster than everything cloud-based for a reasonably price at my scale.
deltamidway3 hours ago
He's shown me his set up in his basement. It's sick! Talk about your 3d printer next!
broabprobe2 hours ago
I run the same setup Gemma 4 26B on a 2013 Mac Pro (dual graphics cards but they're useless for this). I also get about 5 t/s. It's perfectly serviceable for some tasks!
kzrdude2 hours ago
What is it useful for at slow speed?
rrhjm53270an hour ago
Good job. Thanks for sharing. I have a similar NAS server with 2 Intel(R) Xeon(R) CPU E5-2699 CPUs. I will test as well.
thomasjb2 hours ago
I was inspired by that post also, got a Qwen Coder 1.5B up to 27tok/s prompt eval and 13tok/s decode on an e5-2650v2 inside a GNOME box
jeswin2 hours ago
The transformer architecture is fundamentally unsuitable for local inference, while being efficient at scale. It's a fun experiment to try, but that's about it.
mmastrac3 hours ago
Is it just me or does this post not mention how much RAM they had? I would love to know - I have a dual-Xeon 1U screamer with 96GB of DDR4 RDIMM just sitting around...
FWIW I'm getting a hardware max of 20 tok/s (approx topping out the GPU's compute) on my custom local diffusiongemma port running on an M3.
neomindryanop3 hours ago
hey, I’m the author. That box has 384gb, but loading the model “only” uses about 80gb.
fouc2 hours ago
any reason you went with q8 over q4? I'm wondering if q4 would run noticeably faster or not.
superkuh2 hours ago
Such a system is RAM bandwidth limited and not compute limited Switching to q4 from q8 would decrease the amount of data needing to be loaded by half. The token generation rate would nearly double. But generally if you can do q6 or q8 and you have enough RAM you really should. Even if it's slower.
aniwalunj4 hours ago
Truly amazing. This gives a peek into the future for what's possible.
robotswantdata3 hours ago
Need to run this on my Xeons with AMX
ChrisArchitect3 hours ago
okokwhatever3 hours ago
To me context means everything. Tokens per second is a great metric but in the real world context window is the deal breaker when a real use case is on the table.
dofm2 hours ago
Gemma 4 26B is capable up to 256k or 262k, can't remember which.
Whether the writer's setup affects that choice I don't know.
rvz2 hours ago
...and how many servings can this do?
neomindryanop5 hours ago
Author here. The short version: a viral post ran Gemma 4 on a 2016 Xeon; my Xeons are 2013, and the fork it used assumes AVX2, which Ivy Bridge doesn't have. The build failure was easy. The fun bug was the silent one: two MoE graph ops with no dispatch case on non-AVX2 builds, so every expert FFN output was uninitialized memory. Deterministic, NaN-free, fluent-looking multilingual gibberish.
The fix is open upstream as PR #2138 (https://github.com/ikawrakow/ik_llama.cpp/pull/2138), awaiting review. Fair warning on the AI angle: the patch was written by Claude at my direction. The post is explicit about which parts were me and which weren't. Happy to answer questions about either the bug or the workflow.
otherjason4 hours ago
This reads as pretty clearly AI-generated text, which is against HN guidelines.
FL4103 hours ago
The PR? He said it was AI in the comment you replied to...
I don't think the post itself reads like AI at all, but that's just me.
twoodfin14 minutes ago
The post is absolutely LLM-generated. “Punchy” short sentences, “… has quietly come to mean …”, “The optimized paths weren’t there to execute.”
logicallee3 hours ago
I think "this" refers to its parent comment. Part of it sounds like Claude wrote it. AI-generated comments aren't allowed on HN.
otherjasonan hour ago
Indeed, I was referring to the parent comment.
pkghost3 hours ago
Here's the thing: life also imitates art. If you invert your load-bearing assumption, it could be that he just reads too much slop. But my honest take? You might be right.
dofm2 hours ago
stopitgetsomehelp.gif ;-)
OsamaJaber3 hours ago
[dead]
CurbStomper2 hours ago
[dead]
rvba3 hours ago
Apologies for asking here but literally nobody knows:
Android studio connected to a local model disconnects automatically after 10 minutes. How set this limit to 12 hours or remove it completely?
I could run my LM studio model all night... but I cant, since Android studio times out after a hard limit of 10M.
This is not related to number of tokens.
I tried Googling, searching for settings in Android studio, even created a stackoverflow post - but zero information. Jetbrains mentions "remote agent timeout mechanism" - but after changing it, nothing happens.
NortySpock2 hours ago
If the local model is served via ollama, there's a default timeout of 10 minutes , which can be adjusted either per-call , or (as I did) in the systemd service environment variables
https://docs.ollama.com/faq#how-do-i-keep-a-model-loaded-in-...
You didn't specify what was serving your local model.
rvbaan hour ago
Thank you for your reply. I use LM studio (local server), but can switch to a different tool.
Do you know how to switch it in LM studio?
What I see is that: android studio gives "Error: stream failed" and in LM studio server I see it is still working, then says that client (=android studio) disconnected.
So I assumed it was a setting in android studio.
indigodaddy2 hours ago
Unfortunately, the post comes off as AI-written. Why not just write your own posts?
CookieCrisp2 hours ago
Not everyone is as confident in their writing as they are in their engineering