There is a lot of buzz around this topic but i am really curious about what's actually useful day to day when running AI/LLMs locally on your PC. A few things I'm trying to understand:
Which models are you running locally and on what hardware? What use cases have actually worked out considering the speed and memory limitation? At what point did local start making more sense than just hitting an API? What's still not there yet?
Feels like we're at an interesting inflection point but hard to cut through the noise on what's working v/s just setting it up
denn-gubsky3 days ago
Recently I upgraded my TrueNAS box with Ryzen 7 8900 APU + 96GB DDR5. This config runs Ollama + loomcycle (my agentic Go runtime) + loomboard (the agentic UI), all self-hosted. I tired several models and Ollama configurations. For me is important to fit the whole model in GPU and run multiple agents+tool in parallel (see the blog for details: https://loomcycle.dev/blog/local-llms-on-truenas-and-the-fro...). Gemma4 is fast, but hallucinating a lot, especially when using tools. My current favorite is qwen3.6:27b. This is smart and reliable model, which can run tools in agentic configuration, do long researches, generate code, formulas and diagrams. On my hardware the qwen3.6 speed is about 9-13 tok/s, which is not high, but acceptable. The most tricky part is context size balancing for multiple model instances in limited memory. Ollama splits the context memory between model instances and may fail to run if agentic runtime allocates larger context than size than it is available in the context storage.
PaiDxng4 days ago
Local is best when you're iterating fast on prompt templates or need to run the same task hundreds of times—API costs add up quickly there.
PaulHoule4 days ago
For a long time it has been a win to do clustering, retrieval, ranking, classification, anomaly detection and other non-generative tasks with local models from the BERT family. These are so much better than all the things I tried from 2000-2020. ModernBERT for feature generation plus classical algorithms from scikit-learn works so well, plus scikit-learn has great tools for eval, model selection, probability calibration and all the other things that make the difference between "write another arXiv paper" vs "deploy a product into production"
If you read the splogs you'd think it is all about the "zero-shot" scenario where you write a prompt asking what decision you want the model to make. Yeah, it is amazing that it does so well with so little input, but side by side "many-shot" models that learn a decision surface in hyperspace beat the pants off "few-shot" models that look at nearest neighbors and "zero-shot" models that do whatever they do. It's not even close.
The reckoning for generative AI may near and it might be like something right out of "Where's your Ed at?" When people see honest prices for current models I think a lot of people might decide the juice isn't worth the squeeze. The story in the next few year is not going to be bigger models that only Elon Musk models but a relentless fight to get costs down.