LarsDu884 minutes ago
I'm surprised neither OpenAI nor Anthropic made this move first. The Chinese open weight models are pulling ahead and commoditizing their value proposition.
Baking models onto silicon would've been the next logical move to get a moat.
Google is already doing this and has an experimental project on top of already having TPUs and cramming their quantized flash onto individual TPUs for inference.
whythismattersan hour ago
The demo: https://chatjimmy.ai/
walrus01a few seconds ago
I know it's a relatively tiny model, but damn, is that thing fast.
It also mostly passes the "schlong" test
wxw18 minutes ago
I freakin' love this demo. It feels magical.
itvisionop22 minutes ago
OMFG this thing is fast.
nsxwolf28 minutes ago
It doesn’t believe it’s running on that chip, it’s arguing with me
shaewest20 minutes ago
It's running a very small, non-reasoning model at the moment. But more generally, almost all LLMs argue on the hardware/model they are/are on.
metadat3 minutes ago
What would tokens/sec performance look like for a reasoning model? An order of magnitude slower?
dumberquestions17 minutes ago
Which model? Or how many active parameters?
_whiteCaps_13 minutes ago
Llama 3.1 8B model
[deleted]13 minutes agocollapsed
mikeayles23 minutes ago
AMD could have saved their money and used their own hardware! I've got a language model doing 60k tok/s on AMD hardware already, a Xilinx Kria K26 SOM, with the weights baked into URAM/BRAM with zero DRAM in the token loop. Same thesis as Taalas: single-stream decode is bandwidth bound, so stop fetching weights from far away.
Caveats stacked high, obviously. It's 3.16M parameters (tinystories, and I also have a kevin-speak lemmatised version), the tokens are characters, and the 60k record is 16 streams that each remember exactly one token of context, so it's blisteringly fast at saying nothing. The honest build with full context and KV caching still does ~19k tok/s on one stream though.
I keep messing with the blogpost with the live demo, but I'm planning on flipping it to live in the next day or two
tandr11 minutes ago
Well, technically it is their hardware now...
A_D_E_P_Tan hour ago
This is probably a win-win. The team gets paid, and we get greater assurance that their best ideas and architectures -- which are truly impressive -- are going to see the light of day in actual products.
badatnamesan hour ago
They were too small for this to be a meaningfully sized purchase for AMD, there's real risk they get sucked into a team that ultimately delivers sqat, not to mention the chances of anything being delivered in an even remotely consumer-priced bracket are definitely out the window
bhouston37 minutes ago
Toronto Canada startup btw.
kridsdale115 minutes ago
Works well, I remember driving by the ATI building as a kid.
cmrdporcupine19 minutes ago
Seems to be somehow some kind of offshoot from or connected to Tenstorrent, which is just down the road. Founder looks like he was/is maybe at Tenstorrent and previously associated with Keller?
Always fantasize about applying at Tenstorrent, but wrong side of Toronto. 2 hour commute.
syntaxing40 minutes ago
Honestly, this is starting to make more and more sense. SOTA models are starting to converge to certain architecture and capabilities. I wouldn’t be surprised we end up with a base model ASIC + “fine tune” card where it’s a physical LoRA style adapter.
encyclopedism23 minutes ago
Imagine a multi-modal model with 1000's of tokens per second. Realtime inference for a host of applications. This is a BIG deal and will change the landscape in unfathomable ways.
The https://chatjimmy.ai demo was impressive.
Once models settle down this makes sense. Imagine a cartridge with a physical model on it. You purchase a cartridge and stick it in your computer/phone/server. Want to upgrade? By a new 'cartridge'.
This should bring inference cost down dramatically, I wonder how OpenAI/Anthropic feel about that.
kevin_thibedeau8 minutes ago
Then we can have machine psychologists pull cards when they run anok.
VladVladikoff26 minutes ago
Wouldn't this mean someone with sufficient hardware could lift the SOTA model weights off the chip? Or are you saying that these chips would only be used internally by these companies and not sold to the public?
dumberquestions12 minutes ago
I wouldn't expect companies not sharing their weights today to be any more likely to share them if they're on hardware, this doesn't sufficiently hide weights from a local user.
snek_case19 minutes ago
The weights are very unlikely to be on the chip itself. That wouldn't work for SOTA models that are terabyte scale, even quantized. This is probably an accelerator for specific kernels in the model, but the weights are likely loaded from memory. The chip may have SRAM to store some of the weights temporarily during inference.
syntaxing24 minutes ago
I don’t get why this is an issue? You can run Claude/OpenAI SOTA models through Amazon bedrock. These weights have to live somewhere to run on Bedrock.
amazingamazing18 minutes ago
One idea would be to use an open model.
smokel34 minutes ago
The technical aspects of SOTA models are not publicly documented. How do you know if something is converging?
syntaxing26 minutes ago
SOTA American models are not. SOTA Chinese models are. From a physics aspect, closed source models cannot be too far from open source ones in terms of size. There’s only so much you can squeeze out a B100 style cluster even with fancy Dflash style diffusion model for the speculative model.
_aavaa_31 minutes ago
If we had deepseek v4 flash 0731 etched on a chip it would be more than capable enough and fast enough for so many people's needs, even hardcore engineer.
nurumaik21 minutes ago
Will be capable and fast enough for 2-3 weeks until new sota drops
amazingamazing18 minutes ago
If it is capable today why would a new model change this?
catchnear43217 minutes ago
if capability is a commodity then the differentiator becomes taste.
FridgeSeal7 minutes ago
Because new stuff instantly makes anything prior bad and incapable and garbage of course! Did you forget the hype-machine speaking notes??? /s
cyanydeez31 minutes ago
if they were still exponentially increasing, they wouldn't be preparing for an IPO. IPO is where companies go to die and founders escape.
cyanydeez30 minutes ago
I don't think there'll be a fine tune card; you'll have the base model vintage whatever year, and then your GPU will do whatever LoRA layers you want it to do; the LoRA will wrangle older dated models into the current of whatever your looking at.
But yeah, for things like programming, if it can do linux and python and some go and sql and javascript, larger domains can be threaded with LORA
bob102920 minutes ago
I feel like NAND process tech could become useful at solving some of these problems. A GPU where you can update the weights a few thousand times may be sufficient.
addaon7 minutes ago
NAND hasn't been scaling great lately. It seems like PCM or MRAM would both be better fits.
kridsdale118 minutes ago
FPGA model storage?
walrus014 minutes ago
Imagine the size of chip needed to 'etch' something like Qwen 3.6 27B in size.
proxysnaan hour ago
Really hoped to see their hw out in the wild one day
MarkWayneNewtonan hour ago
While this design is self-limiting I think its a good approach. It doesn't take an entirely new architecture or infinite memory to produce significant performance improvement.
badatnamesan hour ago
Well so much for that dream.
Guess we can look forward to picking these up ex-enterprise on ebay for under $5k a pop in a decade or two
fellowniusmonk10 minutes ago
Token quantity will have a quality all its own.
ycui7an hour ago
so qwen3.x-27b on hardware? or better deepseek-v4-flash on hardware .
ilaksh41 minutes ago
I wrote them an email asking for PrismML Bonsai 27b Ternary which is like 6b or something crazy small and would be a lot easier for them to do initially.
rvzan hour ago
Didn't even give them a chance to launch the hardware.