Hacker News

hey,

how does someone get started with doing things like these (writing inference code/ cuda etc..). any guidance is appreciated. i understand one doesn't just directly write these things and this would require some kind of reading. would be great to receive some pointers.

There is also another Mistral implementation: https://github.com/EricLBuehler/mistral.rs Not sure what the difference is, but it seems to be just be overall better received.

Mistral can really end up having its RedHat moment. I think open models will only get more interesting from here.

It's not fast enough to be realtime, though you could do a more advanced UI and a ring buffer and have it as you describe. (ex. I do this with Whisper in Flutter, and also inference GGUFs in llama.cpp via Dart)

This isn't even close to realtime on M4 Max. Whisper's ~realtime on any device post-2022 with an ONNX implementation. The extra inference cost isn't worth the WER decrease on consumer hardware, or at least, wouldn't be worth the time implementing.

I don't think local as it stands with browsers will take off simply from the lead time (of downloading the model), but a new web API for LLMs could change that. Some standard API to communicate with the user's preferred model, abstracting over local inference (like what Chrome does with Gemini Nano (?)) and remote inference (LM Studio or calling out to a provider). This way, every site that wants a language model just has to ask the browser for it, and they'd share weights on-disk across sites.

Just tried out Handy. This is much better and lightweight UI than the previous solutions I've tried out! I know it wasn't you intention, but thank you for the recommendation!

That said, I now agree with your original statement and really want Voxtral support...

I'm looking into porting this into transcribe-rs so handy can use it.

The first cut will probably not be a streaming implementation

Same here. I haven’t found an ASR/STT/transcription setup that beats Parakeet V3 on the speed/accuracy tradeoff spectrum: transcription is extremely fast (near instant for a couple sentences, 1-3 seconds for long ramblings), and the slight accuracy drop relative to heavier/slower models is immaterial for the use case of talking to AIs that can “read between the lines” (terminal coding agents etc).

I use Parakeet V3 in the excellent Handy [1] open source app. I tried incorporating the C-language implementation mentioned by others, into Handy, but it was significantly slower. Speed is absolutely critical for good UX in STT.

[1] https://github.com/cjpais/Handy

Personally I run an ollama server. Models load pretty quickly.

There's a distinction between tokens per second and time to first token.

Delays come for me when I have to load a new model, or if I'm swapping in a particularly large context.

Most of the time, since the model is already loaded, and I'm starting with a small context that builds over time, tokens per second is the biggest impactor.

It's worth noting I don't do much fancy stuff, a tiny bit of agent stuff, I mainly use qwen-coder 30a3b or qwen2.5 code instruct/base 7b.

I'm finding more complex agent stuff where multiple agents are used can really slow things down if they're swapping large contexts. ik_llama has prompt caching which help speed this up when swapping between agent contexts up until a point.

tldr: loading weights each time isn't much of a problem, unless you're having to switch between models and contexts a lot, which modern agent stuff is starting to.

It's obviously not something you'd want to happen _passively_ when visiting a web page, but if the alternative is installing an executable / using a package manager / etc., why not? At least the browser is a more secure sandboxed environment for running untrusted code than most peoples' native OS.

I recently dug into this as I was trying to benchmark the possibility of using Gemini Nano (Chrome's built in AI model) vs a server side solution for a sideproject.

Nano's stored in localstorage with shared access across sites (because Google), so users only need to download it once. Which I don't think is the case with Mistral, etc.

There's some other production stats around adoption, availability and performance that were interesting as well:

https://sendcheckit.com/blog/ai-powered-subject-line-alterna...

You have already gotten used to loading multiple megabytes of bytes just to display a static landing page. You'll get used to this as well... just give it some time :-D

[flagged]

> Streaming speech recognition running natively and in the browser. A pure Rust implementation of Mistral's Voxtral Mini 4B Realtime model using the Burn ML framework.

> The Q4 GGUF quantized path (2.5 GB) runs entirely client-side in a browser tab via WASM + WebGPU. Try it live.

Excluding names (Mistral's Voxtral Mini 4B Realtime), you have 1 pretty normal sentence introducing what this is (Streaming speech recognition running natively and in the browser) and the rest is technical details.

It's like complaining that a car description Would contain engine size and output in the third sentence.

This semi-quantized version targets the Jetson Orin Nano, but only comes with a simple inference engine.

https://huggingface.co/Teaspoon-AI/Voxtral-Mini-4B-INT4-Jets...

https://kyutai.org/stt has an implementation for MLX and mentions iPhones, so it should work on edge devices, Macs and iPhones.

fwiw, that is the right-ish transliteration into arabic. It just picked the wrong language to transcribe to lol

Would check memory, ensure you have free ram. Tested here https://imgur.com/a/3vLJ6no Not perfect dictation, but close enough

Does disabling shields help?