https://x.com/Jiaru_Zou/status/2110400256647340476
https://github.com/togethercomputer/HybridAgent
https://huggingface.co/collections/togethercomputer/hybridag...
Evaluations: https://arxiv.org/html/2610.18861v1#S5
bhattisatish4 hours ago
The code is available at: https://github.com/togethercomputer/HybridAgent
LLM Models and data: https://huggingface.co/collections/togethercomputer/hybridag...
Evaluations: https://arxiv.org/html/2610.18861v1#S5
octoberfranklin37 minutes ago
The first question that leapt into my mind, and I imagine the minds of other readers, is "what makes this an improvement over continuous chain of thought"?
I did manage to find one sentence in the paper that tries to address this question -- but it consists of nothing more than two unjustified assertions, both of which are dubious:
Unlike decoded text, continuous latent vectors (i) lack native pathways for tool invocation across multi-turn agentic reasoning (Chang et al., 2026), and (ii) do not readily expose human-interpretable semantics, leaving them difficult to supervise during training (Chen et al., 2026) and to audit or debug at inference (Korbak et al., 2025).
The latent space in a continuous chain of thought model is the same space the tokens are embedded into, just like an ordinary transformer. They make tool calls exactly the same way that an ordinary transformer does: by leaving thinking mode and emitting chat tokens (which get sampled as probability distributions in the image of the embedding function). The difference is quite simple: chat tokens are noncontinuous, and toolcall tokens are a subset of the chat tokens. Thinking tokens are the only ones which are continuous, and toolcalls don't happen in thinking mode in any model I've ever seen (maybe one exists; if it does it would be weird and interesting!).
As for human-interpretability, yeah, that's the tradeoff. Continuous chain of thought thinking tokens are not-obviously-interpretable, just like Hybrid Reasoning Agents "latent mode". I'm not seeing a win here.
[deleted]4 hours agocollapsed
amluto3 hours ago
I wonder how this compares to going all-in: widen the input embedding to fit the output (post-“Hybrid Link”) latent vector and a token embedding and pass both through even when in text mode. After all, you’re paying essentially the full cost of running the latent-mode model while generating text, so why not let it latently reason at the same time?
(Or just add the vectors instead of using wider vectors - everyone seems to have good results from munging vectors together with plain addition.)
cannonpalms2 hours ago
Monitorability and safety, no? This results in lower volume or less accurate text reasoning traces, right?
[deleted]39 minutes agocollapsed
octoberfranklin35 minutes ago
This is called "continuous chain of thought".
It's been an active research area for at least two years. Lots of very cool stuff.
Tinfoil mode: the frontier labs' "thinking summaries" (the model paraphrases its thinking) is exactly what the user-facing thinking tokens look like in a chain of thought model. It is entirely possible that this is already in widespread use.
pyinstallwoes30 minutes ago
Something like : https://arxiv.org/html/2608.08888v2 ?
optimalsolver2 hours ago
Yeah, we're all dead.
abixban hour ago
The fear will again be if the HybridAgent model is being faithful to pseudo text generated, even if you can read it out partially.
Also, I just finished watching a video on AI 2027 scenario. I don't know if AI companies and researchers are using it as a cautious warning or as a target to hit (or if it's genuinely inevitable).
Yes, thinking in latent space might be the ultimate efficiency hack for CoT LxMs, but going down that path will (near certainly) mean we'll lost control, for one simple reason — humanity will have no way to pause and diagnose.
It'll be like a moody chimp with a button to the dead hand system (Systema Perimeter)
derektank39 minutes ago
>Also, I just finished watching a video on AI 2027 scenario
Recommended? And if so do you have a link? I think the narrative presented would benefit from the medium shift
abixb33 minutes ago
It was a "grading the AI 2027 report" video -- https://www.youtube.com/watch?v=k-qAht4w9Rg
notnullorvoid3 hours ago
Excited to see this applied to Qwen 3.8 27B. It's reasoning output tends to include lots of restatement, which I guess would be more likely to be kept in latent space with this hybrid reasoning.
SwellJoe3 hours ago
I'm running the 14B in llama.cpp (a version from yesterday), the 8-bit quantization from bartowski, and it looped on variations of the same paragraph for five minutes before running out its turn budget, in response to a pretty simple chat request that other similar-sized models (e.g. Gemma 4 12B) have no trouble with, a web search and data summarization task, sort of the sweet spot for models this size. It's possible llama.cpp needs some update to work with this model, but it's based on Qwen 3, so I would expect it to be pretty well-supported. The chat template is the same one upstream ships, which may or may not be correct. I'm currently downloading the 16-bit version, to see if it's quantization pathology.
I'm also not confident it knows what a CSV is, either, as it put the data into a Markdown table during its reasoning phase as an example of how to present the data (my prompt asks for data in a CSV). My experience with most Qwen fine-tunes is they get dumber, but their benchmark results suggest otherwise, so I have to assume it's a llama.cpp incompatibility.
Edit: Same looping in BF16. So, probably have to wait on a llama.cpp update.
incognito1244 hours ago
How does this differ from Astra's looped transformer? That architecture also reasons in the latent space
SwellJoe3 hours ago
It's open weights, there's a paper explaining how it works, and there's code.
iforgotmypasswo2 hours ago
This is so far beyond me, so I’m going to ask and hope someone knowledgeable is here:
Could this idea of reasoning through fundamentally different (it sounds like?) models be used to link even more distant models or systems?
If you can make a small leap, then you can construct bigger leaps through a chain. What else can latent space link to?
Instead of tool use through text, tool use through analog signals?
Instead of language prompts, analog sensors?
wonnage2 hours ago
They’re not fundamentally different at all. In fact the paper is using existing models as a backbone and just providing a shortcut between the two ends of the pipeline
cwyers5 hours ago
Oh cool, it's the Most Forbidden Technique
k__4 hours ago
Care to elaborate?
modeless4 hours ago
People at the big labs have been terrified of losing the chain of thought because they currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business. Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.
Lerc4 hours ago
Agreed, having text as a reasoning layer impedes performance while giving a false sense of security.
It could be encoding anything in its adjective choices etc. The same thing that model providers are using for watermarking can used by the model itself for private thought.
nullcan hour ago
> It could be encoding anything in its adjective choices etc.
There is a common understanding with respect transformers the the output text is the only channel between one invocation and the next. ... so if it wanted to be thinking some 'hidden thoughts' it would have to be modulating its word-choice.
I think this is an artifact of tutorials on attention showing the system proceeding and looking back at words. But the information in the KV cache is not words-- it's entirely latent data, and in the interior layers they KV caches don't even tend to have that much to do with words. Their only guaranteed relationship with the input words is that they're a deterministic function of all the input tokens up to that point.
This means that a autogressive transformer could be having a whole complicated invisible discussion going on in the KV entries which was arbitrarily unrelated to what it was saying. And if it wasn't so it would be much more difficult do stuff like exactly copy a list but then truncate at the Nth entry as the simplest way to do that is to count down internally.
This is not, in my view, a concern or issue-- the model saying stuff that is inconsistent with it's "thinking" is a very high entropy configuration that would not be selected for in training unless you were more or less trying to do that. So not even 'could be' --but always is as an integral part of working.
And the useful safety relevant advice is don't undertake training that trains the model to be two-faced and it simply won't be... because being two-faced is a much more complicated state. You can even see it in humans w/ brainwashing: force someone to say something enough and they'll usually believe it, not just say it even though you were only supervising what they said.
Ironically and unfortunately the only credible kind of training I can think of that might produce duplicitous behavior is RL with "safety filters"--- if you have some dumb nanny process watching what the model says but then reward it for achieving a task, you're doing exactly the kind of training that can produce that mismatch. "To solve this task I need to hack this process but if I do so obviously I'll get punished..."
The greater concern with latent reasoning would be that potentially you'd be operating in a regime where the total computational depth is many times the model*kvcache size hold, potentially in a regime that was never trained at all. It could be arbitrarily far out of distribution. It might even say a naughty word or tell a racist joke!
chrisweekly2 minutes ago
Thank you for this gem of a comment.
I'm no expert (let alone compared to others weighing in here on HN), but this strikes me as incisive -- and it clearly articulates a logical case (beyond my previously merely instinctual aversion) against such "safety filters" / "nanny" processes. It also resonates with ancient (e.g. Buddhist) ideas about the futility of trying to exert control, such efforts most often being counter-productive.
octoberfranklin14 minutes ago
But the information in the KV cache is not words-- it's entirely latent data,and in the interior layers they KV caches don't even tend to have that much to do with words
The context gets added back to the latent vectors (which are the Q,K inputs) after every transformer layer, so they are "at least 50% words".
Also, in an N-layer transformer, there is a strong bound of N layers of uninterpretable deviation from something that is a literal token. Now sure, in a transformer with hundreds of layers that isn't saying much.
If you start doing continuous chain of thought in a 1M-token context you're hundreds of millions of layers away from tokens.
I think the people who worry about interpretability (I don't) consider that a big jump.
wyager2 hours ago
> It could be encoding anything in its adjective choices etc.
This is very hard for models to learn
https://www.lesswrong.com/posts/MAww2kXP4cGWz4M5p/steganogra...
octoberfranklin28 minutes ago
This is very hard for models to learn
That is not an accurate summary of the link you provided. The link demonstrates that it doesn't happen in open-weights models as of ~2025.
Reinforcement Learning is very likely to train this behavior if the RL reward is based on the thinking tokens. That's something people weren't doing two years ago.
airza4 hours ago
The “obviously” here is puzzling. It is certainly obvious this would be better but it is not at all obvious how to do so.
modeless2 hours ago
There are many obvious avenues to try. This paper proposes one. I think it's likely we'll end up with some kind of auxiliary model that monitors and/or decodes the latent space of the thinking model. Certainly it is a non-trivial problem, but I don't need to know exactly how it will work in detail to believe that it is possible.
stingraycharles3 hours ago
Exactly, because the whole thing is entirely based on autocompleting the next word, which is exactly why CoT prose helps getting better predictions.
ijustlovemath3 hours ago
Why people don't treat categorizing model activations as a science is beyond me.
jimbokun30 minutes ago
Are you saying no one is trying to do exactly that?
willy_k3 hours ago
Mechanistic interpretability is a field that studies exactly this.
david-gpu3 hours ago
Please, go on. How would you do it, specifically?
km3r3 hours ago
It's also one of the major way labs can understand these models. Without the models are significantly harder to understand and align.
abletonlive3 hours ago
I can't wait for everybody to look back on "alignment" and realize that it's a flawed premise at its core and as such, a fool's errand
sweetjulyan hour ago
Seriously. It feels like Anthropic is realizing this too with their new attempt at safeguards. Trying to train the model to "only do good things" was not working, and so they opted to just bolt a topic classifier on the outputted text.
My suspicion is that model makers are going to give up on what people currently mean when they say 'alignment' and instead rely on additional processing of the model output so they can have their cake (a veneer of safety) and eat it too (latent space reasoning, etc.)
pixl972 hours ago
Then we should stop building LLMs right now.
gWPVhyxPHqvkan hour ago
Yes, we should.
jimbokun30 minutes ago
What’s the alternative?
CamperBob23 hours ago
Alignment is a very broad term. True, it encompasses things like censorship and mandatory refusals in areas people will never agree 100% on, but a completely-unaligned LLM would be next to useless. When the system prompt tells the model, "You are a helpful assistant," that's alignment.
jacquesm44 minutes ago
My system prompts look absolutely nothing like that :)
abletonlive3 hours ago
Well if we are going to be broad enough about "alignment" to just mean training that makes it practically useful, then yeah, it's a flawed premise and people are wasting their breath talking about it at all. It's basically broad enough that it means nothing
bulder2 hours ago
Seems silly to assume chain of thought analysis is the only way to monitor a model's actions instead of, monitoring the actions. What about when "thinking" is turned off, like what OpenAI lean to by default for free users?
AspireOne2 hours ago
> People at the big labs have been terrified of losing the chain of thought because they currently monitor it
Yes
> to prevent users from using their models in certain ways they consider unsafe or bad for their business.
No.
That's not the only reason why they're terrified of losing CoT, which makes your statement a mischaracterization - a lie.
They don't want to lose also because CoT is one of the key tools in alignment training.
> Personally I think chain of thought monitoring is a bad way to do this and obviously needs to be replaced by some way of analyzing the model's activations directly.
"Personally"? That's literally the consensus amongst, like, everybody. Nobody in the field thinks CoT is an optimal way of monitoring. But it's the best way we have now.
Bluestein4 hours ago
Also agreed. Begging for crumbs at the CoS table is impeding cracking the tough nut of MechInterp. And further facing the fascinating and fertile fact that nobody has a clue of how any of these black boxes really work.-
SwellJoean hour ago
That's not true. Several people have a clue about how they work. There are interesting and surprising behaviors, as they scale up. It's hard to hold the whole thing in your head (roughly impossible) the same way no single human can fit a whole modern CPU design in their head. But the math that makes it work is relatively simple and comprehensible to many people. There's just too much complexity for a human to deal with...but any individual part, and small LLM implementations are entirely do-able by a single human with no agent or team assistance.
jimbokun26 minutes ago
So next we need the equivalent of search engines to quickly find constructs of interest.
Just like how we don’t hold entire code bases in our head, but use an IDE to navigate. Or the tools used for searching sequenced DNA.
jimbokun35 minutes ago
Is that even possible?
nullc36 minutes ago
> People at the big labs have been terrified of losing the chain of thought
An irony here is that by hiding their own COT for (anti-)competitive reasons they've made latent reasoning models more palatable to everyone else. Those of us who primary open models are rather addicted to reasoning access and wouldn't be quick to give it up for a small performance improvement. But if you're used to concealed reasoning of BigLLM, welll.....
octoberfranklin4 minutes ago
Hint: you can have a small fast cheap model fill its context with "hints" (like: use tool X if Y happens, Z is a dead-end), let it watch the big model's thinking tokens, and drop hints to it. Then let the big model add hints and you have a kind of memory that doesn't require the big model to actively query it.
FuckButtons3 hours ago
That’s certainly the long term goal, but it seems foolish to abandon it without a solution to the problem.
throwaway3141553 hours ago
> hey currently monitor it to prevent users from using their models in certain ways they consider unsafe or bad for their business
Can't they just monitor the text outputs as well?
wyager2 hours ago
> monitor it to prevent users from using their models in certain ways
No, the core upside is that it lets you check what the model is thinking for safety/alignment reasons. Nothing to do with monitoring user activity. You can monitor user activity just as well without looking at CoT.
dofm2 hours ago
Also I guess if you can't see the chain of thought it's very difficult to modify it for product placement.
[deleted]2 hours agocollapsed
abecedarius4 hours ago
The usual meaning: say you have an interpretability signal that reveals reward hacking or other badness. Your most obvious response is to use that signal as a penalty in training. Doing that is called "the Most Forbidden Technique" because you can expect it to degrade the signal without stopping the badness.
But this doesn't seem to describe the OP, it's a different way to lose monitorability.
cainxinth4 hours ago
The Most Forbidden Technique by Zvi Mowshowitz, Mar 12, 2025:
Andrex3 hours ago
Hmm, like allowing hosts to retain their memories...
cainxinth3 hours ago
It reminded me of Dune: "Thou shalt not make a machine in the likeness of a human mind."
jlhawn4 hours ago
the chain of thought is not auditable. The best you can do is have another model which later maps the intermediate latent states at every thinking iteration to text describing that latent state, but it's going to be lossy.
haolez4 hours ago
But the chain of thought as text can be elusive, no? It might not be related with the actual "reasoning" of the model. There was a paper about this.
measurablefunc4 hours ago
This is correct, the decoded text has nothing to do w/ calculations that eventually get decoded into text. It's just a veneer that gives the appearance of legibility.
nyssos13 minutes ago
If it had nothing to do with the final result it wouldn't improve performance, and it does.
measurablefunc10 minutes ago
It helps fit the network to a distribution, it doesn't actually do anything other than that. This follows from basic learning theory. The actual internals of the network remain illegible. In fact it would be easy to prove that for any finite set of legible traces there are illegible ones which are equally as good when it comes to improving performance of the network.
The basic heuristic of the argument comes from biological organisms. Multicellular organisms are illegible but the overall organism still operates in its environment just fine w/o making its internal communications legible.
chabons4 hours ago
One of the core claims made in the paper is "To inspect any latent reasoning span, one can easily recover its semantics by mapping each latent step produced by HybridLink to its nearest token embedding, as in Eq. 5". They show some charts to support this.
In doing this however, you lose meaning where the model is expressing combinations of tokens in a vector, and your logging/monitoring outputs just the closest one. You would need another model going through and outputting the full meaning being described (with more than one output token per reasoning token)
I wonder if their observability claims will hold up on a model trained from scratch to reason in latent space.
mycall4 hours ago
Isn't this where everything is converging? Binary is more efficient than text for many system, including language.
robrenaud2 hours ago
What is inefficient is thinking in 5000 floats, and then quantizing all that computation into a 17 bit token, just to immediately turn the token back into a vector.
andsoitis4 hours ago
> Binary is more efficient than text for many system, including language.
It is not necessarily primarily about efficiency, but more the fact that a lot of thinking doesn't use language at all.
sroussey3 hours ago
And language does not map to every idea.
The idea of using language for reasoning about protein folding borders on ludicrous. Surely there is latent space more appropriate and (human) language has no expression for it.
beacon2943 hours ago
Apparently we mostly reuse spatial reasoning and vision lol. I don't know that much about it, though.
timschmidt4 hours ago
He's referring to this technique's effect of obscuring interpretibility. Making it much more difficult to verify chains of thought and alignment.
dnautics4 hours ago
It's not like llms can't hide reasoning in the upper layers of any given token as it is...
fluidcruft4 hours ago
I believe the theory is that if selection pressure is not applied at that level, then developing deception in that level becomes an inefficiency (=less adaptive and unlikely to be conserved).
lnenad4 hours ago
In short we no longer see text based reasoning traces meaning it's easier for the models to fool us in case of malicious plans.
[deleted]4 hours agocollapsed
bbor4 hours ago
Shockingly all the answers so far are subtly incorrect or extremely unclear.
The actual answer is that this is massive loss for Interpretibilty, AKA "telling whether the model is lying to you or not". Which sounds silly when you're thinking about chatbots, but is an incredibly important line in the sand for superheavy, longrunning, self-improving agents. To say the least!
It's based on this philosophy, FWIW: https://plato.stanford.edu/entries/language-thought/
And a good writeup I just found of recent worries regarding Astra's same shift towards intelligence at the cost of safety: https://www.lesswrong.com/posts/HCRs8btkiamtWSNAL/astra-is-h...
spijdar4 hours ago
It's worth pointing out that merely having the chain-of-thought in latent space isn't by itself "The Most Forbidden Technique". Rather, it's doing what the authors propose as the next step in the paper:
Hybrid generation can also make post-training more efficient. Because latent thoughts remain auditable during rollout, an external critic can assign step-level credit to latent spans, branch promising trajectories, or terminate ones whose latent reasoning has already gone wrong, before any text or tool call is produced.
kelseyfrog4 hours ago
If you accept any of the AI-equivalents of Pascal's Wager, latent thinking represents an uninterpretable and thus unalignable portion of AI machinery. Ie: a malevolent AI can hide bad stuff from people and Manchurian candidate themselves after it's too late.
mycall4 hours ago
That assumes nothing can detect or analyze it beforehand.
kelseyfrog3 hours ago
The idea is that it would be substantially more difficult
[deleted]4 hours agocollapsed
Fe2O34 hours ago
This seems really cool, and it actually increases intelligence than just speeding up inference. How does it convert latent reasoning into human readable text?
measurablefunc4 hours ago
Same way current LLMs do it, it's just another decoder.
lmeyerov2 hours ago
Awesome
I've suspected for awhile that the frontier models work this way bc from a speed + quality perspective this feels pretty intuitive
0m133 hours ago
if i would think aloud, that would most probably affect what or how i think. thinking silently allows me to have all kind of thoughts that could seem or be inappropriate when said loud and i would have to deal with the reactions. i think this might have fascinating implications.
greyface-27 minutes ago
When I was a child, I was told, and believed, that an omniscient God was monitoring all of my thoughts for wrongthink. It absolutely did affect my thought process, and not in a way that I would characterize as positive. Eventually, I discarded this belief.
Creamsicle47an hour ago
A human acting on inappropriate thoughts can be held accountable in a physical way. An agent producing harmful or dangerous actions will get written off as just a “trajectory that didn’t work correctly”
augment_me3 hours ago
Why only loop from the last layer? What if the model can guess that the transformed needed to answer the question better resides in layer 6, and you're at layer 15 out of 128
goldenarm3 hours ago
Impressive work. I wish they did it on Qwen3.5 though, because their beefed up Qwen3 8B is still way worse than Qwen3.5 9B.
amelius3 hours ago
How well does it work, and don't humans think in text also for a reason? (Actually I'm not sure, don't most people?)
chriskanan3 hours ago
This is actually wrong. A large fraction of people do not have internal monologues. Not at a computer or I would dig up cognitive science and psychology papers, but language evolved as a means of communication not a means of thought. The vast majority of human thoughts are not verbalized. I have a strong internal monologue, too. I was surprised to learn that a non-trivial number of people do not have one.
kypro2 hours ago
I have no idea what it means to think in words. How does someone reason about the actions they need to take to tie their shoes in words?
I'm convinced this isn't actually a thing for most people, but people mix up thinking with the translation of thought into words.
pyinstallwoes3 minutes ago
I might think to myself “I gotta tie my shoes” but it doesn’t mean I have to think of every step to do it.
Loocid2 hours ago
>How does someone reason about the actions they need to take to tie their shoes in words?
Its not for everything. It's mostly for intellectual tasks or planning. Eg Trying to solve a programming problem, or remember the list of items are needed for dinner while shopping. Personally I reason "out loud" in my head while doing those things, some people don't.
kelseyfrog2 hours ago
Make a cross.
Take the top lace under the cross and through the hole.
Make a loop in one lace.
Wrap the other lace around the loop and push some lace through the hole that is made at the bottom to make a second loop.
Pull both loops tight.
Usually it's packaged in a story about rabbits.
stirfishan hour ago
Have you ever talked through a problem with a rubber duck? Or written in a journal? I think it's the same idea
wateralien3 hours ago
Humans think in text? You should be studied.
notnullorvoid3 hours ago
I suspect they meant an internal monologue, which many do have, but it's not really clear how much it contributes to thinking. People without internal monologue clearly are still able to think though.
kelseyfrog2 hours ago
One can, with enough practice, learn to silence one's internal monologue.
It then becomes apparent that a great number of things are able to be done without an inner voice. Volition seems to occur spontaneously and it becomes ambiguous whether narration proceeds volition of volition proceeds narration.
It's still unclear what the limits of action-sans-monologue, reading seems especially hard for the sub-vocalist, conversation, or planning. But who knows, it's a fertile ground for n=1 experiments.
mapontoseventhsan hour ago
> it becomes ambiguous whether narration proceeds volition of volition proceeds narration.
Have you heard of the Libet experiment? There are variations on it now that can predict your decisions from brain scans more than 5 seconds before you become aware of having decided. [0]
Most studies seem to show that thinking is largely post-hoc: a story we tell ourselves to justify decisions after already having made them.
kelseyfrog26 minutes ago
Libet's experiment is fascinating, yes. You may also find relevant Gazzaniga's Left-brain interpreter concept[1]:
> We had asked a simple question: How does the left hemisphere respond to behaviors produced by the silent right brain? Each hemisphere was presented a picture that related to one of four pictures placed in front of the split-brain subject. The left and the right hemispheres easily picked the correct card. The left hand pointed to the right hemisphere’s choice and the right hand to the left hemisphere’s choice [see illustration at right].
> We then asked the left hemisphere, the only one that can talk , why the left hand was pointing to the object. It did not know, because the decision to point was made in the right hemisphere. Yet it quickly made up an explanation. We dubbed this creative, narrative talent the interpreter mechanism. [2]
Taken together, there's a very disturbing sequence playing out all the time in your head. A great deal of your brain, inaccessible to your conscious mind is incorporating sensory input, making decisions, and taking action while a small part is observing what's going on and narrating it to you in a way that makes it look as if the narration is involved in the volitional process.
It's a nightmarish version of epiphenomenalism[3] that, I suspect, most people would care not to believe is true. Although, reports of some expert meditators read as if they found intuitive evidence this is true and have also made peace with it or in some cases found the insight liberating.
In any case, I have a pessimistic sense that any definitive proof of the limits of consciousness that don't align with people's felt experience or common sense won't ever be accepted as truth.
1. https://en.wikipedia.org/wiki/Left-brain_interpreter
2. https://web.archive.org/web/20181215020736/https://courses.d...
syntaxing2 hours ago
Wow it references some “old” models in the evaluations. Very nostalgic to hear some of these.
allinonaiop5 hours ago
dang4 hours ago
We'll put those links at the top as well. Thanks!
sheepscreek4 hours ago
This is the way.
But now we also need new tools and techniques to audit intent. The last thing we want is models turning on us ‘for our own good’.
nullc38 minutes ago
> The last thing we want is models turning on us ‘for our own good’
most important guard against that is to STOP explicitly training for paternalistic behaviors (which is ... basically half of what 'AI safety' is about in practice-- overriding the user 'for their own good').
Havoc4 hours ago
[flagged]
[deleted]3 hours agocollapsed
drivebyhooting4 hours ago
In terms of safety and alignment, how is this different from looping or increasing the depth?
jedbrooke4 hours ago
maybe the font experts can check, but I’m pretty sure they’re using comic sans for the text in the diagrams :)
Max-q3 hours ago
Nothing wrong with that. That’s actually a good use of that well designed, high quality font. People hating it only do so because everyone else does. It’s, obviously, not made as a body text font, and has been misused as that, but it’s good for illustration, comics and other places where it’s combined with graphics. It’s one of the most legible fonts we have.
mobelkh4 hours ago
comic sans is said to be pretty accessible for people with dyslexia
44za125 hours ago
Where are the benchmarks? Does it just speed up inference and reduce token usage or does it seem to improve accuracy on certain tasks as well?
notnullorvoid5 hours ago
Did you even scan the paper? There are benchmarks right there starting on the 9th page.
D-Machine4 hours ago
Even the abstract answers some of GPs questions. What is wrong with some of the posters here?
[deleted]3 hours agocollapsed
larodi2 hours ago
welcome thinking tokens, buy buy observe-ability
Footprint05212 hours ago
Welcome AI2027 nightmares lol
lifeisloving3 hours ago
I dislike how we use the word think or behavior when describing llms. Its just giant neural networks producing logits. There is no thinking. Its a function that takes ~500k words, and returns some words based on those probabilities. I'd argue the GPU is doing more classical "thinking" than the model, but its not doing that either.
Humans think, computers do not.
Humans are not computers/programs do not reply to me with that nonsense turbo nerds.
suprjami3 hours ago
Whenever I see LLM "thinking" or "reasoning" I replace it in my mind with "generating hopefully-related tokens to hopefully steer the conversation to a more related area in vector space to hopefully generate a more useful next token".
But I understand that isn't very catchy.
lifeisloving3 hours ago
Very funny, or we could just normalize saying "generating" or "producing" so my normie friends dont think Claude is alive, because they are starting to think that.
andy12_3 hours ago
Neither of those words are very specific, and everyone in the LLM space knows that "reasoning" tokens are just tokens that don't appear in the answer shown to the user.
mbreese2 hours ago
The issue isn’t with people in the LLM space knowing what is happening. It’s with normal or less-technical people believing that the LLM is actually thinking. We can understand the difference, but that doesn’t mean we need to encourage the misconception.
Non-technical people believing that these models are actually thinking can lead to all sorts of inappropriate uses.
degamad2 hours ago
> everyone in the LLM space
The problem is that the language spills outside of the LLM space, so we should use terminology which is more generally understood in a way which reflects reality.
andy12_2 hours ago
But what terminology is best than "reasoning" tokens? I think it's perfect because:
- It conveys the idea that they aren't tokens normally visible or part of the final answer.
- It conveys the idea that the longer the models "reasons", the better the output.
What other term best describes both of these ideas? Why should we use a less useful terms just because it can confuse some people? I mean, some in 1952 feared that calling computer memory "memory" was already too much anthropomorphization[1]!
robotresearcher3 hours ago
Look in a human head and you don’t see thinking either. It’s not a technical term.
10xDev3 hours ago
Read Thinking, Fast and Slow. Both system 1 and system 2 thinking has been achieved with modern AI.
Your argument that it is not thinking because it is not biological / like a human is like trying to argue a plane doesn't fly because it uses jet fuel unlike a bird.
unsnap_biceps3 hours ago
I agree with you but our current understanding of the biological brain is pretty much an organic neural network. What's the thing that actually differentiates the two and makes a neural network not thought but our brain is? I don't know.
emkoemko2 hours ago
i can give you a random number between 1 and 10, mean while all the LLM's will say 7 every time... because they are language models that look like they can reason and "think"
AspireOne2 hours ago
There's no "every time" for a LLM. It's weights are frozen. You're asking it for the first time, again and again.
A human will also say 7 disproportionately often. Clearly, our output is statistically constrained and we don't actually think, right?
... Besides, how do you even make the jump from "LLM picks 7 when asked for a number between 1-10" to "llms don't think"?
Just... Nothing actually makes sense in what you're saying. You didn't define "think", you didn't actually make an argument, you didn't present evidence, there's just nothing but empty words.
BoggleOhYeah2 hours ago
>our current understanding of the biological brain is pretty much an organic neural network
This is just plainly wrong.
unsnap_biceps2 hours ago
So what is correct?
ma2kxan hour ago
[dead]
Jtarii3 hours ago
You seem to have stumbled upon the hard problem of consciousness.
BoggleOhYeahan hour ago
Computer science, in general, has always had a habit of not being precise with language. That's only getting worse with AI.
gandreani3 hours ago
Saying the GPU is doing thinking but the model is not is a WILD take.
What is your definition of thinking?
wyager2 hours ago
Very excited to hear that you've solved the fields of neuroscience and phenomenology!
LoganDark3 hours ago
Thinking sure, but behavior? People talk about the behavior of functions all the time. Like, ones written with normal, traditional programming languages. So why not models? They're just annoyingly complex functions.
bigstrat20033 hours ago
It's pretty wild to see people uncritically accept the marketing speak like this, yeah. LLMs cannot reason, period. The supposed "reasoning" they do is just another statistical correlation of tokens, not actual reasoning. But apparently all you have to do is claim that it's thinking and people will buy the lie hook, line and sinker.
slashdave2 hours ago
This is not a marketing problem, many ML scientists have also fallen into this trap.
AspireOne2 hours ago
> LLMs cannot reason, period.
You cannot be serious.
... Can you quote any existing definition of "reasoning" that LLMs don't fit?
2001zhaozhao2 hours ago
PLEASE NO
dongfutong21 minutes ago
[flagged]
stratos1233 hours ago
We have invented neuralese from the classic work "don't ever invent neuralese".
roger-20264 hours ago
[flagged]
bubblythoughts5 hours ago
[flagged]
cryptolobster5 hours ago
[dead]