Hacker News

This slot machine has access to your bank account and can decide how much to play on its own!

You don't have LLM-based processes if you think there is no lock-in. There may be no lock-in for coding if you enforce decent rules (but still some ambiguous docs can be interpreted differently), but any non-trivial pipeline/system, these models are not stupid but each has some quirks. Sometimes for some reason they will ignore some instruction while all other models have no trouble following it. These things accumulate.

Plus there's subjective stuff even for coding, people learning how to deal with it. Even on HN you can already see cloude/codex camps each strongly convinced that one is better than the other.

My employer just finalized a contract with Anthropic, for enterprise Claude Code use. Which means that unless there is a _major_ downgrade in service quality, we are now locked in for the next few years (but at least for a year, although vendor contracts are renegotiated less frequently).

Just checked the dashboard, and we seem to have the exact same $200 credit as others, enterprise or not. Token inflation affects us just like everyone else.

It feels a bit like buying the same box of chocolates every day, but the size / weight of the box is shrinking... the price remains unchanged!

The disconnect between the reality of and the consumer sentiment of this particular realm of products seems to be one of the most dramatic and widespread I’ve personally ever seen.

You'd also expect that in a market where the players think they're in a bubble.

Anthropic can't afford to pull shit like this and sacrifice quality.

And yet, the Java language exists.

For market share, ANTHROP\C needs to optimize for the vast mediocrity that are mid-bell-curve users and enterprises.

This adaptation tends to come with significant drag for the right ends of the bell curve firms or teams.

Unless ANTHROP\C have a separate objective function by cohort and ensure that doesn't regress, improving results for the emerging middle will nerf tools from point of view of those with high in-domain expertise.

It’s really weird when you go to one of the open models and suddenly the same context window stretches nearly 3-4 times as long.

Then it tells you nothing will break, spend a bunch of tokens on the migration, hit a wall, having an oops moment and telling you it made a mistake assuming a key fact instead of verifying and present you the option to roll everything back or rewrite the rest of your system.

By contrast, I keep trying to find an incantation that can get it to AT LEAST read the comments surrounding a code target before a grep replace if it isn't going to read more than the first 60 lines of any doc. (Btw, the receiver of a -tail 60 doesn't seem to know it was cut, it insists it "read" the whole file.)

It seems to me ANTHROP\C have harnessed hard for not spending tokens to bring content into context. I wish they'd left us a "LEROY_JENKINS" flag: read and think before you code. In Claude Code anyway, it appears to default to:

  export CLAUDE_CODE_LEROY_JENKINS=true

Interesting that I have the exact opposite experience with Opus 4.8 being nearly unusable dumb in the past couple of days. I was trying to explain this as the new Sonnet release announcement may have overloaded their systems again, but let's see in a few days. Right now it hurts more to my workflow than helps.

Is this a new thing? At least I only noticed this recently that instead of looking at sources now it prefers to decompile and read java byte code for some reason.

Because they still haven't fully captured the market for Agentic Development.

Did Anthropic have Opus 4.8 and Sonnet 5 switched in the Agentic Search chart at first?

Most benchmarks notoriously undercount multi-turn instruction following too. And it's where open source models (and Gemini) lose big time.

I would expect a model's result each time to be of a similar quality to the other times. There's something wrong if it does a way better or worse job, at the same problem, sometimes. It's possible, but I haven't heard anyone saying that they do.

At least we quit with the "i asked it this question and here's what it said" comments. They were truly awful for the first 6 months or so.

Or the "I have my own personal benchmark..."

"Claude and its political bias thinks the supreme court should..."

Some napkin math -- total global labor compensation is about 50% of the GDP, which puts it in the USD 50 - 60 Trillion range: https://ourworldindata.org/grapher/labor-share-of-gdp

This source claims that knowledge workers alone (probably because they are paid much more) account for 35 - 50 Trillion of that: https://github.com/danielmiessler/Substrate/blob/main/Data/K...

If LLMs can boost their productivity even by an average of 5% (studies from ~2024 put it in the ~30% range depending on task) that is ~1.5 - 2.5T in value annually. Even if the AI industry can capture a fraction of that, that is a huuuge monetization opportunity.

Note, at 5% productivity boost, humans are not just in the loop, they are the loop. AGI or large-scale replacement of humans is not even needed, but the financial opportunity is already immense, and it scales with how much human productivity can be improved (i.e. how much work can be offloaded to LLMs.)

Now, I don't think AGI will happen soon (or has already happened, depending on how you define it) but I do think humans will be a much smaller part of the loop and large-scale job displacement will happen once companies figure out how to properly use AI.

At this point, the financial upside for the AI industry is extremely high but will be limited by the social turmoil that will inevitably ensue (which we're already seeing brewing in the data center backlash.)

That's a really good point. I think if there wasn't the insane amount of money involved and these were treated as tools instead, they would probably be MORE productive. I think a person working hand in hand with an AI instead of delegating is the sweet spot of making things fast while also not losing understanding or control of the system. You are absolutely right that these companies can't justify their valuations if they do that though. I just got a new mac to run models locally, and so far the results have been positive with some small hiccups. I'm thinking the future of this tech will likely be better tooling with better IDE integrations rather than "Claude plz make me a SaaS kthx"

Dario has publicly claimed each model has been profitable, even accounting for its training costs; it's just that each new model is exponentially more expensive to train than the last, so the income lags and it looks like the company is losing money overall.

Now, we can't know if this is true unfortunately, but it's not directly contradicted by anything that's known publicly at least. I thought it was an interesting way to frame it and makes the whole situation look marginally less bad.

My two cents is that the way to square this circle is that the valuations should be lower and they should be spending a lot less on constant retraining.

Unfortunately (from my perspective) it seems like the US companies are increasingly stuck in their current model. I think it's a competitive disadvantage.

But obviously most of the real insiders seem to disagree with me, so I'm probably wrong :)

At some point it's going to plateau, maybe already has. Then they will switch to FPGA/ASIC-based model-specific hardware for lower consumption. I'm pretty sure the "space data centers" won't use GPUs, they are not radiation-tolerant whereas FPGAs can be.

https://www.cerebras.ai/blog/gemma-4-on-cerebras-the-fastest...

> no way to justify their valuations if they get downgraded to a pair programming tool

I think there is. Pair today doesn’t mean they’re locked into that forever.

> There’s no way to justify their valuations if they get downgraded to a pair programming tool.

Honestly I still don't see how they justify their valuations, period. If anything they're serious liabilities.

Open-weight models are improving and reaching "good enough" levels for more and more tasks. They're also known quantities; you know what you're getting with them and don't have to worry about the model silently (or not so silently) being switched out from under you (whether that's because Anthropic/OpenAI decides you're not worthy of their latest and greatest for one reason or another, or they switch you to a quantized model to save on compute, or they simply sunset the specific model you've been relying on).

And if the open-weight model doesn't run on your local hardware already, there are any number of hosting providers that will handle that for you (so you're back to just paying for colocation/cloud usage instead of nebulous tokens).

Closed models are improving as well, sure, but diminishing returns will eventually kick in (as they already have for various tasks, as I said).

So if not their models, where does their value come from? Just simple network effects/lock-in? "Normal" users will drift to other options if they start showing more and more ads, and enterprise customers will surely be looking for opportunities to avoid lock-in and reduce risk.

I think the last argument I've heard is that these valuations are basically a bet that Anthropic and/or OpenAI will achieve AGI that can fully replace human labor, so they'll essentially be able to sell that replacement labor to everyone. They haven't managed to pull that off, yet, however. Businesses that have tried to replace humans almost immediately realized either that the AI's capabilities were oversold or that they at least needed a human in the loop still, to some degree. And even if they do achieve AGI, that would surely become an issue of national security (they're already flirting with that today), so who's to say governments won't simply nationalize the best AI labs and either remove them from the economy entirely or perhaps even provide models as a public service to level the playing field?

That all sounds like a giant gamble, if anything. And it's incredibly frustrating to watch as someone that's been unemployed for a year because (a) budgets are being burned on tokens and (b) LLM-generated applications are flooding hiring teams and preventing real people from being seen. (Not to mention, as someone that spends a lot of time in gaming circles, the fact that DRAM and flash storage is quickly becoming inaccessible is just an additional frustration that means people can't even find temporary relief in entertainment.) I can only hope this bubble finally implodes before I lose my house.

They own the means of production for the leading models but they're far from monopolizing them since the techniques are well known. At this point it's a matter of having a head start and lots of capital to pay for the data annotation and GPU time to train them. Others are playing catch-up but they're hot on their heals which is the biggest reason for them to continue spending like crazy to keep their leads.

For the non-bleeding edge they have a lot of competition with more competitors showing up every day.

The way this is playing out is not surprising, it's similar to any other technological breakthrough as it becomes commercialized. Eventually those means of production will become commoditized as well.

these are capital intensive commodity businesses. They can be plenty big - see railroads or airplanes... or refining... but that doesn't mean that most value won't be added elsewhere.

I find these nefarious intention theories shallow. It can both be the case that the endstate is them owning the means of production without that being the intended guiding goal. Companies can chase profit without being Leninistic boogeymen.

> What is your motivation? Privacy and/or data protection?

For me, those things are nice benefits but not the motivation. It's just a desire to own my tools and remove the magic wherever possible, and not pay a big corporation who is constantly tweaking/changing things that I might not like. It's the same reason I migrated everything I host off of Azure and onto a VPS, and why I moved from Jetbrains Rider to Neovim, and so on.

You can let a local LLM run loops all night or all weekend and it costs nothing. It works offline. There's no rate limits. Privacy. You aren't tied to the whims of the US Government banning models. You can tie it in to HomeAssistant for home automation tasks very easily.

I'm using 4-bit as well, with the MoE model. I also use the MLX versions which are optimized for Apple CPUs (from what I understand anyway, I'm just an LLM layman). According to my oMLX dashboard, I'm getting about 50 tokens per second out of this model – not blazing fast, but more than fast enough to be useful to me.

https://huggingface.co/mlx-community/Qwen3.6-35B-A3B-OptiQ-4...

[dead]

I didn't realize GPT 5.3 Codex was that good.

OpenAI claims to have made their new Terra model as good as GPT 5.5, but with half the cost per intelligence. Hopefully, this will bring it closer to the price you're expecting (or even better considering GPT models have good acceptance/success rates according to benchmarks).

Buy 5 accounts at 20usd each. It’s 100 and lasts decently on single threaded work

In my experience it spends a lot more tokens to do things. I wrote a tiny extension for omp that counts the number of "Actually" in the response, and if it exceeds a threshold stops execution and waits for me to tell it what to do. Even then it frequently just ignores basic instructions like "only write boilerplate, I will fill in the functionality"

Imo MiniMax and MiMo are a lot more reliable (and cheap)

Not opus level, but close enough and cheap enough to get the job done

Like?

A suppose a solution might be going with a customizable harness like pi and merging Anthropic’s system prompt with a personalized custom one to remove all contractions

We're speed running HAL 9000

This is why Fable was so good. It followed instructions and it was in no way lazy.

I wasn't working with anything sensitive, but it really does feel like it sometimes condenses even something low like three bullet points to two.

IMO, they were quite good with checklists even a year ago, and tried to tick off each one.

It isn’t a dream, it’s a reality for some of us here and it will be increasingly so for everyone else. Amazingly, USG intervening slowed the dynamic greatly (fortunately?)

The problem is obviously who will be left. There’s a lot of scifi to catch up on.

I think that they are simply evaluated on prompt to solution benchmarks.

I haven't. Thanks for the heads up will give it a try! I use opus to comment on code design quite often though. It became a pattern that I made a skill for me to ask for second opinions https://news.ycombinator.com/item?id=48733092 Would love to hear your feedback if you don't mind!

Fascinating! How did you learn about this?

Not in my experience, it tends to pick up subtle orientations given in a question (like "which is better, A and B?" and in the context you add you list a few things for A and B) and will absolutely run with them even if they're not true. Has been an issue with Gemini models since at least 3.0. Maybe that makes them great roleplaying models, but for factual information they just run with the slightest hint in one direction or another and never really push back objectively.

I get similar results in my own tests. And Gemini 3.1 Pro is consistently on top of my ratings. Not everyone is coding monkey, I prefer staying a programmer.

If you were right, the karma system would largely take care of this. It really sounds like this is more of your personal view

I'm biased because I run an inference company, https://synthetic.new. That being said I think we're pretty good at serving at GLM-5.2 — and other models, like Kimi K2.7! — and our privacy policy is quite good: zero data retention for prompts and completions on API requests. Our average streaming TPS for GLM-5.2 (aka, tokens after factoring out time-to-first-token, which varies based on geography) is 97tps over the last 24hrs, although it's slightly lower at peak traffic in the mornings PST where it's 50-70 tps. We're also subscription-based which is nicer for coding than e.g. Fireworks which is per-token billing.

Fireworks.ai is solid. And if you care more about speed than cost they have a "fast" variant that I think just throws more hardware at the model for about 2x the cost.

Run it on Amazon Bedrock or GCP vertex. No problems at all.

> the Chinese one won’t reply to subpoenas so thats a value add tbh

That's not something that's definite. They are not quite like the Russians. A lot of the governments in Asia are overly pragmatic and will happily strong arm their companies to throw users under the bus for the sake of a trade deal. There's a reason why Snowden ran to the Russians and not China.

Also, if they have any subsidiaries in the US, they may not have a choice in the matter.

This is interesting, I haven’t actually heard you suggest that the labs are focusing on this benchmark before. Have you come around to this position as a result of the quality of pelicans you’ve been getting?

The reason I thought this was an interesting benchmark is because it’s a non-image generating model creating an image using SVG code, so it kinda spans capabilities.

If an AI lab trained a model specifically for animals riding bicycles it seem trivial to modify the prompt and determine if it was trained specifically for that or if it’s generalized a skill and can also generate a proper orangutan walking on stilts or an armadillo on a skateboard, this sort of thing?

That's one possibility albeit quite charitable. I'd be inclined to think the same personally if GLM 5.2 wasn't also rocking in other areas too.

That's something that Sonnet 5 couldn't get right even on a static SVG.

That's only correct for specific models and not what parent was referring to.

Stable Diffusion 3, an open weights model, was laughed at at release for not being able to even generate a woman laying in grass. The community attributed this to the heavy dataset filtering. Since then other open weights releases have been made with no NSFW capabilities and the community claims they're not as good as anatomy as well.

You can google "stable diffusion 3 woman in grass" and press the images tab to see how the model failed spectacularly.

There are image models with censorship at every stage from pretraining to posttraining.

Most recently Ideogram released an open weight model that will denoise into a grey image with the text "Blocked by safety filter" notice for certain prompts

Of course, because it's open weights people have found defeats

Sure, but I think doing it this way allows them to later on say they were transparent about it. Completely hiding this would make it very difficult for them excuse when getting caught.

I would indeed be more inclined to use it if the tokens per second were better. Though I would be then using their more expensive Opus less though. Perhaps it is strategy.

I think I might have written a comment similar to yours maybe 6 months or a year ago. I'm not quite sure to respond to these sorts of replies. I have used LLMs/Claude Code quite extensively professionally and was a very early adopter, have built tooling around LLM/agentic development, and genuinely embraced it. They aren't useless, but the short term gains you think you're getting come at a very steep price that you may not actually account for consciously for quite some time, if ever.

The skilled seniors better stop downplaying what actually led them to be skilled in the first place, and realize that the conditions to develop that skill has been gone and almost deemed unproductive in today's workplace.

Not disagreeing that LLM's are a force multiplier, but I highly doubt whatever value will end up finding multiplying in the next generation of seniors, at this rate. It's surreal to me that I have to point out that recognition AND recalling are both necessary components of skill acquisition, because humans largely knew this since the dawn of education.

I believe you are miscalculating the effect of skill atrophy, there is benefit and actual experience gained by doing the work yourself. You are an experienced dev and already have a lot of tools and knowledge under your belt so at the moment it is hard to see the actual issue, as this is just a productivity multiplierfor you. But give it a couple of years working under these conditions, your tech savvy nature will be severely diminished.

Have you really found claude to much more more capable than eg deepseek? Anthropic has little to no chance of producing a competitive business model in the long term.

Finally, someone said it. 20+ years in software and my productivity and velocity is wild right now.

you're right, but if skills atrophy so will the efficacy of the tool.

I am another skilled senior, have been coding since I was 7, although you have a few more years of experience on me, and am commenting here just for the goldilocks moment, as I have read and reflected on both of your comments and find my reality is somewhere in the middle.

On personal projects, where I am in charge of all the hats (product development, UI, UX, backend, security, server admin, etc) -- absolutely crazy force multiplier. You get a nice suite of backend and e2e tests running, with full business scenario layered on top of that, and constantly running agents to do the coding, another agent on a higher level of reasoning to review that work, and sometimes occasionally poping into another competitors model to review their work just for added comfort -- it feels like wizardry. I am not vibing it, but I wouldn't say I am carefully scrolling through every line. I review whats fundamentally important, especially when it comes to data, overall structure, and large, cross cutting concerns, but I would be lying if I say some code doesn't land that I don't read. But I have the security of the test suites and validations , so I pour more effort into that.

It's a nice self reinforceing loop.

All of this might sound like I agree with you, and to some extent I do, but I am realizing as the apps I have built out like a cannon shot out of hell with tremendous speed and polish right out of the gate are starting to slow down. Feature adds are getting more complex. My memory is not what it used to be. Each run and pass through the code consumes more of my tokens and limits. I am starting to do less in the same amount of time. Codex did a vertical slice of a feature for me (well defined and well planned). It contained functionality that has historically plagued us developers -- the dreaded time. I used xHigh GPT 5.5. It had obvious bugs, but I wanted the robots to catch it. I popped it in claude (on the new sonnet 5! heyo!) -- Claude caught the bugs. Even said they "immediately stood out" I wondered how this happened. Frontier model from company A was evaluated by workhorse model from company B. All of this again took massive amounts of usage. And time.

And this is -- best case scenario, perfect world, everything is in perfect alignment.

Now for the work reality.

Multiple product and experience owners. Multiple dev teams. Different enterprise teams support services you rely on. You don't have full unfettered access to frontier models. You have to use copilot, or some other enterprise harness, and you run out of credits for the month, you are SOL. It's not as good as your claude, you think to yourself, but hey, its familiar enough, and you have 5k credits left for the month for Opus 4.8, better make the best of it. But now you burned half of them working on that Transactional Bug that was mixing synchronous and asynchronous semantics that the other guy's model should have picked up on. What happened? Maybe he didn't use Opus, maybe he used Haiku, maybe his prompt was bad. Who knows. Gotta fix it. Oh, you gotta reach across the isle and put in a request to get the Enterprise team to look at this caching inconsistency on user data that you need and is really the source of your race conditions. Tick tick tick. Model limits approaching. You start wondering if you just did all this by hand like "in the old days" would you have got it done correctly faster? Or at least, cheaper. You'll never know.

> MFIC

Getting another agent to validate the first agent is a tower of sand.

> my velocity is apparent at https://github.com/pmarreck/)

Forgive me, but the active repos all look like reimplementations of existing good open source code (which of course is ideal training data) - rm_safe has rip for example. Or prototypes. Is there anything that actually has a user base > 1?

We've just done an official evaluation at work, using extensive statistics on our gigantic monorepo in a company with ~2000 devs over the course of 2 years, everyone from hardware engineers to regular old frontend engineers. It's a highly profitable and mature public company, and has been for going on a decade at this point without missing a beat. We were given infinite access & budgets to basically any and all AI tooling we could imagine, and we have several "AI Native" teams (whatever the fuck that even means). We're doing agentic coding, we have harnesses of all kind, skills, we have many teams doing spec-driven development, designers using all the various things like Figma Make and access to tools like Devin/Factory Droid/Claude Code/Codex/etc.

This is all to say, we as a company are using AI a lot in all possible corners, but thankfully our leadership isn't schizophrenic and isn't mandating everyone hit token limits or whatever, it's more of a "Let's see what works and what doesn't" type of thing, and we measure a lot of statistics. Nobody here really cares whether LLMs are the next coming of Christ or not, as a company there are many people (even in SLT) that are indifferent to LLMs, and many who are reasonably hyped.

I wish I could link to the actual document we were all shown since it has a beautiful breakdown of the methodology and a fine-grained breakdown of the stats and the categories measured, but in the grand scheme of things, ALL the AI tooling we have implemented (at least on the engineering side of the equation) has contributed to a total of... drum roll please... 7 (seven) Percent overall productivity increase! The most productive teams saw a productivity increase of around 20%, while some teams actually saw drops in productivity into the negative percentage points. My team, none of us really give a shit about AI and we're somewhere in the 3-5% range on certain categories of tasks, which I'd say is a fairly good assessment.

Productivity here is measured in many ways, including but not limited to speed of MR review and merge times, feature/ticket/roadmap closure/delivery, rollback/revert incidence rate, how often people interact with the MR review bots and implement their suggestions/fixes, how many times people check back on AI transcriptions/meeting notes (hint: Nobody looks back on any of it, it's all just noise that gets generated and never actually referenced outside a few extremely rare cases) and many more things I'm forgetting. It is an imperfect number of course, because measuring productivity in engineering is a sisyphean task, but in my opinion it is accurate to the reality on the ground and outside of all the hype and marketing bullshit.

So, I remain thoroughly unconvinced of these personal anecdotes of people being "massively" more productive, especially once you factor in the fact that we now have a 2000EUR budget/month/dev for all the AI tooling, those productivity numbers start looking pathetic once you factor in the costs (which are only increasing as the AI companies need to start recouping the gazillions they've burned). Some teams have started begging to disable coderabbit and other similar tools in their MRs because they're producing nothing but walls of noise that makes reviewing any MR a nightmare of sludging through endless slop of useless bullshit, ours included.

You very conveniently avoided skill atrophy, the biggest issue for existing engineers to deal with (not even going into topic of cancelling whole hiring-junior-and-raise-senior approach which is just shortsighted and retarded to be polite, but general greed has overtaken the field so its not shocking this went out of window). Everybody using llms excessively is measurably doing worse, ie students. I have hard time believing out industry is somehow immune to that and refuse to do the experiment on myself if avoidable, which it sort of is for me.

Its like drug that will give you few years of great high, and ruin rest of your life afterwards. Use it by all means, I don't care about your output, nobody here does, you do you.

I do care about my long term skills, which aren't about piping some llm outputs. My employer ain't dumb fuck who is pushing for llms at all costs as much as possible. Anyway, most of my day work are processes, discussions, pushing things through - llms can't do a shit here, its personal conversations, connections, often psychical contact to get things done on time. Startup world would be different but I am as far from that unstable environment as I am from say gaming industry, just not worth my time outside SV area.

So I just use llms to verify my coding results, they are fine for that, but I do the creativity. Its by far the best part of my software dev work, why the heck would I be automating that away? Its like automating sex away so you can have more time... reading HN or some other way to just waste time, dumb approach from all angles.

Of course this changes if one is working on personal projects, self-employed, small startup etc but most folks here are not in that category.

Btw, since this morning it's fixed another 200+ crashing test cases.

> Why even bother posting, especially as a reply to a completely unrelated comment?

If you give Anthropic money they will make your life worse in another aspect, it's relevant to all their models. The best principle is to not give money to people who want to harm you.

Anthropic has gone past fearmongering and well into terrorism. I think people on Hacker News should not recommend working with terrorist orgs.

How is that ironic knowing that an authoritarian imperialist country is leading the world in closed models ?

> We replaced all the commercial products our clients and ourselves used with our own

You’ll never guess what product your clients are looking to replace with their own next.

Dario’s own mouth https://x.com/coinbureau/status/2071330294452666695/mediavie...

https://www.judiciary.senate.gov/imo/media/doc/2023-07-26_-_... https://xcancel.com/coinbureau/status/2071330294452666695 https://www.techpolicy.press/transcript-senate-hearing-on-pr...

> "Once the weights of a model are public, they cannot be retrieved. If a model possesses dangerous capabilities, it is permanently out in the wild... We need to consider regulatory frameworks that account for the unique risks of open-source distribution of highly capable frontier models."

I'm a senior skilled developer and I find Anthropic $20 + Open AI $20 + OpenCode Go $10 offers more value than $100 on any particular service.

Juggling between all different models/agents is quite simple with Zed.

A caution about OpenCode Go though, the entire company seems to be run by AI so there's lot of billing related issues with zero support. I subscribe new every month as I lost money due to double payment with automatic subscription.

For non coding related tasks I use local models.

P.S. If anyone is interested to read more about my setup, let me know I'll publish a blog post.

I'm a heavy enough user that I have both the OAI and Anth $200 plans. I always use at least 50% of my weekly Opus quota at Extra setting (meaning I use double the limit of the $100 plan, at minimum). Max I rarely touch because it is twice as slow and the incremental capability gain is minimal. Usually if Opus can't sort something well at Extra, the answer isn't to use Max but to hand the issue off to GPT-5.5 at XHigh.

Agreed I think your strategy is optimal. This is what I landed on as well

This is actually very counterproductive with Opus 4.8 - you are wasting a lot of time.

For Opus 4.8 training with overblown internal dialogue and second opinions - Max effort burns just tokens and wastes time without much value. Spinning wheels.

In practice I don't think any harness (happy to be corrected here!) uses the lesser capability models for writing code. The cost trade-offs are rarely worth it.

They are often used for reading code though.

To expand on this, while the "big model to write a plan, small model to write the specific code" idea is quite common it trips up on edge cases.

In theory the flow works like this:

- small fast models read lots of code, and pass details to the large model to write a plan

- large model takes those details and writes a detailed plan

- medium models write the code

The issue happens when the medium model hits something that the plan didn't take into account (which happens a lot - the big model didn't actually read the code). Then it has to either guess, or pass back to the large model.

If it guesses, the plan usually starts to fall to bits.

If it passes back to the large model, inevitable the large model has to start reading lots of code. In that case you are paying the expensive tokens to read so you might as well have it write the code too (many less tokens are written than are read)

It might be possible to get this to work, but I haven't seen anyone who has tried agentic work with frontier models be satisfied with this hybrid setup.

I'd note that Amp (mentioned above) is probably the leader in using multiple providers in a coding agent but still uses frontier models to write code.

You might want to check out Amp: https://ampcode.com/

Which is your own harness and your own evals for your tasks I guess

I think that's the whole selling point of lovable?

They want to encourage diversifying model use.

Distillation attacks? Volume of calls?

I can confirm this, I too I'm not seeing much of a difference in practice

Wrong! Look at it better. It shows that Opus has superior performance but at higher cost.

Agreed. The graphs clearly show that opus 4.8 performs strictly better at the same cost per task

Yes sorry for the typo

The point is Anthropic has advertised their models in this way. There are plenty of models that can be used in long running situations that have proven to be more capable. Opus 4.8 is not that, and ironic given it's their top public model.

"dangerous cyber skills, such as developing software exploits" is very plainly referring to the same thing you are, but is more precise industry terminology rather than the loaded slang "hack".

I think this is more aimed at the US gov't than anything. They want to be clear that it's not very good at hacking, so that the gov't won't ban it.

I'm sure they're well-aware that this also will make it worse at building secure systems, but the gov't isn't restricting releases based on that.

I think you misunderstood what their vision is, or rather what their possible futures are. They are many steps ahead of almost everyone, both in wargaming possibilities and the actual realized path. What doesn’t make sense to you may be the only safe option for them.

True, but Trump & Co. did give them permission to let Americans continue using it, but Anthropic turned that down (for now at least...).

I don't think so. During the time I was using Fable 5, I was getting it to clean security bugs that Opus 4.8 had introduced ... bugs which weren't localised to a single PHP file but were caused by cascading data flow through multiple PHP files. I'm not an expert on security but I know I wouldn't have found these myself. I knew from day one of Fable's release that it would do thorough security audits and fix loads of flaws, even offering up PoCs to help show that it fixed them, as long as I didn't explicitly ask it to do a security audit. I just said, "My codebase is a mess," and it went on for an hour doing a thorough security audit and helping plug numerous holes. This was before the "fix my code" story came out.

It was already released. US government is the only reason it's not available to us mere mortals anymore

Oh, it done found like 50 of those

How do you know, if you use ther API and don't self-host the full model?

Surely the Chinese government will see US gov's intervention and say "Government control of business is stupid, our industry will have more independence from CCP control for the benefit of the world".

…as long as you don’t ask them about certain dates or squares.

Also, I wouldn’t expect Mythos-class models to be allowed to be openly released by the CCP. Thinking otherwise is pure naivety.

As contradictory as it sounds, they (Anthropic) are probably trying to dance the fine line where its public models can write secure code but cannot exploit insecure code.

Of course. But is it really impossible that Dario’s directive to the marketing team is “try not to make us look bad, but also be honest about our models’ capabilities, so people can stay informed”?

I find it interesting how two different directly opposed messages seem to have both been interpreted as being nothing but marketing speak.

While I'm still not sure I would characterize that as bragging, you're right that that is a fair interpretation. However, another Fair interpretation of that is something along the lines of "the downside or cost of this positive thing is this following negative thing."

I was talking about reporting on testing and capabilities. Yes, open models provide a greater amount of information about the development of the model and how to run it yourself, but I am quite confident that literally no AI company, open or closed, conducts and reports so thoroughly on testing about the capabilities of their models.

"Latitude 41" is a business name?

If you gave me this prompt, I'd say "Which Columbus? None of the ones I know about are at a latitude of 41 degrees north or south?"

https://news.ycombinator.com/newsguidelines.html#comments

This is the same concept as Chain of Thought.

Just that when "native" thinking is used it's hidden from the end result via special tags. If you force a model to reason about a result before producing the result, you get more accurate results.

Because "reason" comes before the selection, it has to think through why it is producing the result beforehand (e.g. produce a block of text that makes the correct answer statistically more likely to be sampled from the distribution. Giving it a property name does influence the direction of the thinking, but it's the same concept. You can call the property "yourThinking" too.

It sounds like you may be thinking too highly/mysteriously about how LLMs work.

At the end of the day they stream completely unstructured text outputs and all behaviors on top of that are just parsing XML-like tags to do tool calls, hide thinking etc (which they were trained to produce in certain circumstances).

There is no special "thinking process" it is a stream of text in the response that is simply wrapped in <Thinking> tags (or similar)

It's a good question, but for multiturn conversations even cached context adds up quickly. My experience has been that spawning off subagents for defined tasks in a large overall plan generally makes me come out ahead.

I'm sure folks' mileage will vary though.

Gemma 4, Kimi K2.5, MiniMax M2.5, gpt-oss, GLM 5, Qwen3 Coder Next, DeepSeek V3.2, Devstral 2, are all available on AWS Bedrock and all are about Haiku level

Sure, my point about "things will eventually catch up to, and surpass Fable" still stands.

Have you considered getting better at coding so you can build stuff yourself instead of waiting for models you might not be able to get access to anymore?

It would be good if Anthropic provided some kind of feedback or even toggle to auto-route requests for models being used at thinking levels that would be a better value using a different model.

Sort of like, getting an automatic upgrade at a car rental or hotel if there is availability.

Having used it quite a bit when it was out, it's not. It's certainly better, but in some ways it's worse. It's trained to be more "agentic" and even in cases where I wanted to talk things through first and I would explicitly tell it not to do something, it would take action on my behalf without checking first.

It's also still just prone to the kind of "stupid" mistakes we see from all LLM's. Like it can write great code, but it doesn't really have common sense without enormous guidance.

I consider myself to be in that cohort as well. :)

The graphs do that already. I was expecting them to try to explain how good it was at simple tasks.

Not the original commenter, but personally I noticed my quota usage didn’t feel like it was being spent at a much lower rate when using Sonnet even on a relatively low thinking budget and based on a few comments here it seems I might not be the only one. Has anyone else noticed this? Wasn’t it different in the past? I thought I would be getting to use Sonnet much much more than Opus but it did not feel that way despite being on 20x plan.

I don't think all the negativity is from crazies, but big chunks of it are certainly motivated. I certainly left out numerous other categories.