Hacker News

The streetlight effect:

> A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, "this is where the light is"

All of your suggestions are better but they're hard, so someone casually evaluating an AI isn't going to do them.

IMHO, It's not the oneshotting.

It's the "starting from empty slate" greenfield that's the real problem.

We used to make fun of Engineers who follow a README on a framework, test it on an empty project, and say "this framework is the best for our 10 year running production app". Greenfield mentality is always the solution to all problems and problem to all solutions.

One should still measure oneshotting, it's an important self-measurement metric - but against an established, large codebase.

Unless I'm missing something, the prompt he gave must have been fairly detailed because both games are basically identical.

But for a more practical issue, the ultimate goal of LLMs is to replace software engineers, or at least enable everybody to become a software engineer, to use a more up-beat phrasing that's no less accurate. And so an LLM's ability to reliably construct something from a poorly defined, contradictory, or otherwise flawed prompt, while accurately inferring intent is probably the first finish line.

> I seriously dont' know all this big hullabaloo about one shot prompting.

It's a relatively objective way of testing LLMs, and I think it's pretty representative of how strong models are overall.

The outcome of this test mirrors how GLM 5.2 and Opus 4.8 work for me: they're both similarly capable of fully executing a given task, but Opus tends to have a bit more "taste" in how it handles unstated details or implicit requirements.

> what you'll get is a series of assumptions made by the model

Yes, but that's why we use these models in the first place. We don't want to explicitly write down all the details because that would mean writing code. So we write a higher-level, human-language spec, and let the LLM fill in the blanks. The question is how good they are at doing that.

I guess the experiment is interesting to determine if a model can produce something subjectively valued as "good" based on fairly vague and open-ended specifications. The benchmark is not to determine if the output fits the input, but whether the output is internally consistent: it's a game, but does it behave as one would expect that any game behaves? Does it end when you each the goal, do you die when hitting the spikes, are there weird edge cases in behavior when you move around?

I think however that they should have used the same harness and also repeated the experiment a few times to judge the variance in results.

It's a proxy for what you actually want to measure.

Note that after the model generated a bunch of (intermediary) code, they still have to have it tested and get bugs fixed (via the agent/harness). In this "one shot" you still have agent loops against human defined objectives.

And these toy examples give some insight as to how the model performs. If the test were "here's some code written by $corp, please take these tickets and work on them" it may be a "real" example but nobody would be able to make sense of actually how "hard" it is, or how "well" the model did the job, besides the workers already familiar with the context.

At least everyone knows what a 3D game is.

It's true that no one is trying to one shot anything serious right now, but it's still an important metric. Claude Code and Opus really took off when they improved the harnessing enough that it would self-correct many of its mistakes without needing user input. In fact I think long-term autonomy (in the range of several hours) and self-correcting is going to be where we see most improvements in coming years.

One shotting is useful to test but only with a huge prompt (eg, build something according to this spec).

I agree generating millions of tokens from a handful of input tokens doesn't convey anything meaningful to me.

> a single prompt wont' constitute the complexity of a software project.

The top agent is for steering, but all subagents are mostly oneshot prompts

If a model can take a series of increasingly complex instructions and satisfy the requirements without human intervention, we can pretty easily decide how well overall the model does. And, judging better models just means adding more requirements to a task. So, I think it's a useful method (Even if it's not a realistic use case).

Of course, with a software engineer at the helm - the models are going to be able to be guided to produce much better output. (Or worse, depending on the engineer!)

I think you're underestimating the elegance of "hey build X". It already captures a lot of what you're interested in.

Additionally, with "Hey build X" nobody is happy with the methodology and people rightfully complain about the set up.

Using your suggestion the methodology would require a lot of presumptions & arguments regarding why you choose it and think it relevant to people.

Either people would not "get" it quickly enough or would disagree/not be interested on the setup because its not how they use LLMs.

On one hand, that's sort of true for practical uses - and benchmarks notoriously undercount multi-turn settings.

On another, being able to reliably tackle minor tasks with no handholding is very valuable in itself. Sometimes implementation details are important, but often, the most important thing is to Get It Done.

> I'd rather see a coding agent that can follow steps in a plan file to a T while following guardrails and adhering to the proper coding conventions in the human reviewed spec.

Guardrails/conventions should be enforced in linters, formatters, static analysis tooling; not specs/prompts.

I also love the term zero-shot in the AI benchmark world. So logical. So intuitive.........

When the model produces reasonable results from one prompt, you could assume that it will also return reasonable results through the follow up prompts.

The argument is flawed, there is no logical reason to assume a single prompt won’t be sufficient to constitute the complexity of a software project. It may not be practical in many cases but there is too much variability in what is considered a complex software project and in the sufficiency of instruction in a single prompt to make that claim and say it’s “by definition.”

"We did multi-shot prompting to try and get these two games into comparable states using these two different models."

"Well obviously you provided better follow-up prompts to the one that came out better."

Also nothing about human-provided plan files and guardrails preclude the one-shot benchmark test. Heavens, I almost said "real coding," but in "real agentic program creation" you'd obviously be doing multi-turn interaction with the agent, but how can you provide a fair test when the model's output n determines your n+1 response?

Blame anthropic, they decided to make these type of one-shot examples the primary focus of the Fable 5 release, and relegating benchmark scores to the pdf.

That's precisely the difference between an engineer and a business guy.

The business guy would say "hey build me this and that" and would get _something_ to show of.

An engineer will have a long conversation with a llm about the exact requirements, tech stack, tradeoffs. He would understand what is built, how is it built, and refine on the fly until he gets something sensible.

It won't be as fast as "build this", but the result will be much better and more maintainable.

For the enginering workflow, you don't need Fable. Any model better or equivqlent to Sonnet 4.6 would do. Yes, sometimes it will hallucinate, sometimes it'll be wrong, but it's our job as engineers to correct it and have full ownership of the result.

Single prompt performance is interesting because best agentic results of yesterday turned out to be best single prompt results of today.

If we stopped developing LLMs the the only reasonable way to benchmark them would be to compare yheir performance with all the tricks we can build on top of them. Sine the are still developing rapidly any apples to apples comparison is worthwhile.

Of course this particular benchmark is not really single prompt but rather "agentic without steering".

Yet this is how virtually everybody is benchmarking and fine tuning.

Since Opus 4.6 I've seen later Anthropic models being more and more capable on one hand, but also less useful on multi turn open tasks.

It feels like with each model they are more and more prone to go "their own way" and jump into the implementation as soon as they can.

I can't but blame it on benchmarks and fine tuning around prompt-to-solution work.

I think that’s the point of the Superpowers SKILL

The thing with one-shot prompting is that it tests the ability for the model to make good choices on its own, rather than only instruction following.

Instruction following has been down for years, and while there are of course metrics that continue to improve as the frontier advances (for example, the ability to continue following the original instructions even as context grows), you can't really get that much better at performing a list of instructions as-written if the instructions are sufficiently precise enough that there's no wiggle room for interpretation (which seems to be what you are describing).

For example, one of the things that got me the most excited for Fable 5 was its ability to work for over eight hours straight on a single instruction and seemingly faithfully the entire time. That was something I observed personally after trying out the same workflow that runs for maybe two or three hours with Opus and then still needs followups. Fable needed no followups. That's a game changer for me compared to the prior state of the art.

That kind of stuff is going to end up being the most beneficial to people who are touching the edges of their knowledge or even exploring completely new areas. And that type of work is exactly the kind of work that makes agentic coding so powerful, even as much as it gets harder to judge the quality of the work when you lack the skills yourself. It's a good thing that the quality increases across the board, even for skilled practitioners.

For example, even people who know how to write inference engines or how matmul kernels work or how to optimize model architecture can't always predict just the sheer breadth of things agents can try to improve performance, and sometimes you get over some wall and reach a completely different optimum that you just wouldn't have reached in any reasonable amount of time by applying traditional knowledge even if you're an expert in the field.

That kind of stuff is amazing. And that's exactly the kind of stuff that one-shot prompting is testing for. It's kind of like testing for the model's "innovation", as much of an oxymoron that is.

Hi, I am the author, I completely agree! I set out to run a vibe test on this one, not a benchmark, the real benchmarks are listed. My test shows what the models can do when both tasked with a long-running, technically difficult, one-shot task.

I think your test you describe (collaborative, task delegation, task completion, TTD, steerability) is a great format for a future test that I will definitely try out.

On the other hand, I did just leave my pi agent running GPT 5.5 overnight on a clearly defined, long running task. It's been running about 10 hours now and it's mostly done. So this kind of use case is also valid.

Thinking about it, I would say that the majority of agentic work I do, by a long shot, is subagents which are launched from the main session, using a prompt of its choosing. Those could be considered short versions of these fully autonomous tasks.

sure that's why we look at a mix of formal benchmarks, one longer analysis of a side-by-side, and various other people who we trust to form an opinion, all covered in the article - not intended to be a formal benchmark, there are enough of those.

Totally agree, a single one-shot prompt can't prove anything.

I used it the other day for something of low importance that other models simply weren't figuring out and I didn't want to burn up Opus 4.8 on. (It had to do with overriding left-click on a macOS menu bar and then making Ctrl+click or right click bring up the menu like left-click normally does, and doing all this conditionally.)

Switched the model to GLM-5.2 halfway in the middle of a troubleshooting session (didn't even bother to reprompt, just changed it in the middle of its reasoning), gave it a few minutes, problem fixed. This is with the subscription based allocation on OpenCode Go, where a problem like this would completely burn up my Opus for the current 5 hours or even the current week.

Its also nice that you can see its entire reasoning trace. I can see it going off the rails - or see something I forgot to tell it - and stop and correct it. Or I'll learn WHY it made the choice it did and not have to question it after.

This mirrors my experience. I have been using it in Pi. It is smart and output is good but it is not efficient in getting there.

Also pricing, I wanted to give a try, but when pricing is only 30% cheaper than Opus, I wouldn't go for it with these issues.

Right, just give the text llm access to a vision specific agent and that problem can be solved. Or if you really want let it even call Opus with an image - seems like you’d still save money

Cool to hear, what kind of tasks have you been using GLM for? And what other models have you found useful through Ollama?

I have noticed that Opus and GPT 5.5 are very good at adjusting their thinking / reasoning intensity depending on the task at hand, something the open weights models are still not as good at.

In addition to that, some of the open weights models like GLM 5.2 or DeepSeek v4 Pro tend to be MUCH slower when generating tokens, which contributes to the perceived slowness. Although I wouldn't call models like GLM 5.2 slow by any means, e.g. it is currently one of the fastest models inside Notion today.

Probably the data center where the model is running more than anything. Another option is if Opus is using anything like a Mixture of Experts approach, in which case the amount of the model loaded in memory at one time could be smaller than GLM.

Could just be infra. I'm betting Anthropic is much better prepared.

No one is doing that for a model this size it would have to be so heavily quantized that it wouldn’t be useful - or you’d need to spend a half million dollars on hardware. People use hosted APIs. Open weight means cloud vendors can host it.

I think a bunch of real humans started to adopt the LLMs writing style.

This is excellent feedback thank you! These LLMisms in writing are a challenge I am living with currently and trying to improve on. The technical writing industry is taking a huge knock right now with companies demanding more work in less time with a big drop in quality, day to day I get less and less time to work on the quality in the prose of my work. We are working at the frontier of this right now, so we are the most heavily effected, but also get to experiment with the changes first which can be both stimulating and very frustrating.

Yes, and it's really grating. It's like half of all new writing is done in the same "voice" now.

This is the way but Anthropic doesn’t make it easy, so I use GPT 5.5 in that role since I can use my subscription in OpenCode or OMP.

I also use MiniMax-M3 in utility roles like explore/library tasks.

I’ve had a z.ai subscription for several months so I’m on the older pricing. I’m really not sure it would make sense to do this at current rates - I could bump my Codex plan instead.

I mostly use Opus for skill development. Once I have a solid skill implementation with a good eval, I move ongoing execution to a cheaper model running under Goose. With the eval you can see if the cheaper model works well enough.

Taking a view from outside the USA, European companies just had Fable taken away due to US export controls, and before that Anthropic announced it is holding their data for 30 days. There is immediate value to these firms to build their infrastructure around an AI that won’t be pulled away from them. And outside of Europe, other countries are more price sensitive and don’t have the same fear of building relationships with Chinese companies.

My impression is that individual subscriptions are the loss leading hook. The money is made on Enterprise token contracts.

Employees and students used to coding with thousands of dollars worth of tokens (on a 20/100 dollar plan) will push enterprise to spend.

Having a Chinese model that is competitive won't displace this enterprise spend. But an open model hosted in the US/EU might.

The existence of GLM 5.2 puts a ceiling on how much OpenAI/Anthropic can charge for API Access.

The value of these models is that you can run them on your own hardware.

A company can buy a NVIDIA B300 and serve it's developers in house with unlimited tokens.

This is an important point. I suspect API pricing will eventually disappear just like how paying for an MMS disappeared. It's an antiquated model. The bulk of the work is being done on "coding plans" is my wild guess.

It's annoying that the plans are so restrictive beyond usage limits. Understandable maybe, but annoying. In practice, only Anthropic (and maybe Google) are really restrictive though. They really scared me away with their policy of charging API rates after the fact if they consider your usage not TOS-aligned. This might be an ungrounded fear that I have, but I feel this is something they'd do so they scared me away.

> But the only people that use API billing for their coding work are large corporations

As well as people using 3rd party harnesses like OpenCode.

> At the same time, none of these companies will use a Chinese API for their employees

So who are Amazon Bedrock (who serve GLM) targetting?

Individuals are presumably going with one of the cheaper US providers such as DeepInfra ($0.18/M cached input for GLM vs $0.50 for Opus) or Fireworks AI.

> At the same time, none of these companies will use a Chinese API for their employees.

nice try but you intentionally ignored the entire Chinese market & Chinese big corporates. there are 130 Chinese companies in the fortune 500 list, with an average revenue of 80 billion USD each. do you think they are going to sign up for Claude, Codex or GLM? now consider South East Asia, Africa, Middle East, Middle Asia and South America, tell me why their large corporates won't be using GLM API billings?

your western centric view of the world is totally out of date, like it or not, 2026 is vastly different from 1996, the US no longer controls high tech whatsoever.

Also, I was testing out the GLM 5.2 using Openrouter because that's where I've got an account with some money and then when I wanted to perhaps subscribe for a better deal at z.ai, their infra was clearly overloaded to the point the 5.2 was timing out on 100% of chat requests, so perhaps I will try later when the infrastructure catches up with the model capability. Only then I can make sure their subscription is worth it.

I'm on glm pro subscription and I get so so so much more usage than Claude or Codex! I hammer on glm all day. It's a more expensive plan, but I would need a much much much bigger plan for codex or Claude to do what I do.

This framing local LLMs as free is stupid. Basically pay 100+ months worth of API costs up front isn't free in the slightest. And it will be slower than non-local, your hardware will be outdated in 12 months and probably won't be able to run SOTA at anywhere near non-local speed in max 20 months

GLM-5.2 performing like it would from a good provider - 8x B200s, so $450k. (No personal experience here)

GLM-5.2, severely quantised, 512GB Mac Studio, somewhere between $10k-$35k for a used M3. Or run it on a CPU with 768GB of RAM by getting an old PowerEdge with DDR4 for around $5,000.

Qwen-3.6-35b-q6, runs well on an RTX 5090 ($4000 + cost of a PC), runs medicore on an Intel Arc B70 ($1000 + cost of a PC plus lots of fiddling to get the setup to work right).

Gemma is a good candidate for the cheaper stuff, but I lack personal experience with using it locally

The price of a small house.

Practically nobody.

There is, for example, OpenCode Go subscription, which for $10 a month gives you a decently generous quota of GLM-5.2, among other models.

And z.ai themselves also have subscriptions.

> For example when one has $100 Claude Max and use it up through the month, it might not be more expensive than GLM, or at least not 5x.

https://z.ai/subscribe

I’m currently trying to figure out whether a downgrade from Max 5x to Pro in combination with one of those would save me money and if so, how much.

Edit: seems like Anthropic Pro + GLM Pro (Yearly) would let me almost halve my costs of Anthropic Max 5x. Only concerns are about GLM 5.2 not having vision support and also being kinda slower and also not being as good as Opus.

Yes this is true. This test was run on a $20 pro Claude subscription. I would definitely love to try use both models on the highest plans for a whole month and compare the two, great format for a future head-to-head comparison.

Is it fair when the one is heavily subsidized and the other one is not?

I think it's most fair to compare the plain token pricing that is used by everyone.

GLM has subscription plans too.

Yes I 100% agree. Time-taken can be improved (with harnesses, subagent workflows etc.) and varies based on task.

That is a great suggestion that I am definitely going to look into, thanks!

I may be biased and interested as I'm going to give you an affiliate link, but really honestly Synthetic LLM provider is a beast! They provide perfect GLM5.2, awesome token/s, TTFT and price.

Coupled with a local Headroom (https://github.com/headroomlabs-ai/headroom) you'll be able to use a LOT without hitting your 5h window :)

Definitely the best $ value for me considering the reasonable performance of GLM5.2.

They provide a rolling window quota, so you're never really out of quota contrary to other providers, you can adjust day to day.

Check it out if interested : https://synthetic.new/?referral=kwjqga9QYoUgpZV

---

Docs & all models : https://dev.synthetic.new/docs/api/models

Article states it's not multimodal. I guess that means for webdev it means you can't take a screenshot to indicate errors etc.

I hate to be that guy, but real privacy policy on training data/it being hosted somewhere where I'm not worried about secrets being stored/leaked.

Latency? Just saying there's other things to consider.

We've had the great small Qwen 3.6 early April that many could actually run on their laptop. Then similar from Google a few weeks later (Gemma4, better in prose, worse in code). Then the super cheap large Deepseek V4 a few weeks later. Then antirez DS4 build that made that actually runnable on MacBooks and Mac Studios. And now the "near-frontier / near-Opus" GLM 5.2.

For people who follow open LLMs, none of these were quiet and all were the most interesting open model release for a few days/weeks. In one or two months, it will be some other model again. Now I do appreciate the real rapid improvements in open models. But there's also a ton of hype and fast-fashion around all of this.

To me DS 4 is still the most interesting due to much lower costs. Also DS 4 training isn't done yet.

From my Opus vs DS 4 Pro personal benchmarks, 16 different real-life work tasks, DS 4 has performed as well as Opus 4.8 high overall but with few drawbacks:

- on the 16 tasks, one needed several prompts to be steered back into the topic

- its review capabilities seem much worse

- DS4 had the cleanly better solution in 3 cases out of 16, with Opus "only" doing cleanly better 2 times out of 16. But still, I want to emphasize, is the worst case scenarios that imho matter the most, not the best ones, and on that front Opus outperformed.

That being said I spent less than 2$ of API working 4 days, which is more or less what I would've spent with Anthropic APIs for less than one task.

You mean Fable?

Hi, author here, I cannot give an exact number for how many token the verification step took, but the verification GLM 5.2 ran was very stupid and definitely a waste of time. It read the pixel color data to try and verify the scene rendered properly. Which is really bad. Opus opened the game in a Playwright browser and took screenshots to verify the actual image. Which helped a lot.

Pro tip: You could use a multi-modal model to verify images as a subagent spawned by GLM 5.2, to get around this issue.

I could be wrong but I believe this is a non-vision model. Please weigh in to correct me bc I would love to be wrong

Yes I agree 100%. My next guide would do better to use identical harnesses.

I used GLM (4.7, 5 and 5.1) both through their coding plan and now through OpenCode zen. Both where painless to get.

How is it not easy to get the coding plan?

opencode zen go?

I see your point. Just the fact that one model does have vision and one does not might be an interesting point of comparison, however.

I hear you

Not my experience at all, I have written about comparing DS4 vs Opus 4.8 on 16 real life work tasks on multiple posts.

Also, every single lab does RL on benchmarks, which is why Opus 4.6 was the last truly great assistant, after it, all models tend to drift into implementation asap.