Hacker News

I think there are also other aspects:

- Some people simply ask a lot more questions than others (this ignores whether they like or dislike AI), i.e. some people simply prefer to find things out more by themselves, and thus also use other resources like Google or Stack Overflow as a last resort. So their questions to an AI will likely be more complicated, because they already found out the easy parts by themselves.

- If I have to make the effort to explain to the AI in a sufficiently exhaustive way what I need (which I often have to do), I expect the answers of the AI to be really good. If it isn't, having explained my problem to the AI was simply a waste of time.

I don't think that dichotomy is true at all, at least not with experienced software people.

Many folks I know are skeptical of the hype, or maybe full on anti/distrustful, due to reasons I think are valid. But many of those same people have tried llm tools, maybe chatgpt or copilot or cursor, and recognize the value even w/ huge misgivings. Some of have gone further with tools like claude code and seen the real potential there, quite a step beyond fancy auto-complete or just-in-time agents...but even there you can end up in rabbit-holes and drowning in horrible design.

In your incredibly reductive scale, I'm closer to 'love' than 'skeptical', but I'm often much of both sides. But I'd never write a prompt like 'write me some typescript' for any real work, or honestly anything close to that, unless its just for memes or demonstrations.

But no-one who programs for a living uses prompts like that, at least not for real work. That is just silly talk.

I’m confused as to why anyone would think this would be possible to determine.

Like can we determine the productivity of doctors, lawyers, journalists, or pastry chefs?

What job out there is so simple that we can meaningfully measure all the positive and negative effects of the worker as well as account for different conditions between workers.

I could probably get behind the idea that you could measure productivity for professional poker players (given a long enough evaluation period). Hard to think of much else.

Part of that, may be what we measure “product” to be.

My entire life, I have written “ship” software. It’s been pretty easy to say what my “product” is.

But I have also worked at a fairly small scale, in very small teams (often, only me). I was paid to manage a team, but it was a fairly small team, with highly measurable output. Personally, I have been writing software as free, open-source stuff, and it was easy to measure.

Some time ago, someone posted a story about how most software engineers have hardly ever actually shipped anything. I can’t even imagine that. I would find that incredibly depressing.

It would also make productivity pretty hard to measure. If I spent six months, working on something that never made it out of the cręche, would that mean all my work was for nothing?

Also, really experienced engineers write a lot less code (that does a lot more). They may spend two days, writing a highly efficient 20-line method, while a less-experienced engineer might write a passable 100-line method in a couple of hours. The experienced engineers’ work might be “one and done,” never needing revision, while the less-experienced engineer’s work is a slow bug farm, which means that the productivity is actually deferred, for the more experienced engineer.

When I ask ChatGPT (for example) for a code solution, I find that it’s usually quite “naive” (pretty prolix). I usually end up rewriting it. That doesn’t mean that’s a bad thing, though. It gives me a useful “starting point,” and can save me several hours of experimenting.

But nevertheless, productivity objectively exists. Some people/teams are more productive as others.

I suppose it would be simpler to compare productivity for people working on standard, "normalized" tasks, but often every other task a programmer is assigned is something different to the previous one, and different developers get different tasks.

It's difficult to measure productivity based on real-world work, but we can create an artificial experiment: give N programmers the same M "normal", everyday tasks and observe whether those using AI tools complete them more quickly.

This is somewhat similar to athletic competitions — artificial in nature, yet widely accepted as a way to compare runners’ performance.

We can determine productivity for the purpose of studies like this. Give a bunch of developers the exact same task and measure how quickly they can produce a defect-free solution. Unfortunately, this study didn’t do that – the developers chose their own tasks.

Actually, we can’t quantify most of the things we would like to optimize.

Team members always know who is productive and who isn’t, but generally don’t snitch to the management because it will be used against them or cause conflicts with colleagues. This team-level productivity doesn’t necessarily translate into something positive for a company.

Management is forced to rely on various metrics which are gamed or inaccurate.

what about the $ you make? isn't that an indicator? you've probably made more than me, so you are more successful while both of us might be doing the same thing.

I find the swings to be wild, when you win with it, you win really big. But when you lose with it, it's a real bite out of your week too. And I think 10x to 20x has to be figurative right, you can do 20x by volume maybe, but to borrow an expression from Steve Ballmer, that's like measuring an airplane by kilograms.

Someone already operating at the very limit of their abilities doing stuff that is for them high complexity, high cognitive load, detail intense, and tactically non-obvious? Even a machine that just handed you the perfect code can't 20x your real output, even if it gave you the source file at 20x your native sophistication you wouldn't be able to build and deploy it, let alone make changes to it.

But even if it's the last 5-20% after you're already operating at your very limit and trying to hit your limit every single day is massive, it makes a bunch of stuff on the bubble go from "not realistic" to "we did that".

I agree I feel more productive. AI tools do actually make it easier and makes my brain use less energy. You would think that would be more productive but maybe it just feels that way.

Stage magicians say that the magic is done in the audiences memory after the trick is done. It's the effect of the activity.

AI coding tools makes developers happier and able to spend more brain power on actually difficult things. But overall perhaps the amount of work isn't in orders of magnitudes it just feels like it.

Waze the navigation app routes you in non standard routes so that you are not stuck in traffic, so it feels fast that you are making progress. But the time taken may be longer and the distance travelled may be further!

Being in stuck traffic and not moving even for a little bit makes you feel that time has stopped, it's boring and frustrating. Now developers need never be stuck. Their roads will be clear, but they may take longer routes.

We get little boosts of dopamine using AI tools to do stuff. Perhaps we used these signals as indicators of productivity "Ahh that days work felt good, I did a lot"

I can’t believe such numbers. If this was true why don’t you quit your job and vibe code 10 ios apps

Where I have found Claude most helpful is on problems with very specific knowledge requirements.

Like: Why isn’t this working? Here Claude read this like 90 page PDF and tell me where I went wrong interfacing with this SDK.

Ohh I accidentally passed async_context_background_threading_safe instead of async_context_thread_safe_poll and it’s so now it’s panicking. Wow that would have taken me forever.

For the sake of argument 20x means you have basically suddenly got access to 19 people with the same skill set as you.

You can build a new product company with 20 people. Probably in the same domain as you are in right now.

You're getting 6 months worth of work done in a week?

> How I measure performance is how many features I can implement in a given period of time.

When a measure becomes a target, it ceases to be a good measure.

I cringe when I see these numbers. 20 times better means that you can accomplish in two months what you would do in 4 years, which is ridiculus when said out loud. We can make it even more ridiculous by pointing out you would do in 3 years the work of working lifetime (60 years)

I am wondering, what sort of tasks are you seeing these x20 boost?

> I'm pretty sure

So were the people taking the study. Which is why we do these, to understand where our understanding of ourselves is lacking.

Maybe you are special and do get extra gains. Or maybe you are as wrong about yourself as everyone else and are overestimating the gains you think you have.

I'm between 73 and 86 times more productive using claude code. You're not using it well.

Have any open source work you can show off?

Same, I’ve done stuff that should have taken me 2-3 weeks in days

I have exactly the same experience.

I think the productivity gains most people rave about are stuff like, I wanted to do X which isn't hard if you are experienced with library Y and library Y is pretty popular and the LLM did it perfectly first try!

I think that's where you get 10-20x. When you're working on niche stuff it's either not gonna work or work poorly.

For example right now I need to figure out why an ffmpeg filter doesn't do X thing smoothly, even though the C code is tiny for the filter and it's self contained.. Gemini refuses to add comments to the code. It just apologizes for not being able to add comments to 150 lines of code lol.

However for building an ffmpeg pipeline in python I was dumbfounded how fast I was prototyping stuff and building fairly complex filter chains which if I had to do by hand just by reading the docs it would've taken me a whole lot more time, effort and frustration but was a joy to figure out with Gemini.

So going back to the study, IMO it's flawed because by definition working on new features for open source projects wouldn't be the bread and butter of LLMs however most people aren't working on stuff like this, they're rewriting the same code that 10000 other people have written but with their own tiny little twist or whatever.

The sample size isn't 16 developers, it's 246 issues.

> search all over to figure out how to use it.

Leaflet doc is single page document with examples you can copy-paste. There is page navogation at the top. Also ctrl/cmd+f and keyword seems quicker than writing the prompt.

This is pretty similar to my own experience using LLMs as a tool.

When I'm working with platforms/languages/frameworks I am already deeply familiar with I don't think they save me much time at all. When I've tried to use them in this context they seem to save me a bunch of time in some situations, but also cost me a bunch of time in others resulting in basically a wash as far as time saved goes.

And for me a wash isn't worth the long-term cost of losing touch with the code by not being the one to have crafted it.

But when it comes to environments I'm not intimately familiar with they can provide a very easy on-ramp that is a much more pleasant experience than trying to figure things out through often iffy technical documentation or code samples.

I find I’m most likely to use an LLM to generate code in certain specific scenarios: (i) times I’m suffering from “writer’s block” or “having trouble getting started”; (ii) a language or framework I don’t normally use; (iii) feeling tired/burnt out/demotivated

When I’m in the “zone” I wouldn’t go near an LLM, but when I’ve fallen out of the “zone” they can be useful tools in getting me back into it, or just finishing that one extra thing before signing off for the day

I think the right answer to “does LLM use help or hinder developer productivity” is “it depends on how you use them”

It can get over some mental blocks, having some code to look at can start the idea process even it’s wrong (just like for writing). I don’t think it’s bad, like I don’t think writing throw away code for prototyping is a bad way to start a project that you aren’t sure how to tackle. Waterfall (lots of research and design up front) is still not going to work even if you forgo AI.

This has been my observation too. It's a tool for the lazy.

> They averaged producing 47% more code on the AI tasks, but took only 20% more time. The report here biases over these considerations, but I’m left wondering: was the extra code superfluous or did this produce better structure / managed debt better? If that extra 47% of code translates to lower debt and more consistent throughput over the long term, I might take it, given how crushed projects get from debt.

Wouldn't it be the opposite? I'd expect the code would be 47% longer because it's worse and heavier in tech debt (e.g. code repeated in multiple places instead of being factored out into a function).

I have an extremist take on this:

All source code is technical debt. If you increase the amount of code, you increase the amount of debt. It's impossible to reduce debt with more code. The only way to reduce debt is by reducing code.

(and note that I'm not measuring code in bytes here; switching to single-character variable names would not reduce debt. I'm measuring it in statements, expressions, instructions; reducing those without reducing functionality decreases debt)

Honestly my experience from using AI to code (primarily claude sonnet) is that that "extra 47%" is probably itself mostly tech debt. Places where the AI repeated itself instead of using a loop. Places where the AI wrote tests that don't actually test anything. Places where the AI failed to produce a simple abstraction and instead just kept doing the same thing by hand. Etc.

AI isn't very good at being concise, in my experience. To the point of producing worse code. Which is a strange change from humans who might just have a habit of being too concise, but not by the same degree.

As the fullstacker with a roughly 65/35 split BE/FE on the team who has to review this kinda stuff on the daily, there's nothing I dread more than a backender writing FE tickets and vice versa.

Just last week I had to review some monstrosity of a FE ticket written by one of our backenders, with the comment of "it's 90% there, should be good to takeover". I had to throw out pretty much everything and rewrite it from scratch. My solution was like 150 lines modified, whereas the monstrous output of the AI was non-functional, ugly, a performance nightmare and around 800 lines, with extremely unhelpful and generic commit messages to the tune of "Made things great!!1!1!!".

I can't even really blame them, the C-level craze and zeal for the AI shit is such that if you're not doing crap like this you get scrutinized and PIP'd.

At least frontenders usually have some humility and will tell you they have no clue if it's a good solution or not, while BEnders are always for some reason extremely dismissive of FE work (as can be seen in this very thread). It's truly baffling to me

This is good if front end is something you just need to get through. It's terrible if your work is moving to involve a lot of frontend - you'll never pick up the skills yourself

Interesting, I find the exact opposite. Although to a much lesser extent (maybe 50% boost).

I ended shoehorned into backend dev in Ruby/Py/Java and don't find it improves my day to day a lot.

Specifically in C, it can bang out complicated but mostly common data-structures without fault where I would surely do one-off errors. I guess since I do C for hobby I tend to solve more interesting and complicated problems like generating a whole array of dynamic C-dispatchers from a UI-library spec in JSON that allows parsing and rendering a UI specified in YAML. Gemini pro even spat out a YAML-dialect parser after a few attempts/fixes.

Maybe it's a function of familiarity and problems you end using the AI for.

This is exactly my experience as well. I've had agents write a bit of backend code, always small parts. I'm lucky enough to be experienced enough with code I didn't write to be able to quickly debug it when it fails (and it always fails from the first run). Like using AI to write a report, it's good for outlines, but the details are always seemingly random as far as quality.

For frontend though? The stuff I really don't specialize in (despite some of my first html beginning on FrontPage 1997 back in 1997), it's a lifesaver. Just gotta be careful with prompts since so many front end frameworks are basically backend code at this point.

It works with low-level C/C++ just fine as long as you rigorously include all relevant definitions in the context window, provide non-obvious context (like the lifecycle of some various objects) and keep your prompts focused.

Things like "apply this known algorithm to that project-specific data structure" work really well and save plenty of time. Things that require a gut feeling for how things are organized in memory don't work unless you are willing to babysit the model.

I've been hacking on some somewhat systemsy rust code, and I've used LLMs from a while back (early co-pilot about a year ago) on a bunch of C++ systems code.

In both of these cases, I found that just the smart auto-complete is a massive time-saver. In fact, it's more valuable to me than the interactive or agentic features.

Here's a snippet of some code that's in one of my recent buffers:

    // The instruction should be skipped if all of its named
    // outputs have been coalesced away.
    if ! self.should_keep_instr(instr) {
      return;
    }

    // Non-dropped should have a choice.
    let instr_choice =
      choices.maybe_instr_choice(instr_ref)
        .expect("No choice for instruction");
    self.pick_map.set_instr_choice(
      instr_ref,
      instr_choice.clone(),
    );

    // Incref all named def inputs to the PIR choice.
    instr_choice.visit_input_defs(|input_def| {
      self.def_incref(input_def);
    });

    // Decref all named def inputs to the SIR instr.
    instr.visit_inputs(
      |input_def| self.def_decref(input_def, sir_graph)
    );

The other nice benefit is that I don't have to trust the LLM. I can evaluate each snippet right there and typically the machine does a good job of picking out syntactic style and semantics from the rest of the codebase and file and applying it to the completion.

The snippet, if it's not obvious, is from a bit of compiler backend code I'm working on. I would never have even _attempted_ to write a compiler backend in my spare time without this assistance.

For experienced devs, autocomplete is good enough for massive efficiency gains in dev speed.

I still haven't warmed to the agentic interfaces because I inherently don't trust the LLMs to produce correct code reliably, so I always end up reviewing it, and reviewing greenfield code is often more work than just writing it (esp now that autocomplete is so much more useful at making that writing faster).

This feels like a parallel to the Gell-Mann amnesia effect.

Recently, my company has been investigating AI tools for coding. I know this sounds very late to the game, but we're a DoD consultancy and one not traditional associated with software development. So, for most of the people in the company, they are very impressed with the AI's output.

I, on the other hand, am a fairly recent addition to the company. I was specifically hired to be a "wildcard" in their usual operations. Which is too say, maybe 10 of us in a company of 3000 know what we're doing regarding software (but that's being generous because I don't really have visibility into half of the company). So, that means 99.7% of the company doesn't have the experience necessary to tell what good software development looks like.

The stuff the people using the AI are putting out is... better than what the MilOps analysts pressed into writing Python-scripts-with-delusions-of-grandeur were doing before, but by no means what I'd call quality software. I have pretty deep experience in both back end and front end. It's a step above "code written by smart people completely inexperienced in writing software that has to be maintained over a lifetime", but many steps below, "software that can successfully be maintained over a lifetime".

Same. It’s amazing for frontend.

It's a very well controlled study about... what the study claims to do. Yes, they didn't study a different thing, for _many_ reasons. Yes, we shouldn't haphazardly extrapolate to other parts of Engineering. But it looks like it's a good study nonetheless.

There are some very good findings though, like how the devs thought they were sped up but they were actually slowed down.

React and tailwind already made lot of tradeoffs to make it more ergonomic for developers. One would expect that LLMs could unlock lean and faster stack instead.

As a backend dev who owns a few internal crappy frontends, LLMs have been the best thing ever. Code quality isn't the top priority, I just need to plumb some data to an internal page at BigCorp.

We were stuck near local maxima since before LLM's came on the scene. I figure the same concentration of innovators are gonna innovate, now LLM assisted, and the same concentration of best-practice folk are gonna best-practice--now LLM assisted. Local maxima might get sticker, but greener pastures will be found more quickly than ever.

I expect it'll balance.

imagine having interstate highways built in one night you wake up and you have all these highways and roads and everyone is confused what they are and how to use them. using llm is the opposite of boiling frogs because you're not the leader writing, you're just suggesting... i just realized i might not know what im talking about.

On the flip side, why would I use AI today if it presents no immediate benefit. Why not wait 5 years and see if it becomes actually helpful.

If they do improve at 5-10% a month then that'd definitely be true (tbh I'm not sure they are even improving at that rate now - 10% for a year would be 3x improvement with compounding).

I guess the tricky bit is, nobody knows what the future looks like. "The internet is a fad" in 1999 hasn't aged well, but a lot of people touted 1960s AI, XML and 3d telivisions as things that'd be the tools in only a few years.

We're all just guessing till then.

Trough of Disillusionment (followed by the Slope of Enlightenment and Plateau of Productivity): https://en.wikipedia.org/wiki/Gartner_hype_cycle

This isn't true, and I know it by what I'm working on and sorry, I'm not at liberty to give more details. But I see how untrue this is, every working hour of every day.

They're great at helping me figure out how to make something work with a poorly-documented, buggy framework, which is indeed a large fraction of my job, whether I like it or not.

You're using it wrong -- it's intended to be a conversational experience. There are so many techniques you can utilize to improve the output while retaining the mental model of codebase.

Respectfully, this is user error.