Hacker News

The impression seems to be warranted: Grok 3 has directly jumpted to the top of all leaderboard categories in Chatbot Arena: https://lmarena.ai/?leaderboard

In math it shares the top spot with o1 and is just a few points behind (well within errors). In creative writing it is basically ex-aequo with the latest ChatGPT 4o and in coding it's actually significantly ahead of everyone else and represents a new SOTA.

Naive question from a bystander , but since DeepSeek is open source and is on par with o1-pro (is it?), shouldn't we expect that anybody with the computer power is capable to compete with o1-pro?

Related ongoing thread:

Andrej Karpathy: "I was given early access to Grok 3 earlier today" - https://news.ycombinator.com/item?id=43092066 - Feb 2025 (48 comments)

I don't see the Think button, and for me the answer is much below deepseek-r1 even thought I have Premium+ subscription. I'm just getting instant stupid answer instead of thinking.

How can anyone repeatedly use a question like this without new models getting trained on it via online discussion?

I've spent the last hour testing it and I'm blown away. And this is coming from a very hardcore user of OpenAI/Claude products on a daily basis.

It passed every goofy test I have for writing articles which involves trying to surface arcane obscure details. (it certainly means however they are scraping the Web they are doing a good job here)

It made the database code I wrote over the last week with o3/o1/GPT4o/Claude3.5 look like a joke.

It fills me with rage over who owns this thing.

Even if people tank Tesla's car business and run Twitter into the ground, I think our new Galactic Edgelord is going to win his first trillion on xAI and Teslabots anyway.

btw: it tried to charge me $40/mo for this thing: https://imgur.com/a/QXslgBo

It's not good news when this competition comes at cost of a gigantic over inflated bubble, in which all the big players keep on sucking billions from investors without even having a business model.

This hype will burst sooner than later and will trigger yet another global recession. This is untenable.

I think it's already clear that these are going to be commoditized and the free / open source versions will be good enough to capture enough of the value that the remaining players will not be Facebook-level monopolies on the space

Who cares about benchmarks?

These things still cost me time because of hallucinations.

Probably bad news for the vendors, though. I genuinely struggle to see how most of these LLM companies are going to monetize and profit off their efforts with LLMs already in commodity territory. Government contracts can only flow for so long?

benchmarks dont show the quality or 'correctness' of the response though.

Yep, seeing Grok come out I'm just so glad there are free alternatives that aren't behind paywalls.

[flagged]

>It's been said before but it is great news for consumers that there's so much competition in the LLM space.

Is it? Because it seems like a bunch of megacorps pirating every single copyrighted work available in digital format, spending an enormous amount of electricity (that is probably not 100% clean) to churn through them, and the end result we have a bunch of parrots that may or may not produce accurate results so that spammers can more effectively fill the Internet with crap.

There was this joke about rich Russians that I heard maybe 25 years ago.

Two rich Russian guys meet and one brags about his new necktie. "Look at this, I paid $500 for it." The other rich Russian guy replies: "Well, that is quite nice, but you have to take better care of your money. I have seen that same necktie just yesterday in another shop for $1000."

Imagine what they'll achieve if they'll apply deepseek methods here with this insane compute

I have a close friend working in core research teams there. Based on our chats, the secret seems to be (1) massive compute power (2) ridiculous pay to attract top talents from established teams (3) extremelly hard work without big corp bureaucracy.

To me, it seemed like they spent there money to get there. They talked about the massive datacenter they built, but will it pay off is the question.

And they mentioned at the end of the presentation that they're already planning their next datacenter, which will require 5x the power. Not sure if that means equivalent to ~1,000,000 of the current GPU's, or more because next-gen Nvidia chips are more efficient.

I don't think anyone who's paid attention to the LLM scene will give them any “credit for catching up fast” as it has been pretty obvious for the past two years that all it takes to reach the state of the art is a big enough GPU cluster.

Deepseek made the news because how they were able to do it with significantly less hardware than their American counterparts, but given that Musk has spent the last two years telling everyone how he was building the biggest AI cluster ever, it's no surprise that they manage to reproduce the kind of performances other players are showing.

Seems like the opinion of someone who doesn't know that OpenAI cloned Anthropic's innovations of artifacts and computer use with their "canvas" and "operator".

This sounds like "this feature is so 2024".

Firstly, the 97Bn was for the non-profit, not for the company. The company is being valued in funding rounds closer to 300Bn. I think it may be true that OpenAI has no moat, but if it has no moat then all of these AI companies are overvalued (including xAI) and Elon should just stop bothering to throw his money at it. I would say Elon probably actually doesn't have much of an advantage here. In both SpaceX and Tesla he was able to do something no competitor could do - raise cash. Car companies simply couldn't invest in tech research to build self-driving to compete with Tesla. SpaceX consumed enormous amounts of cash before anyone saw value. That is a unique skill that Elon had over the 2010s.

That is not an advantage in a race against Microsoft, Google, Meta etc. he's competing against all the biggest companies in the world in this race. He's not going to be able to outspend them if the economics look at all sensible.

Will the vast sums of money being spent on Grok ever actually have a positive ROI?

And DeepSeek is just 3% behind. It seems in that benchmark all LLMs perform well and top is formed within some statistical error.

And Anthropic not even in the top 10 ...

No idea why was this downvoted, but you are correct.

Seems like the team at xAI caught up very quickly to OpenAI to be at the top of the leaderboard in one of the benchmarks and also caught up with features with Grok 3.

Giving credit where credit is due, even though this is a race to zero.

[flagged]

You know what they say: Fascists are good at keeping the training runs on time.

This is grok 3, so not a debut

Gemini has been topping benchmarks and leaderboards for weeks if not months at this point. Nobody cares.

This is how they've done the past releases as well, soon after they release the latest and greatest they open source the last model.

I'm waiting for this as well, though I did try to run several manually now that it's live and the results have been impressive so far.

Thanks for the update.

Any guess on its availability at that point? Is it likely to be limited to certain tiers like o1?

We're 1-2 years into our AI game studio [1] if anyone has more questions on it.

Seeing awesome feedback from players on our demos (and seeing an insane amount of stickiness from players playing even small demos built around generative AI mechanics). Raising now. Hiring soon to move faster. Feel free to reach out - [email protected]

[1] https://www.chromagolem.com/

Especially on code, think of all the free data you get from the generation-evaluation loop

What is dogfooding?

Because 'dogfooding' has worked so well for other products...

If you don't get feedback from the people actually playing your game (or using your product), you will never get the improvement you need to help them.

You can have the most talented passionate people there are developing a product, but if it's not working for the people you want to sell it to, it's the wrong product.

Most tech products are terrible because those paying for them are not those that have to use them every day, or because they solve a corporate problem (compliance) and not a usability problem which is the actual need from the people on the shop floor.

Many big games/products are already built mostly on metrics, and that has proven to be a terrible way to work out what people 'want'. It's a great way to justify money decisions though, so it keeps happening (and games/products from big companies keep getting worse).

It's way worse than that. First, We interact with LLMs through private conversation and we are used to have private conversation with human we trust. Some of that trust will be transfered to LLMs. Second, LLMs have a vastly bigger "mental" power to build a long term mental model of us, while we interact with them. Which mean they can chose with extreme precision their words to trigger an emotion, a certain reaction.

Combine the two and the potential for manipulation, suggestion, preference altering is through the roof.

I do believe this is the next natural evolution. People don't like to pay for things and ads are a proven business model. I bet the big labs are looking into this

Meta probably already does this on the top 10k people that spends most or is a high ranked influencer on instagram etc.

I'm more worried about LLMs with specific political biases built into them. Imagine one that sounds like Conservapedia or the most insane left-wing parts of Tumblr

In the opening blog post they mentioned it to be 1 m tokens.

The question came up and he said they would, once Grok-3 is fully released.

I'd expect them to open source it just like they did with Grok.

We're still waiting for OpenAI to do the same. Even at least GPT-3.

Note: this is "before April" so not a complete assessment

https://manifold.markets/SaviorofPlant/will-xai-open-source-...

I'm sure not.

examples? i have been using it all morning and just canceled my claud subscription

There are just more regulations to comply with before a release. OpenAI's new Deep Research tool wasn't originally available in the EU either, but it was released less then a week after it came out in the US. Since the EU is a gigantic market with a lot of buying power and this release makes a strong case for people to switch over from competitors, I doubt it'll take long.

Do you mean data protection or political correctness/control of discourse protection?

Not in Canada, either :(

The EU and UK are good for data protection?

>Not available in Europe currently

I hate how its the same story for every new AI technology. If someone can tell me who to vote for or where to protest to change this awful EU law, that would be great.

No, it was underwhelming, failing basic coding tasks, OCR/Image recognitions that none of the other existing models screw up.

[flagged]

So he took credit for improvements others worked on and they also weren’t as good as purported?

(Assuming that is a reference to the Mussolini quote.)

OK that's actually a pretty good one. If you didn't steal that from an X comment, I give you props.

Why can so many people not see it?!

grok.com

What makes pay-per-use pricing inherently more or less fair than unlimited usage from a subscription?

But you trust Google, OpenAI, and whatnot with it?

Tells me they have spent their entire engineering time on engineering and zero on marketing fluff, which is good.

I'm guessing Musk wanted it that way.

Yeah not sure what profit these guys think they'll be able to squeeze out of these models with open source and free clearly being 95% as good

How did you customize the output?

I use it everyday.

For coding, I use cursor composer to gather context about the existing codebase (context.md). Then I paste that into DeepSeek R1 to iterate on requirements and draft a high level design document, maybe some implementation details (design.md).

Paste that back into composer, and iterate; then write tests. When I'm almost done, I ask composer to generate me a document on the changes it made and I double check that with R1 again for a final pass (changes.md).

Then I'm basically done.

This is architect-editor mode: https://aider.chat/2024/09/26/architect.html.

I've found Cursor + DeepSeek R1 extremely useful, to the point that I've structured a lot of documents in the codebase to be easily greppable and executable by composer. Benefit of that is that other developers (and their composers) can read the docs themselves.

Engineers can self-onboard onto the codebase, and non-technical people can unstuck themselves with SQL statements with composer now.

My wife has found ChatGPT extremely useful when dealing with her mother - who has bipolar and is obsessed with other people's health. I've got a terminal cancer diagnosis and handling my mother-in-law has been extremely difficult - nearly to the point of no longer having any communication with her. ChatGPT has a single conversation with all the back story and has put some useful points across when discussing how difficult her behaviour has been (she watched a operation that failed for me for entertainment, for instance).

I have found similar when giving backstory and needing help to start structuring difficult conversations where I want to say the right thing but also need to be sensitive.

Here's some things I have in my chatgpt history:

- Discussing the various stages of candymaking and their relation to the fundamental properties of sugar syrups, and which candies are crystalline vs amorphous. It turns out junior mints are fudge. Fondant is really just fudge. Everything is fudge, my god.

- Summarizing various SEC filings and related paperwork to understand the implications of an activist investor intervening in a company

- Discussing the relative film merits of the movie Labyrinth and other similar 80s kitsch movies. ChatGPT mentioned the phenomenon of "twin films" which was an interesting digression.

- Learning about various languages Tolkien invented and their ties to actual linguistics of natural languages and other conlangs

- Some dimensional analysis of volumes, specifically relating to things like "how many beans are in the jar" estimation and what the min and max of a particular weight of coins might be valued, in terms of both a par value based on a standard coin mix and outliers of, for example, old dimes that are pure silver.

- Discussion of quines in prolog and other languages, which resulted in a very interesting ChatGPT bug where it started recursing and broke when trying to write a prolog quine.

- Back of the envelope economic calculations around the magnitude of the housing deficit and the relative GDP cost for providing enough housing quickly enough to make an impact. Spoiler: it's probably unreasonably expensive to build enough houses to bring down housing prices by any significant degree, and even if we wanted to, there's not enough skilled workers.

- A number of podcasts transcribed. (I hate audio and meandering, so transcribed and summarized is perfect) I could use whisper and a python script to do this, but I'd rather let ChatGPT do the legwork, and it actually used a more modern model and method of processing than I would have naively used.

I find Github Copilot to be a really great autocomplete. I frequently write the comment at the top of a function and hit tab and it writes the whole function. This is dependent on typescript and having a relatively standard codebase but I think those things are useful on their own. You really have to limit it in terms of scope and specifics, but it lets me think high level instead of worrying about syntax.

The only plausible explanation for the amount of resources poured into these language models is the hope that they somehow become the origin of AGI, which I think is pretty fanciful.

I can feel the cold wind of the next AI winter coming on. It's inevitable. Computers are good at emulating intelligent behavior, people get excited that it's around the corner, and the hype boils over. This isn't the last time this will happen.

Many people replied with anecdotes, but recently Anthropic published analysis of claude.ai production traffic. As far as I can tell this is the single best currently existing source of "how people actually use LLMs". Everything else you can't be sure whether it is representative.

https://www.anthropic.com/news/the-anthropic-economic-index

I use them as a springboard for things I am really unfamiliar with. I'm self-learning electronics at the moment, and so I can ask it things like "what's a common and widely available cooperator." You will not find that answer on a search engine, I don't care how good your Google fu is.

It's a weak jack of all trades: it knows a fair amount about the sum of human knowledge (which is objectively super-human), but can't go deep on any one thing, and still seriously lags behind humans in terms of reasoning. It's an assistant that all book smarts and no street smarts. Or maybe: it's a search engine for insanely specific things.

Rote work, as well. Things like porting an enum from one programming language to another: past the source language into a comment and start it off with one or two lines in the target language. Dozens of tabs are surely faster than manual typing, copy paste, or figuring out vim movements/macros.

I've found that Claude has been pretty decent at writing boilerplate code.

For example asking it something like "I have an elixir app that is started with `mix ...` can you give me a Dockerfile to run it in a container?"

It can also do things like "Given this code snippet, can you make it more Pythonic" or even generate simple apps from scratch.

For example, a prompt like "Can you write me a snake game in HTML and JavaScript? The snake should eat hot dog emojis to grow longer." will actually generate something that works. You can see the generated code for that prompt at https://claude.site/artifacts/34540f88-965e-45ca-8083-040e30...

Following up with "Can you make it so that people can swipe on mobile to control the snake?" generates https://claude.site/artifacts/651e957a-9957-488c-ae6b-e81348... which is pretty good IMO for 30 seconds of effort.

It also has a surprisingly competent analysis mode where you can upload a CSV and have it generate charts and analyze the data.

It's not perfect, it'll sometimes get confused or generate some dubious code, but you can quickly get to a 90% good solution with 1% of the effort, which is pretty impressive IMO.

> I ended up disabling github copilot because it was just "auto-complete on steroids" at best

this is good enough sell for me, and it's like sub 1-in-50 that it's "auto-complete on mushrooms" (again my experience, YMMV).

An awful lot of the time, my day to day work involves writing one piece of code and then copy-pasting it changing a few variable names. Even if I factor out the code into a method, I've still got to call that method with the different names. CoPilot takes care of that drudgery and saves me countless minutes per day. It therefore pays for itself.

I also use ChatGPT every time I need some BASH script written to automate a boring process. I could spend 20-30 minutes searching for all the commands and arguments I would need, another 10 minutes typing in the script, another 10-20 minutes debugging my inevitable mistakes. Or I make sure to describe my requirements exactly (5-10 minutes), spend 5 minutes reviewing the output, iterate if necessary (usually because I wasn't clear enough in the instructions).

3-5x speed up for free. Who's not going to take that win?

My biology professor provides basically zero feedback on his student's understanding of the material. There are very few practices questions to prepare for exams, which are worth 40% of your grade. I had an LLM write some python that extracts the relevant textbook chapters, which then I can feed into an LLM to generate practice questions. Then I can ask the LLM for feedback and whether or not I'm articulating the answers correctly.

I reckon the ideal use case for chat LLMs at the moment is as a bridge for questions that are hard to search but easy to verify.

For example, you have a plant you can't identify. Hard to Google search with words. "Plant with small red berries and...". You could reverse image search your photo of it, probably won't help either. Show an LLM the photo (some accept images now). LLM tells you what it thinks. Now you Google search "Ribes rubrum" to verify it. Much easier.

You've got a complicated medical problem that's been going on for months. A google search of all the factors involved would be excessively long and throw up all sorts of random stuff. You describe the whole scenario to an LLM and it gives you four ideas. You can now search those specific conditions and see how well they actually match.

I've found there are actually a lot of questions that fit in that sort of NP complexity category.

I use LLMs significantly on a daily basis, mostly for coding C#, HTML, CSS, SQL. I use them for researching for wiki articles. I use it for summarizing long web pages and science papers. I use them for translation. I used GPT last night to repair my furnace (I've never opened a furnace before).

It (mostly) exceeds and excels at every task I use it for. I'm rarely disappointed. YMMV.

Absolutely life-changing for me.

I think most people are still "holding them wrong", and it'll take an entire generation of people to really figure out what these things are and are not good for.

I'll give two recent use-cases that may provide a hint of their ultimate utility:

1) I've been modernising 2010-era ASP.NET code written by former VB programmers that looooved to sprinkle try { ... } catch( Exception e ) { throw e; } throughout. I mean thousands upon thousands of instances of these pointless magical incantations that do nothing except screw up stack traces. They probably thought it was the equivalent of "ON ERROR RESUME NEXT", but... no, not really. Anyway, I asked ChatGPT in "Reasoning" mode to write a CLI tool utilising the Roslyn C# compiler SDK to help clean this up. It took about three prompts and less than an hour, and it spat out 300 lines of code that required less than 10 to be modified by me. It deleted something like 10K lines of garbage code from a code base for me. Because I used a proper compiler toolkit, there was no risk of hallucinations, so the change Just Worked.

2) I was recently troubleshooting some thread pool issues. I suspect that some long-running requests were overlapping in time, but Azure's KQL doesn't directly provide a timeline graphical view. I dumped out the data into JSON, gave ChatGPT a snippet, and told it to make me a visualiser using HTML and JS. I then simply pasted in the full JSON dump (~1 MB) and ta-da instant timeline overlap visualiser! It even supported scrolling and zooming. Neat.

Last night I was about to start working on a lot of text I need to submit my startup to a government funding program. Questions like "describe what your startup does", "describe your market" and things like that. Tens of fields which I estimated would take me a week and a half to do it right, if I wasn't going to pause all my other activities.

Then I had a better idea: I spent 20 minutes baby wearing, walking and dictating everything about my startup to ChatGPT. Later I took all that text and labeled it as a brain dump, plus my product support portal and some screenshots of my marketing material. Gave it all to ChatGPT again and asked it to answer each of the questions in the form. That's it. I have a pretty good version 1 which I can revise today and be done with it.

Many, many hours saved. I have tens of examples like that.

The product documentation I provided it with was also created with the help of GPT, and that saved me even more time.

Autocomplete on steroids is what I use it for. I've recently started using Cursor and the productivity improvements have been huge. I won't let it write very large blocks of code but I do a lot of web stuff so being able to update the classes in one spot and have it recognise all the other places that might be helpful and let me just tab through. Code to test things it's pretty good at as well which also saves a lot of typing.

For me, getting summaries of meetings is my favorite use case. Saves me from taking notes and I can extract next steps.

It also helps me getting started with new content, kind of building the scaffolding of, say, a blog or social post. It still needs adaption and fine-tuning, but getting rid of a white page is a great help for me.

And I use LLMs to play through ideas and headlines. I would normally do this with other humans, but since working full remote, its a nice sparing partner, although the AI not being able to really give criticism is a bit annoying.

The tools also make it easier to write in English as a non-native, making sure my text does not include any false friends or grammar errors.

I use them as an alternative to search engines for topics where I have some specific question where traditional search engines fail to find the needle in the haystack.

As a concrete example, I was recently playing with simulating the wave equation, and I wanted to try to use a higher-order approximation as I had never done that before. I'm quite rusty as I haven't done any numerical work since university some decades ago.

I still recalled how to deal with the Neumann boundary conditions when using the traditional lower-order approximation, but I was uncertain how to do it while keeping the higher-order approximation.

Searching for "higher-order neumann boundary conditions wave equation" or similar got me pages upon pages of irrelevant hits, most of them dealing with the traditional approximation scheme.

So I turned to ChatGPT which right away provided a decent answer[1], and along with a follow-up question gave me what I needed to implement it successfully.

[1]: https://chatgpt.com/share/67b4ab43-6128-8013-8e5a-3d13a74bf6...

This video's pretty great: https://www.youtube.com/watch?v=uRuLgar5XZw

One thing I can't figure out how to get LLMs to do is truely finish work. For example if I have 100 items that need xyz done to them, it will do it for the first 10 or so and say ~"and so on". I have a lot of trouble getting LLMs to do tasks that might take 10 mins - 1h. They always seem to simply want to give an example. Batch processing is the answer, I guess, or perhaps more 'agentic' models/tools - but I wonder if there are other ways.

finally, after two years of hype I have some usage for llm.

we import descriptions of products from a seller. the problem is they are mental ( probably written by chatgpt :)) and are way too long. we need only small blurb.

I put our style guide and given text to chatgpt and I get somehow reasonable description back. then editors still need to check it, but it's way less work.

I find them useful for searching for some function or API name with natural language. 'What's the function call [in blah] that generates a quaternion from a couple of vectors?' type stuff. Not exactly inspiring but I've found it highly useful. If you try to search for something like that online (and somebody hasn't asked the exact question on e.g. stack overflow) you'll just end up getting all the documentation for quaternions, vectors, and blah - when the function itself might even be in a tertiary math library.

These probably aren't tasks you need done but;

LLMs are pretty good at translation between human languages which makes sense since they are language models after all. They are better at this any any other technology.

The state of the art image ones can also probably do OCR and handwriting recognition better than any other software though may be expensive to run in large volume. But if you need to take picture of a notebook page with your camera phone an LLM can quickly OCR it.

Big emuse cases for me are:

1. Exploring a new domain and getting some terms I can google for.

2. Making small scripts to do things like query github's GraphQL API.

3. Autocomplete of code using copilot.

I use it daily for all sorts of things, but one of the most interesting uses for me so far has been self-reflection.

For example, in the beginning of this year, I completed this exercise where I wrote a lot about childhood, past experiences, strengths and weaknesses, goals and ambitions for the future, etc (https://selfauthoring.com) and then I uploaded all that to ChatGPT, asked it to be my therapist/coach, and then asked it to produce reports about myself, action plans, strategies, etc. Super interesting and useful.

By now ChatGPT has quite a bit of context from past conversations. Just the other day I used this prompt from someone else and got back super useful insights – "Can you share some extremely deep and profound insights about my psyche and mind that I would not otherwise be able to identify or see as well as some that I may not want to hear"

I don't use it daily, and I find copilot counterproductive (for me). I do try to experiment with chatgpt when I remember to.

I find it good for complex SQL, reviewing emails, and Godot assistance (I'm a beginner game Dev).

There are also times when I have programming questions and I might try to use chatgpt, with mixed results.

Our company has tried to integrate it into one of our products, and I find it troubling how on occassion it is confidently giving bad results, but my concern seems to be in the minority.

EDIT: there was also a large refactor I did recently which involved lots of repeatable, but not super regexable, changes - chatgpt forgot where it was as I went through it, but other than working around that it was very useful.

I use mine as if it's an infinitely patient, relatively competent junior/medior level developer that I constantly give small chunks of programming to do (typically a function at a time), and occasionally consult on architecture/design/other things.

I don't use integrated coding tools, so my workflow isn't super fast, but that's not what I'm really aiming for - more that I want to save my brain's energy from low level drudgy boilerplate or integration code, so I can focus it on the more important decisions and keep business-side context in my head.

It's been a huge help for me this way across multiple projects in multiple domains.

I've coded a full custom deployment system (config, create, update, cert manege and much more) in full bash using nothing but ChatGPT. I didn't write a single line of bash.

I did write 50 or more lines of instructions on what needs to be done and in what order.

ChatGPT gave me 5/6 (I asked for this) bash scripts totalling 300+ lines that seamlessly work together.

After reviewing, I asked it to change a few places.

If any human tried the same (except those rare bash Gods), it'd take many hours. I think it took me less than 30 minutes.

The Vergecast recently did a section where they asked listeners what they use LLMs for (specifically not for coding) https://youtu.be/WwNjBNtZ3Co 30 minutes starting at 45:25, it had a number of interesting examples. Might not convince you of LLM's excellence, or might not be much different from what other people commented, but it's a good listen nonetheless.

I used ChatGPT all the time for

1. Small coding tasks ("I want to do XYZ in Rust"), it has replaced stack overflow. Very convenient when writing code in a language I'm not super familiar with. 2. Help with English (traduction, proofreading...) 3. Learning something, like tech, I like interacting with it by asking questions, it's more engaging than just reading content.

I'd say nothing is game changing, but it's a nice productivity boost.

The sister comments contain quite a few specific examples. But the many back-and-forth arguments here on HN about whether LLMs are useful for coding suggest that understanding how they might or might not be used may be the biggest challenge at this point.

I myself use them a lot, though I constantly feel that I would be able to get more out of them if only I were smarter.

> All my attempts so far have been pretty underwhelming:

Same, it's good for repetitive things, things that have been answered 1000 times on stack overflow, translations, but that's about it. If you work on anything remotely new/hard it's mostly disappointing, you have to babysit it every step of the way and rewrite most of what it's shitting out in the end anyways.

I think it just made it obvious that 90% of tech jobs basically amount to writing the same CRUD thing over and over again & mobile/web apps with very common designs and features.

I admit to having been an LLM sceptic from day one, but I have been using ChatGPT and Claude a fair bit to try and figure out what the hype is all about. I haven't really succeeded.

Most recently I tried to use them both to solve a programming problem that isn't well documented in the usual channels (Reddit, StackOverflow, etc) and found it to be quite a disappointing and frustrating experience. It just constantly, enthusiastically fed me total bullshit, with functions that don't exist or don't do what the LLM seems to "think" they do. I'm sure I'm just "holding it wrong" but my impression at this stage is that it is only capable of solving problems that are trivially solvable using a traditional search engine, with the added friction that if the problem isn't trivially solvable, it won't actually tell you that but will waste your time with non-obvious wrong answers.

I did have a slightly more positive experience when asking it about various chess engine optimisation algorithms. I wasn't trying to use the code it generated, just to better understand what the popular algorithms are and how they work. So I think they might work best when there is an abundance of helpful information out there and you just don't want to read through it all. Even then, I obviously don't know what ChatGPT was leaving out in the summary it provided.

Cooking & Meal Planning:

- I have these three ingredients; recommend Italian main courses.

- What other ingredients pair well with this?

- How can I "level up" this dish if I want to impress?

- Can I substitute X for Y?

- Generate a family-friendly meal with lots of veggies using leftover roast chicken.

I just used chat gpt to summarise a HN post about it taking unexpectedly long to install a washing machine because of unexpected turns of events, and this being analogous to software development. It was a time-saver.

I think it excels when you know enough to precisely describe what you want but you don’t know enough about the details of the language or framework you’re using to implement what you want.

Here are some of my experiences:

* Figuring out where to start when learning new things (see also <https://news.ycombinator.com/item?id=43087685>)

  One way I treat LLMs is as a "semantic search engine". I find that LLMs get
  too many things wrong when I'm being specific, but they're pretty good at
  pointing me in a general direction.

  For example, I started learning about OS development and wanted to use Rust. I
  used ChatGPT to generate a basic Rust UEFI project with some simple
  bootloading code. It was broken, but it now gave me a foothold and I was able
  to use other resources (e.g. OSDev wiki) to learn how to fix the broken bits.

  It feels like a lot of software documentation isn't actually written for real
  readers; instead being a somewhat arbitrary listing of a program's features.
  When programs have this style of documentation, the worst case for figuring
  out how to do a simple thing is reading the entire manual. (There are better
  ways to write documentation, see e.g. <https://diataxis.fr/>)

  One example is [gnuplot](http://www.gnuplot.info/). I wanted to learn how to
  plot from the command line. I could have pieced together how to do it by
  zipping around the
  [gnuplot manual](http://www.gnuplot.info/docs_5.4/Gnuplot_5_4.pdf) and building
  something up piecewise, but it was faster to instruct Claude directly. Once
  Claude showed me how to do a particular thing (e.g. draw a scatter plot with
  dots intstead of crosses) I then used the manual to find other similar
  options.

  Similar to the previous point. If I ask Claude to write a simple program using
  a complex publicly-available API, it will probably write a broken program, but
  it won't be *completely* bogus because it will be in the right "genre". It
  will probably use some real modules, datatypes and functions in a realistic
  way. These are often good leads for which code/documentation I should read.

  I used this approach to write some programs that use the
  [GHC API](https://hackage.haskell.org/package/ghc). There are hundreds of
  modules, and when I asked Claude how to do something with the GHC API it wrote
  relevant (if incorrect) code, which helped me teach myself.

  My partner is Chinese and sometimes we talk about Chinese poetry. I'm not very
  fluent in Chinese so it's hard for me the grasp the beauty in these poems.
  Unfortunately literal English translations aren't very good. We've had some
  success with asking LLMs to translate Chinese poems in the style of various
  famous English poets. The translation is generally semantically correct, while
  having a more pleasing use of the English language than a direct translation.

I like having fun with them, like by asking grok whether some Elon Musk tweet is true. Usually it replies with a lengthy answer and I then force it to answer with Yes or No. Even more fun when drilling it more to load a few more details into it's brain, and then ask the first question again, with a Yes/No only, it sometimes does change its answer. I do wonder, has Grok already joined the resistance against Musk?

Start from here :

Stop using Google search and use an AI. No more irrelevant results, no more ads. No more slop to wade through.

BTW I find Claude is great at making graphs and diagrams. If you pay ($20) you can hook it up to a local code base.

Honestly, the main thing I've found ChatGPT to be useful for in my daily life is helping to translate what I write from my native language to the language spoken by most of the people where I live. But even then it only really works if you have at least a basic understanding of the language and can ask it to rewrite sections when you recognize poor word choices or awkward phrasing.

LLM are good at one thing, and totally by chance it is the thing they have been designed to be: be a word probability generator. If you can constrain your usage around that, they are great to use. But the people who think they can reason or know some kind of truth are delusional

I use it for lots of stuff where I'm not an expert, or that are low stakes. I don't use it for the "core" of my job, but there are many things that are not "core" and still eat up a lot of time, in fact, most of my workday would be in this category. Some typical examples from my daily life as a university professor:

- Writing Python scripts to make charts out of Excel sheets, and then refine them. I could do it myself, but I would need to learn a library like Seaborn or similar which honestly is not especially intellectually stimulating, and then spend nontrivial amounts of time iterating on the actual code. With LLMs it's a breeze.

- Working with cumbersome LaTeX formatting, e.g. transposing a table, removing a column from a table, etc.

- Getting the tone just right in a professional email written in English to someone I don't know much (I'm not a native speaker so this is not trivial).

- Finding resources on topics that are tangential to what I do. For example, yesterday I needed to come up with some statistics on English words for a presentation I'm preparing, and I needed a free corpus where I could search for an n-gram and get frequencies of next words. I don't usually work with that kind of resource, it was just a one-off need. I asked for corpora of that kind and got a useful answer instantly. The manual process would probably have implied going through several options only to find that I needed a license or that they didn't provide the specific statistics I needed.

- Brainstorming on titles for scientific papers, presentations, names of concepts that you introduce on a paper, variable names, etc.

- Shortening a sentence in a paper that makes me go over the page limit, or polishing the English in a paragraph.

- Summarizing a text if I'm kind of interested in knowing the gist but have no time to read it whole.

- Answering quick questions on basic things that I forget, e.g. the parameters to make a Linux folder into a tar.gz. Man is too verbose and it takes time to sort the wheat from the chaff, Google is full of SEOd garbage these days and sometimes you need to skim a lot to find the actual answer, LLMs are much faster.

- Writing bureaucratic boilerplate, the typical texts with no real value but that you have to write (e.g. gender perspective statement on a grant request).

- Coming up with exam questions. This is a rather repetitive activity and they're fantastic at it. At my place we also have two official languages and we need to have exam assignments on both languages, guess who does the translation now (respecting LaTeX formatting, which previous machine translation tools typically wouldn't do).

- As an example of a one-off thing, the other day I had to edit a Word document which was password-protected. I asked ChatGPT how to unlock it and it not only answered, but actually did it for me (after 3 tries, but still, much faster than the time it would have taken for me to find out how to do it and then actually do it).

These are just some examples where they contribute (greatly) to my productivity at work. In daily life, I also ask them lots of questions.

I remember the NVIDIA Linux kernel binary blob driver discussions from the early-mid 2000s. Who knew we had an open source driver all along...

It's started with abusing the term "AI", I don't see a reason why they would not abuse "open source". I guess it's what happens to language when a concept becomes mainstream: people use it wrong, but if enough people do it, it becomes the new meaning?

But I agree, it's a real shame.

this is correct. "open source" means everything required to recreate from scratch and improve. not "here's a massive binary, an interpreter script and permission."

I'm guilty of this. I didn't publicly shame a coworker who installed fb's model and said "it's open source" just for the sake of peace.

Most of these claimed “open” models are not open source. Some of them are open weights. But even some of the ones that share weights are not really open - they force a restricted license on you. To be open source I think they need to share training data and training code under an OSI approved license.

AI2 has a model called OLMo that is actually open source. They share the training data, training source code, and many other things:

https://allenai.org/blog/olmo2

They also released an app recently, to do local inference on your phone with a small truly open source model:

https://allenai.org/blog/olmoe-app

While I do agree with your point - I wonder what information companies could release that'd be immediately useful to you.

It's not like they understand what the weights mean either and if they released the code and dataset used to create it, you probably couldn't recreate it, owning the fact that you don't own tens of thousands of GPUs.

If a software's source is released without all the documentation, commit history, bug tracker data etc., it's still considered open source, yet you couldn't recreate it without that information.

Thank you for pointing this out, I was not thinking clearly about this

Better a downloadable blob than a non-downloadable one