k1m4 hours ago
I agree with Roy Fielding on this:
> It is a bad design trade-off to send a bunch of header fields on every request just to tell the server all of the possible variations of preference held by the user, particularly when there is a very small chance that any of those dimensions are applicable to the target resource. It has been a bad design trade-off ever since the very brief period in 1993-94 when folks didn't know which image format would be usable on all UAs and there was no CSS or javascript to allow for client-side adaptation.
> ...The caching impact of proactive negotiation is far worse than the one extra round trip per site for reactive negotiation, and even that round-trip isn't necessary in formats that support client-side adaptation.
On the caching impact, Simon Willison wrote:
> ...you can’t deploy an application that uses content negotiation via the Accept header behind the Cloudflare CDN — for example serving JSON or HTML for the same URL depending on the incoming Accept header. If you do, Cloudflare may serve cached JSON to an HTML client or vice-versa.
Note: I posted this in another comment with links to those two quotes which I couldn't copy easily now - will add later.
chuckadams3 hours ago
If you do content negotiation, then it’s imperative to send “Vary: accept” in your response. CF and all other CDNs will automatically do the right thing when they see that header.
Content negotiation still has its uses, but most of the time you’re better off using different endpoints.
9dev4 hours ago
Conceptually, I’m not sure I agree. There’s elegance in clients saying "I want this resource, and I’d like to get your markdown version of it. If you don’t have one, I’ll also take HTML." And if you couple that with optional "file extensions" at the end of the url to force a specific format (say, /foo for automatic negotiation, and /foo.html, /foo.json, or /foo.md for the respective media type,) you have a very easy to use API that adapts to the client; not the other way around.
I take the point that it makes caching harder, but I don’t think that should overrule ergonomics concerns.
k1man hour ago
In general I think I just don't like the idea of one URL being able to return different content. Forces me to think about what each system I give that URL to may be sending in content negotiation headers. Would rather the HTML is returned and alternatives listed in HTML head.
But for HTML and Markdown in particular, there's been so much useful work done in the semantic HTML space and microformats, that I don't know why anyone interested in this wouldn't just improve their HTML markup and leave it to the agent to do the rest. Convert to markdown or extract the useful HTML before handing it to model.
meindnoch3 hours ago
>On the caching impact, Simon Willison wrote:
Wrong: https://developers.cloudflare.com/cache/concepts/vary/
usef-3 hours ago
Yes. This feature is brand new: https://developers.cloudflare.com/cache/changelog/
Simon wrote that in 2023: https://simonwillison.net/2023/Nov/20/cloudflare-does-not-co...
k1m2 hours ago
Thanks. I posted that comment before they had added support - https://news.ycombinator.com/item?id=48353325 - didn't know situation had changed.
But it's something I think developers should think about if they rely on caching. If it took Cloudflare this long to support this, there may be other systems which still don't.
Link to Roy Fielding comment:
https://lists.w3.org/Archives/Public/ietf-http-wg/2013JanMar...
singpolyma33 hours ago
If CloudFlare isn't honouring vary: accept that's a pretty serious bug
chuckadamsan hour ago
They just started fully honoring it last month. I guess my comment above was overly optimistic.
joshum975 hours ago
I feel like I am going insane. Who in their right mind would feed an LLM raw HTML in the first place??
HTML is a markup language. User agents present it in a way that makes sense for the user—visually, or through assistive technology. Nothing about adding LLM “users” should change this—their user agent, the harness, should present HTML in a way they can natively understand, by translating it to Markdown.
We are not going to rewrite the entire web because harness developers are too lazy or stupid to pull an HTML to markdown package from npm. If some sites want to do so anyway, good for them, in many cases I’d love to skip the CSS/JS and read the markdown directly (or better, nicely formatted). But don’t blame website authors for your harness wasting your tokens.
Semiapies22 minutes ago
Who in their right mind would feed any random LLM crawler their website's content?
k1man hour ago
Agree. I think many people forget that not long ago, HTML markup on many sites was a lot richer than it is today. Making it trivial to produce a good trimmed down markdown version.
The reason it may be more difficult today is because we've lost a lot of that. Some of it because of modern JS frameworks, but some also because publishers simply don't want to make it easy for the useful stuff to be scraped and extracted easily.
I'm not convinced that's changing because of AI agents (it's getting worse in many ways with anti-agent rules). Maybe improving for documentation pages intended for agents. But if it is changing, I think it'd be far easier to improve the HTML and let the agent take care of the rest.
alsetmusic4 hours ago
> in many cases I’d love to skip the CSS/JS and read the markdown directly
I've been enjoying https://defuddle.md since learning about it. Works great. Not affiliated.
usef-4 hours ago
Exa also has an API for it that has worked well for me, returning markdown for a URL, which means you don't need to render js or anything yourself. It doesn't need an account for up to 1k requests/month, which is more than I've ever needed.
Most harnesses have a "zero config" plugin that works, eg: https://pi.dev/packages/pi-exa
(no affiliation to either)
lekevicius6 hours ago
I'll do that once any of the top 4 AI chatbots says they'll start making requests with this header. Before that it's just a neat idea with no adoption.
I also think it's exceedingly unlikely that any of the top 4 chatbots would choose to load websites this way. Too many risks for a 0.01% adoption even years later.
JimDabellan hour ago
Claude Code’s been doing this for at least a year already and I assume Claude Cowork does it too as a result. That’s already millions of users, so it’s not “just a neat idea with no adoption”.
russ_j38 minutes ago
Also it's not that hard at crawl time to transform HTML into markdown before passing to the LLM.
kennywinker2 hours ago
I mean, it's exceptionally easy for them to add it. Even with low adoption. Somewhere in the bowels of chatgpt there is code that queries for html, and then parses that to markdown or something similar. All this would be is adding the logic to request markdown if available, then bypass the html->markdown step if you get markdown back.
collimarco5 hours ago
What about clean, semantic HTML?
It was already optimized for bots and search engines (which are bots) and it has been used for decades. Why we need to serve in markdown now?
There are also many parts of the HTML, like navs, that are useful for bots and AI and may be removed in the markdown version.
meindnoch3 hours ago
>What about clean, semantic HTML?
Which React package is this?
k1m2 hours ago
I agree. I think a lot of people here are assuming that the full HTML retrieved has to go into the LLM eating up tokens. But why wouldn't the agent try to clean up first and remove bloat and convert to markdown itself, before feeding into LLM. Semantic HTML would make that easier.
slowin5 hours ago
Presumably markdown uses far fewer tokens.
simonw4 hours ago
That used to matter to me back in the days when the best models still only accepted ~32,000 tokens, but these days even the models that run on my laptop are happy with ~100,000 and the hosted models I use take ~200,000 or more.
selcuka3 hours ago
They accept more tokens these days, but they are still more accurate with a shorter context [1].
slowin3 hours ago
If it's one of many tool calls, I'd assume that less is more.
honr4 hours ago
Is that even true? I most often use HTML. HTML is about 5%-20% more tokens than a similar Markdown. As a rule of thumb, the number of tags/structural tokens doubles, when going from markdown to html, while the rest don't change much. On the other hand, I can view HTML without any extra/unusual tools. And composing HTML when I need a bit of structure is far easier than composing markdown.
slowin3 hours ago
> On the other hand, I can view HTML without any extra/unusual tools. And composing HTML when I need a bit of structure is far easier than composing markdown.
This is kind of the opposite of reality no? Markdown is just plain text and meant to be human readable. You don't need XML tags to read and write it, opposed to html where you do and you need a browser to properly view it.
honr22 minutes ago
No, it's just that I wasn't very clear.
HTML I can view in any browser / webview / etc. Good markdown viewers are fewer / more special, or end up translating md to html for display.
And by composing, I didn't mean writing by hand. We are talking about prompting, right? Or that is what I thought we are talking about. Composing HTML "components" into a final prompt HTML is easier than composing markdown snippets into the final prompt. That is because with HTML there are several ergonomic libraries to parse HTML to AST and to format AST back to HTML. The libraries (for parsing to AST and back to strings) are more limited with markdown.
xienze3 hours ago
That's highly dependent on what sites you're visiting. Take a look around at a lot of modern sites, there's a sea of divs and spans. Markdown conversion helps LLMs a lot.
honr9 minutes ago
Aah, I thought we are talking about prompting or providing information to AI agents in either html or md form, and comparing the two.
Assuming that is what we are talking about, HTML is easier to work with than Markdown, unless you are writing it by hand. That is, composing semantic HTML is more ergonomic than composing a Markdown formatted document from components / snippets, programmatically. The libraries are just better and more versatile in most programming languages. Typically you go from HTML or Markdown to AST, then you compose them to end up with the final tree, then you format the tree to HTML or Markdown. LLMs treat them basically identically (context in HTML or context in Markdown), so I have ended up forming complex prompts / context parts using HTML.
kaangiray266 hours ago
hoping for this to get mainstream so that I can just view the pages without any ads, js and bloat
Elfener6 hours ago
> But imagine for a second they did do that. You get the API, all the info is there.
> Why do you need AI then?
(from this HN comment, worth a read in full: https://news.ycombinator.com/item?id=43679585)
phoghed5 hours ago
> Let's say you want to plan a trip to Thailand with your family. You could use the fancy AI to do it for you, or you could build a stupid frontend with minimal natural language understanding.
It's giving https://news.ycombinator.com/item?id=8863#9224
qznc6 hours ago
Yeah, that is why this will not get popular.
jiehong6 hours ago
Markdown ads?
qingcharles5 hours ago
Time magazine already serves their pages like this to agents with ads for the agents in them, IIRC.
maurelius26 hours ago
Even without AI this would be a nice feature.
LunicLynx5 hours ago
Hello … AI company that wants easier access to data
arjie4 hours ago
Everything is determined by cloudflare. If they supported Gemini on the same hostname I’d serve it everywhere. If they supported text/plain and text/html everywhere I’d serve that. But they don’t cache per content encoding so I just do what works for them.
They’re far too useful for me.
singpolyma33 hours ago
arjie27 minutes ago
Quite helpful, I’ll see if I can serve text/plain.
Kuyawa5 hours ago
Interesting...
A cherry.jpg image/jpeg file is shown as an image by the browser
A logo in svg like https://news.ycombinator.com/y18.svg too
So browsers should show readme.md as text/markdown and clicking on a link inside that file should also redirect to another markdown file, instant wiki
Browsers should implement that asap and we will follow
meindnoch3 hours ago
It depends on one question: does Markdown support <script> tags?
krapp3 hours ago
Technically, Markdown supports all HTML tags, including script tags.
Practically, Markdown is more of a vibe than a spec and everyone just uses whatever subset makes sense to them.
a2ff6eeb06 hours ago
Hm, interesting avenue for prompt injection.
spockz6 hours ago
How is that different from today where agents just do searches and ingest web pages?
hnlmorg5 hours ago
I don’t disagree with you in principle but there is a difference.
The MD content isn’t intended for human consumption whereas HTML is. So you either have that injection readable to your users, which could be jarring to non-technical readers, or you play the cat-and-mouse game of hoping those pesky crawlers don’t ignore text that’s not human-visible on a rendered page.
spockz5 hours ago
Right. And today we have something similar with search engines going on where sites want to offer the full text to the engine for seo. But maybe they want to do the opposite to agent and poison the well. So human agents still get the “real” data and agents the “something plausible but not quite right” data.
Or everything just disappears behind something like cloudflare or a paywall.
brap3 hours ago
Best case scenario, this ends up being abused in order to feed LLMs crap responses (or worse).
monneyboi6 hours ago
Yeah, this is the way forward. All the POW stuff is such a waste of energy.
The open internet deserves a real solution instead of gatekeepers in the form of brightdata, firecrawl, cloudflare and the likes.
nozzlegear3 hours ago
No, I don't think I will. I publish things for people, not bots.
4lb02 hours ago
Is this an alternative or complements llms.txt?
kimseungyong2 hours ago
[dead]
OutOfHerean hour ago
It's a good thought, but the problem with it is that it can never be trusted by the bot, except for a finite whitelisted set of trusted sites that are known to serve representative markdown that faithfully represents the HTML page, its content, and its links.
[deleted]3 hours agocollapsed
ErroneousBosh5 hours ago
So can I use this to serve actively harmful content to the Cocaine Piracy Parrots?
Maybe tarpit them into reading gigabytes of Markov Chain nonsense, really slowly?
xgulfie4 hours ago
Sounds like a great way to spread disinfo to LLMs, so I'm down
postal66663 minutes ago
[flagged]
shahariaa6 hours ago
[dead]
