Hi Hacker News, I’m Yahia. I built Context.dev (https://www.context.dev/) to make it really easy to integrate web data into your products and agents.
Here’s a demo video: https://www.tella.tv/video/build-faster-with-context-dev-api...
Since it’s an API, here are the docs: https://docs.context.dev/quickstart.
You can send us a URL and get back clean Markdown, rendered HTML, screenshots, extracted images, etc.. You can also send us a domain and get company or brand context: name, description, logos, colors, fonts, social links, screenshots, style information, and related metadata. For more custom use cases, you can send a URL plus a JSON Schema and ask us to extract structured data from the site into that shape. For example, you might ask for pricing plans, product categories, office locations, support links, integration partners, or anything else that is visible on the public site.
The goal is to give developers the output they actually want. Raw HTML is rarely the useful thing; the useful thing is usually Markdown for a model, JSON for an application, a logo for a UI, or a structured company profile for an agent.
Before, I worked at Amazon and Sunrun, and co-founded StockAlarm.io & essense.io, both of which were acquired. Also, I built knifegeek.io, which scraped pocket knives from across the internet and listed them easily. The project is outdated now (coming back soon) but back then it hit the frontpage of hacker news and people seemed to like it: https://news.ycombinator.com/item?id=34604281.
Just before Context.dev, I built Brand.dev. The idea was that your software product should automatically know about your customer if they sign up with a corporate email. The API pulled brand data such as logos, backdrops, name, description, industry, and more from the public web and surfaced it to your product to integrate as part of their onboarding experience. That’s worth doing because conversion rates on onboarding improve dramatically when you go from “enter all this info” to “confirm all this info” (and there was never any privacy concern all the information is public).
That was a nifty niche, but the more customers used it, it became obvious that “brand data” was only one slice of a larger need. People started asking for things like screenshots, structured extraction, and LLM ready data. So I expanded to Context.dev, and applied to YC (got rejected after an interview), then kept going and re-applied at which point I got in as a solo founder.
People use Context.dev in more ways than I can list, but here are some: keeping context up to date on customer websites for chatbots - building beautiful brand assets/ads for customers - enrichment flows using agent harnesses like eve.dev - crawling customer websites into chatbot knowledge bases - turning GitHub repos into branded docs sites - academic journal and PDF crawling. There are a ton more examples at https://www.context.dev/customers.
We know that many crawlers are not behaving like good citizens on the web, and the entire space has a bad reputation as a result. At the same time, customers are not usually trying to buy “scraping”. They are trying to make a support bot work, personalize onboarding, enrich CRM records, generate docs, monitor leads, or let an agent research a company. There are lots of legit use cases. We want to satisfy those while being respectful of everyone involved.
We maintain a caching layer and avoid hammering websites. Customers can configure the cache, but if we find we’re sending too many requests to a url in a certain amount of time, we step in and tone it down. Websites can opt out of our service, and we respect these requests and add them to our block list.
We focus on customers who want to build cool things for their users. Enriching onboarding is a popular use case. So is integrating context about their own websites (things like support bots), and building agents that can automatically reason about complex tasks involving the internet.
We only allow customers to use brand data to identify a specific customer on their software, you cannot use it in your own materials or to imply endorsement.
I'd love to hear your feedback about the product in the comments, thanks!
setgree8 hours ago
I like the clarity, tone, and readability of your webpage. Also your FAQ is refreshing
> When Should I talk to sales? > Talk to sales if you need high-volume pricing beyond 2M credits/month, custom rate limits, SSO / SAML, SCIM provisioning, an uptime SLA, annual invoicing, an MSA / DPA, or a dedicated support channel. Reach us at [email protected] or through the contact page.
Would that this were the norm everywhere, rather than (say) a sales rep from Datadog scraping my phone number from who knows where to ask about my company's needs after I sign up for a free account on a whim :)
TheYahiaBakourop8 hours ago
happy you noticed that, i put quite a bit of love into the ui, i've found engineers care alot about polish & feel of the webapps they use, even for an api product
i'm an engineer by trade, and always hated things like forcing a sales call, or having hidden credit multipliers, i tried to build this with the same ethos i like for my own dependencies (shoutout axiom.co)
twosdai8 hours ago
Are you using residential proxies? How do you handle websites that don't want to be scraped.
EG if I start passing in Linkedin pages what is your expectation of the result that people would see per profile.
EDIT:
Congrats on the launch seriously hard work, just wanting to understand your scraping stance more. I've worked with a lot of tools on this, didn't mean for my initial comment to be adversarial.
TheYahiaBakourop8 hours ago
hi, no worries at all, it came off perfectly fine
yes, we use residential proxies + all requests go are js-rendered, we maintain a caching layer which is 95%+ opted into by customers
it's all included in the credit price, great value compared to alternatives, the business model does rely on scale and our margin gets better the more requests we serve (esp infra cost for k8 + browser fleet)
to answer your e.g., yes public linkedin pages will work fine, anything behind a login we don't really support out of the box until we can figure out a safe way to do so, since that's where red lines are drawn
we step in whenever we see our service is hitting a website more than it should, this usually means reaching out to a customer for clarity on why they are not opting into the cache, we have alot of safeguards around fraud/spam and will let someone know if their request pattern looks like they're causing harm
twosdai5 hours ago
What major edge cases have you found so far for the browser scraping method, its really complex like dealing with Auth, or pages that use pop ups, content blockers . etc...
TheYahiaBakourop3 hours ago
if it needs auth, we don't do any type of logins at the moment, still figuring out how to do it without attracting the wrong type of customer (someone doing shady stuff)
popups / cookie banners are quite easy to get through with enough heuristics for 99%+ of cases
content-blockers are the same thing as the auth scenario, hard to figure out where to draw the line between building something people want and doing something shady
blast7 hours ago
Is "residential proxy" a euphemism for botnets or is there a difference? Genuinely curious.
TheYahiaBakourop3 hours ago
- worst is "extension" based traffic routing where your provider is essentially tricking end users into routing traffic through - opt-in p2p networks can be described as botnets sometimes, i try to avoid these but not as bad - good enough dISP proxies can be classified as residential quality, work quite well, although people have argued w me on their quality - managed business/enterprise residential gateways are ideal, because businesses opt in and allow the traffic - few other options
dataviz10006 hours ago
Have a look at Intercept. [0] I don't have a need for it, likely it is dated and will require some more tuning, and I want to get away from scraping. Creating typed Typescript proxy API for any website might be something you find useful.
> Reverse-engineers any website by doing a breadth search across every transport (JSON, WebSocket, WebRTC, GraphQL, SSE, HLS, PubSub), listing them all, and generating a typed JSON API that bypasses almost all bot protections — including Turnstile. I didn't include the ability, but it bypassed the most advanced ChatGPT + Turnstile. Built with self-improving Claude Code agents that rewrite their own instructions until fresh agents consistently succeed.
> Once connected to a page, it intercepts every byte of network traffic — then actively drives the page to surface endpoints that only fire on interaction. It types into forms, clicks buttons, scrolls, triggers modals, paginates, submits searches, and walks through multi-step flows, watching what each action produces on the wire. Every request gets captured with its method, headers, payload shape, and response, then classified by transport (JSON, WebSocket, WebRTC, GraphQL, SSE, HLS, PubSub). The result is a complete map of the site's real API surface — including the hidden endpoints that only exist behind a click — turned into typed proxy routes you can curl.
TheYahiaBakourop6 hours ago
Fascinating, this is definitely a path i was thinking about for a bit but decided against doing it primarily since it changes the business fundamentally
dataviz10006 hours ago
You are smart!
Another YC startup went this path, scraping targeting endpoints, and they didn't get much traction and they have since pivoted.
> We maintain a caching layer and avoid hammering websites.
This made me think of it. It is another way to avoid hammering websites by using extremely targeted requests for the most part bypassing HTML, DOM, and JavaScript.
Barbing2 hours ago
Why not much traction, was the benefit more to websites than businesses/scrapers?
TheYahiaBakourop2 hours ago
it counts as "reverse engineering" a website which is much worse for the website owner imo, then again im biased
TheYahiaBakourop5 hours ago
100% yes, i found out about that company recently too, it's a tempting idea for sure
modo_9 hours ago
I was using Context back when it was still Brand.dev. I found it to be a great product- one of those rare APIs that immediately made a problem I had disappear. Had it in production within an hour of signing up
Agents need clean/current context from the web, and this is the best way I’ve found to give it to them. The internet is clearly moving in this direction: companies are starting to realize their sites need to be legible to agents. Some are already adapting but many haven’t yet. Context feels like an important part of that transition
Yahia is a great builder. His pace of expansion has been impressive, excited to see where he takes Context.
TheYahiaBakourop9 hours ago
oh wow, happy to see you here, we didn't post this anywhere so its nice to see a customer find us so quickly, thank you for the kind words.
brand data is a shockingly hard problem to get right
erichi8 hours ago
[dead]
m_w_9 hours ago
Unclear what difference exists against Firecrawl - their team has been shipping great features extremely quickly lately, and their core offerings have become really good.
I am interested in KnifeGeek though - looking for a good OTF (ultratech?)
TheYahiaBakourop9 hours ago
There's alot of overlap at the moment, but we're diverging quite quickly.
If you want differences as of right now
- 1 credit = 1 scrape, no hidden credit multiplier - we have world class brand data - we're focused primarily on the infra use-case, rather than gtm & everything else - anecdotally, customers have seen their error rates drop quite dramatically
In general it's a huge space, firecrawl is a wonderful company, it's fun to compete with them, planning to add more things soon which should make the differences clear
m_w_9 hours ago
Interesting - I'll be sure to benchmark it at some point. We've found the best results come from blending providers depending on the task anyways.
Thanks for the quick response - and always happy to see more competition in the space. Best of luck with future features!
TheYahiaBakourop9 hours ago
please do, and if you see anything off let me know, we've yet to lose a single "bakeoff", i normally ask customers to just have cc/codex run it so its somewhat unbiased
on the 2nd point, most industries are not zero-sum, and many of customers use multiple data providers in any case, so agree with you there
thank you!
TheYahiaBakourop9 hours ago
oh also on knifegeek, it'll be live again this weekend when i find a moment to fix the bug causing it to crash, i collect knives/watches so it was a super fun project to work on
tomerlir4 hours ago
Really cool idea! I had something similar a while back. Maybe figure out a way to use MCP so the client can connect directly out of the box
TheYahiaBakourop4 hours ago
ty, it's here! https://docs.context.dev/install-mcp
korrectional7 hours ago
Hey Yahia, I recently saw your startup on the YC registry and found it to be one of the most promising of the batch, so good luck with this! Do you mind if I ask how you managed to get your first users on Brand.dev?
TheYahiaBakourop7 hours ago
hey, happy you like it, to be honest we had no real secret, it takes a ton of time and effort to earn developer trust
i got my first customer after 1 month, and then just worked like hell to make him happy even though it was a very large company paying me $99/mo at the time, they're still a customer too
since then, i just locked in on every customer until they were thrilled, and once we hit 50, word of mouth started to kick in, now we're at 300+ and get a referral every other day which is awesome
i think our community appreciates effort, and when someone goes the extra mile to make sure your problems are solved, you tend to remember that. it doesn't scale super well, but you'd be surprised how far it'll take you
Barbing2 hours ago
Excellent way to run a business.
sheept9 hours ago
Does this respect llms.txt and robots.txt, or have you found it more effective if agents see what humans see?
TheYahiaBakourop9 hours ago
we try to read robots.txt, it's definitely more efficient if agents just see what humans see, we run a custom browser rendering stack
in terms of llms.txt, we're not primarily an AI product (although some features do use LLMs), and speaking to friends who run products it seems to be not very helpful, even though we have one as well, i didn't see it move the needle much
even my own coding agents don't look at llms.txt when looking at our own website, so unsure of whether that standard will survive the test of time
bwm8 hours ago
Awesome! Been great watching this product improve so quickly, can't wait for what's next :)
TheYahiaBakourop8 hours ago
thank you, i try to ship updates often: https://docs.context.dev/changelog
debo_6 hours ago
Remember when the Semantic Web attempted to make the entire web machine-readable?
TheYahiaBakourop6 hours ago
you just found my end goal :)
kartik_malik8 hours ago
love the design... congrats
TheYahiaBakourop8 hours ago
will let my designer know, it's a dev-focused product, and we're all so finely tuned to avoid slop, so design & feel was really important
zuzululu9 hours ago
new frontier models do this already
TheYahiaBakourop9 hours ago
im not sure the models do this, rather the tooling around them, however web search/extraction by the model providers is ridiculously expensive and quite slow, so going into production it makes sense to use a provider (like us)
asdev9 hours ago
So basically web scraping as a service with an API on top?
TheYahiaBakourop9 hours ago
yes but we do alot more that may not be clear at first glance, things like brand data, and for scraping handling pdf, ocr, docx, ppt, xlsx automatically
shipping a bunch of new things soon which should make it clearer, but as of today yeah
archerx9 hours ago
Great, another thing I have to block server side. Reminds me of the image leech protections that had to be in place because bandwidth was expensive. History doesn’t repeat but rhymes as they say.
pavlov9 hours ago
Maybe AI-service-blocklist-as-a-service could be a YC company.
paytonjjones9 hours ago
Better business model would be some sort of micropayment setup - allow humans, but make scrapers pay a hundredth of a cent for access.
owlninja9 hours ago
Is this similar? https://blog.cloudflare.com/monetization-gateway/
paytonjjones7 hours ago
Yep. It will be interesting to see if things move that way or not. AI scraping might get annoying but a lot of sites (eg e-commerce) have obvious incentives to allow AIs to scrape them for free (or even pay AI companies to prioritize them, like they already do with Google).
seper89 hours ago
Seems wildly expensive, furthermore not a single mention of "ip" on homepage? Not using rotating ip's, residential proxies?
AKA unusable for high value data.
TheYahiaBakourop9 hours ago
you're welcome to try it on our demo page (no signup needed), should handle everything just fine, yes we don't mention ips on the homepage
also, while it might seem expensive, we're cheaper than every other option out there, because there's no hidden credit multipliers. every single customer who uses us halved their bill + error rate
gokulrajaram8 minutes ago
[flagged]
Fno4424 minutes ago
[flagged]
ryss207 hours ago
[flagged]
[deleted]6 hours agocollapsed
laalshaitaan3 hours ago
lessgooo! context dev ftw!
TheYahiaBakourop3 hours ago
thank you, im not sure who this is though, but please email me if i can help in any way
badmonster8 hours ago
[dead]
growupdorks5 hours ago
[flagged]
TheYahiaBakourop2 hours ago
ignoring the obvious ragebait, the soc 2 "thing" is the truth, we are type 1 certified, and are in observation for type 2
Barbing2 hours ago
The year is 2003
You’re on the playground
“That’s so gay”
Age: 12
cahaya8 hours ago
Nice, @grok how does it compare to Cloudflare that also provides a REST endpoint for structured markdown data and screenshots?
TheYahiaBakourop8 hours ago
im not sure if grok is on here, i think that's an X thing
but if you were to ask me, we're more fully featured than cloudflare, and anecdotally a ton more reliable in terms of error rates. back when it was brand.dev, i actually tried to use cloudflare's apis and it was quite unreliable, so we built our own stack instead