Scraping lesson that pays off later: save the raw response, not just what you parsed.
Your parser will be wrong one day. A site changes, a field moves, you miss something.
With the raw HTML saved, you just re-parse.
Without it, you can’t go back and scrape yesterday.
The worst scraping failure isn't a 403.
It's a 200 OK with the wrong page.
A block page. A cookie wall. A "select your country" screen.
Your pipeline sees success and saves it.
It looks fine until someone uses the data.
Check what you got, not just whether you got it.
Some of you shouldn't pay for a scraping API. Including ours.
A few stable sites, one country, and nobody panics if it breaks for a day? Build it yourself. It's cheap.
The economics change when:
- You add more sources. Every site breaks in its own way, so 10 small sites can eat more engineering hours than 1 big one.
- You need data from multiple countries.
- Sites start blocking you. Debugging blocks is engineering time you're already paying for, even if it never shows up on an invoice.
The most expensive option is the one in the middle: build it, let it drift, pay for it anyway.
We broke down the costs of DIY scraping vs. using an API in more detail here:
scrapingbee.com/blog/diy-vs-…
Google, OpenAI, Anthropic and Microsoft are backing a shared robots.txt opt-out for AI training.
One rule across all four, with support rolling out mostly through 2027.
It doesn't block crawling itself. It signals that the content shouldn't be used for AI training.
Your website might look perfectly fine to people and completely different to AI agents.
Checkmarks, icons, and other visual elements rendered through CSS can disappear during HTML-to-Markdown conversion if they have no actual text behind them.
- People see the visual elements.
- AI agents reading Markdown might miss them entirely.
Worth checking what your website actually looks like when converted to Markdown. You might be surprised what's missing.
Tip: Grab a free ScrapingBee trial and ask your AI agent to compare the fully rendered page with its Markdown version.
AI is stealing Google's organic traffic.
Apparently, Google is helping too.
Kevin Indig analyzed Similarweb data from 10,000 US websites. Over 2 years:
📉 Organic search: -13.2%
📈 Paid search: +21%
In Jan - Aug 2026, organic search lost 7.1B visits while paid search gained 2.6B YoY.
Paid search offset just over a third of the organic losses.
Good news: you can buy back some of your lost organic traffic.
Bad news: you have to buy it.
Karolis Stasiulevicius retweeted
What actually happens under the hood during Web Hydration? Our lead engineer, Etienne Ellie broke this down live at OxyCon, and the demo repo is now public
github.com/ScrapingBee/scrap…
We spent €3,274 on ChatGPT ads.
Clicks: 39,779. New paying customers: 0 (so far).
ChatGPT Ads became available to French companies on Sept 2. We launched the same day.
What we tested:
- Global vs US targeting
- Optimizing for registrations vs purchases
What we saw:
- CPC: €0.04 - €1.56
- CPL: €1 - €22
- CPA: still no conversions after €3,274 spent
Since ads only show on Free and Go, there may be an audience mismatch. A meaningful share of developers willing to pay for a scraping API may already be on paid ChatGPT plans - and therefore never see the ads.
Anyone running B2B campaigns on ChatGPT seeing different results?
