Real-time web access for your AI

Joined September 2018
Meta's new terms cover crawling for "advertising and business products." They start Nov 20. We read 47 news sites' robots.txt files. 28 block Meta-ExternalAgent, its AI training crawler. 2 block Meta-ExternalAds, the one built for ads.
1
2
107
Meta says robots.txt preferences are respected. The revised terms don't say so. Publishers took that to EU and UK regulators, and the date moved from Oct 30. Save a dated copy of your robots.txt either way.
1
9
Seltz (@seltz_ai) returns web search results to AI agents in about 300ms, as full-text documents from its own index. Five endpoints, one per job: Search for raw documents, Answer for one cited answer, Fetch for URLs you have, Monitor for what's new, Agent for deep research.
1
2
5
229
Pricing is per search, not per result: $3 per 1,000 on base, $5 on pro, and new accounts get $20 free. Setup in Claude Code is one claude mcp add command. In Seltz's run of Exa's benchmark: 327ms median, 0.53 precision at 10. Exa led on top-1 accuracy.
1
84
Are residential proxies legal? Yes. The ones in the headlines took devices without asking. What gets networks shut down is taking devices without consent. Sourcing, use, and local law decide
1
3
127
Agentic commerce could handle up to $1 trillion in U.S. retail by 2030 (McKinsey). It runs on 4 open protocols: ACP, UCP, AP2 and TAP. We checked 36 well-known stores for the file an AI shopping agent reads first. 11 had one. Only 2 of 22 big retailers did.
3
1
4
192
Gaffa runs a real cloud browser behind one REST call. Send a URL and a list of actions, get back markdown, a PDF, a screenshot, or JSON. For residential IPs, Gaffa works with several proxy providers, and Massive is one of them.
2
7
217
Myth: an AI agent doesn't need a location. It already has one. Sites read it from the IP, and the page changes with it: search results, AI answers, prices. If you don't choose it, your server's data center does.
1
2
165
Daily AI agent requests on Cloudflare grew more than 1,700% in a year, by Cloudflare's count. At Birthday Week two weeks ago, it added ways to charge that traffic. A thread on what is new, and what was already there:
3
4
267
For agent builders, "allowed or blocked" is now three lanes: identified, priced, and open. Log which lane each source puts you in, and treat a 402 as a budget decision, not an error to retry.
1
1
68