Scrapling 🕷️

🕷️ Scrapling tip: the adaptive feature stores matches by domain + identifier, not full URL. Saving .price on many product pages can overwrite earlier matches. Use a product ID in identifier to keep each saved match separate 🎯

🕷️ Scrapling tip: page_setup can rewrite API requests before the page sends them. Use page.route() to raise a supported page size from 10 to 100 and load more items per call. The API must allow the larger limit. ⚡

🕷️ Scrapling tip: Network history spans page loads. Use search(after_id=cursor) to read only new saved API replies, then advance the cursor with reply.meta["network_id"]. No more reading page 1's replies again on page 2. ⚡

🕷️ Scrapling tip: page_action lets you read nested frames that response.css() cannot see. Get the frame's HTML with frame.content(), then parse it with Selector. Reuse the loaded browser page without a second HTTP fetch. 🎯

🕷️ Scrapling tip: The adaptive feature has a match cutoff: percentage=80. If a price vanishes, the closest match may be "Sold out". This demo rejects that match. Tune the cutoff on real pages; it is a similarity score, not confidence. 🎯

🕷️ Scrapling tip: Two prices in one tag? Pass ignore_tags=("del", "s") to get_all_text() to skip the crossed-out price. It keeps the text right after </del>, even the lone "$" in this example. You get "$29.95" in one call. 🎯

🕷️ Scrapling tip: Paginating a JSON API with POST? Keep the spider's duplicate filter on. It checks the JSON body as well as the URL. Send page 1, page 2, then page 2 again to the same endpoint. Only two requests reach the server. 🎯

🕷️ Scrapling tip: You can use a regex inside XPath. Bind the EXSLT regex namespace and use re:test() to match a SKU on each row. One query gets Lamp and Desk and skips the Flash sale row. 🎯

🕷️ Scrapling tip: Two GraphQL calls hit /graphql. Which reply has the price? Turn on record_requests=True. Each saved reply carries its POST body in meta["request_body"]. Match the operationName and read the price without sending the query again 🎯

🕷️ Scrapling tip: RemoteSession(new_context=False) uses an open browser's login state. Connect to a browser with remote debugging enabled, then scrape without exporting cookies. When the session ends, your existing tabs stay open 🎯