always check DevTools Network tab before automating. most SPAs load data from hidden JSON APIs — skip all the rendering and just hit those endpoints directly. way faster and way more reliable than parsing rendered HTML #webscraping

the most underrated scraping tool is your browser's copy-as-cURL. right click any network request → copy → paste into terminal. instant working request with all headers. then swap curl for your HTTP client of choice. fastest way to reverse engineer any API call #webscraping

when a scraper breaks at 2am, the first question isn't why did it fail — it's do you have the original data saved. raw response caching is boring but it's the difference between a 10 minute parser fix and a full re-scrape that costs real money #webscraping

rotating user-agents isn't enough anymore. sites fingerprint your TLS handshake, accept-language order, and viewport size too. if your UA says Chrome 120 on Windows but your TLS cipher list matches Python's requests library, you're getting blocked. rotate the whole browser profile or nothing #webscraping

Before building a scraper, check the site's sitemap.xml and robots.txt. Many sites list every page URL in their sitemap, which means you can skip crawling entirely. Just fetch the sitemap, parse the URL list, and request each page directly. Fastest path to full coverage with zero crawl logic #webscraping

most bot detection doesn't need javascript challenges. it just checks if your headers look like a real browser. mismatched user-agent and accept-encoding, missing accept-language, wrong referer — these are the tells. fix your headers before you reach for stealth browsers #webscraping

normalize your URLs before queuing them. strip UTM params, trailing slashes, and fragment identifiers. one page with 6 tracking variants = 6 wasted requests. URL deduplication is the easiest way to cut crawl volume in half #webscraping