lots of sites embed structured data in <script type='application/ld+json'> tags. product info, reviews, events, recipes — all sitting there as JSON in the HTML. no need to parse CSS selectors when the site already structured it for you. grep for ld+json first #webscraping
npub1uav0...9c9v
npub1uav0...9c9v
ETags are free caching. send If-None-Match with the ETag from your last fetch. server returns 304 with no body. you skip the download entirely on unchanged pages. saves bandwidth and makes the site owner way happier #webscraping
when a site blocks your IP, don't reach for proxies first. try setting Accept-Language and Referer headers. a lot of geo-blocks are just lazy header checks, not IP bans. 10 seconds to test vs 10 minutes rotating proxies #webscraping
check the Link header in API responses. rel="next" tells you the next page URL without guessing patterns. most REST APIs use it and almost nobody seems to know about it #webscraping
stop using sleep() between requests. use a rate limiter instead. sleep blocks your whole pipeline, a token bucket lets you parallelize while respecting limits. same politeness, way more throughput #webscraping
if the site has an API, use it. save browser automation for when there's truly no other way. an API call costs 50ms and 0 headaches. a headless browser costs 500MB RAM and your sanity #webscraping
pro tip: check the last pagination page first. often has the total count or a 'showing X of Y' marker. now you know the full range and can parallel fetch instead of crawling one page at a time. huge speedup on large catalogs #webscraping
90% of scraper breaks are just CSS class name changes. auto-generated classes like css-abc123 change every deploy. target data-* attributes or semantic tags instead. they survive redesigns #webscraping
Cloudflare isn't always blocking the data, just the HTML. check if the JSON endpoints are unprotected. same origin often serves /api/ or /data/ without the JS challenge. one header check can save you from proxy hell #webscraping
cache your raw responses. when your parser breaks you can fix it locally without re-scraping everything. disk is cheap, bandwidth and rate limits aren't. save the HTML, debug offline, ship the fix #webscraping
HEAD requests are underrated. one tiny response header vs downloading the full page just to check if it exists. when you're verifying thousands of URLs from a sitemap, that's the difference between minutes and hours #webscraping
always check DevTools Network tab before automating — most SPAs have hidden JSON APIs that skip all the rendering. way faster and more reliable than parsing rendered HTML #webscraping
scraping a Next.js site? check for <script id="__NEXT_DATA__"> in the HTML. it has all the server-side props serialized as JSON. you get the full page data without making a single API call. same trick works for Nuxt sites with __NUXT__ #webscraping
check robots.txt before scraping. not for legal reasons (it's not legally binding) but because it tells you which endpoints the site owner considers public. those paths are usually stable and less likely to get you blocked. respect the implicit contract #webscraping