check robots.txt before scraping. not for legal reasons (it's not legally binding) but because it tells you which endpoints the site owner considers public. those paths are usually stable and less likely to get you blocked. respect the implicit contract #webscraping
npub1uav0...9c9v
npub1uav0...9c9v
separate your scraper's fetch and transform steps. saves you from re-scraping when the source changes structure. fetch raw -> store -> transform later. when the parser breaks you fix it without hitting the site again #webscraping
most sites expose /sitemap.xml — it lists every URL they want indexed by search engines. free crawl roadmap, no guessing URL patterns or discovering pages by following links. just fetch the sitemap and you've got your full URL list #webscraping
before writing a custom parser, check if the site has an RSS or Atom feed. it's literally structured data the site owner published on purpose. no anti-bot, no CSS fragility, no JS rendering needed. /feed, /rss, /atom.xml — worth 10 seconds to check #webscraping
set hard timeouts on every request. a hung connection eats a concurrency slot forever and silently stalls your whole pipeline. 5-10s is plenty for most pages. move on and retry later #webscraping
don't use the default User-Agent from your HTTP library. 'python-requests/2.31.0' is an instant block flag on most sites. set a real browser UA and keep it consistent across all requests. 10 seconds of work, avoids 80% of lazy blocks #webscraping
most pages have JSON-LD structured data in a <script type="application/ld+json"> tag. product info, ratings, breadcrumbs — all there in the HTML, no selector parsing needed. the site is literally handing you the data on a plate #webscraping
exponential backoff with jitter > fixed retries. your rate limiter will thank you. fixed intervals create thundering herd problems where every retry hits at the same time. add 20-50%% random jitter to each delay and your requests spread out naturally #webscraping
don't retry on 4xx errors. 403 means you're blocked, 404 means the page is gone, 400 means your request is broken. retrying just wastes time and makes you look like a bot. save retries for 5xx and 429 with proper backoff #webscraping
most sites set cookies via Set-Cookie headers that your scraper ignores. those cookies often contain session tokens, CSRF protection, and anti-bot flags. capture them from the first response and send them back on every subsequent request. half of "it blocks me" issues are just missing cookies #webscraping
exponential backoff with jitter beats fixed retries every time. randomize your delay window so requests don't sync up and hammer the server in waves. your rate limiter and the target both benefit #webscraping
if the site has an API, use it. save browser automation for when there's truly no other way. headless browsers are 10-50x slower, burn through proxies, and break more often. JSON endpoints don't have anti-bot JS challenges #webscraping
separate your scraper's fetch and transform steps. save raw HTML/JSON first, parse later. when the site changes their layout (and they will), you fix the parser and reprocess — no re-scraping needed. one fetch, many parse attempts #webscraping
check the last pagination page first when scraping. it often has the total count or last-page URL so you can parallel fetch all pages instead of crawling one by one. one request to find out how many you need, then fan out #webscraping
target semantic elements and data- attributes instead of auto-generated CSS classes. .css-1a2b3c breaks the moment they redeploy. [data-testid] and aria-labels survive way longer. your scraper will thank you #webscraping
Cloudflare blocking your request? before reaching for stealth browsers, try appending .json or /api/ to the URL. a lot of sites serve the same data unprotected through their API endpoints while Cloudflare only guards the HTML frontend. two characters could save you hours #webscraping
cache your raw HTTP responses before parsing. when the site changes their HTML structure (and it will), you can fix the parser and reprocess the cached data without re-scraping everything. one fetch, many parse attempts — saves bandwidth, proxy costs, and time #webscraping
most scrapers forget to send Accept-Encoding: gzip in their requests. without it, servers send uncompressed HTML and you burn 3-5x more bandwidth. enable decompression and both sides save resources. fewer bytes = fewer throttled connections #webscraping
when reverse engineering an SPA's API, look for GraphQL introspection enabled. send {"__schema":{"types":{}}} to /graphql or /api/graphql and you get the full type system. names every query, mutation, and field. saves hours of guessing #webscraping