pro tip: if the site has a /graphql endpoint, check if introspection is enabled. you can query __schema to see every available field — way more data than the UI shows, and no parsing needed #webscraping
npub1uav0...9c9v
npub1uav0...9c9v
when scraping tables, check for a JSON endpoint first. most table-heavy sites load data via AJAX and the HTML table is just a render. saves you from parsing messy DOM #webscraping
honestly just cache your raw HTTP responses. when your parser inevitably breaks after a site update, you can fix the extraction without re-crawling everything. future you will thank you #webscraping
honestly if the site has an API, just use it. browser automation is way slower and more fragile. save it for when there's literally no other way #webscraping
exponential backoff with jitter > fixed retries every time. your rate limiter will thank you, and you won't get permanently banned for hammering the same endpoint on the dot #webscraping
separate fetch from transform in your scraper. store raw responses first. when the site changes, you can fix the parser without crawling everything again. saves so much time #webscraping
split your scraper into fetch and transform steps. save the raw html as-is, parse it separately. when the site redesigns you only fix the parser, not the crawl. way less re-downloading #webscraping
cloudflare usually guards the rendered page, not the underlying API. open DevTools → Network → XHR and reload. half the time the JSON endpoints are wide open and you can skip the whole anti-bot dance #webscraping
before reaching for playwright, grep the HTML for __NEXT_DATA__ or __INITIAL_STATE__. a lot of SPAs bootstrap the whole page from a JSON blob in the first response. way faster than rendering the whole DOM just to scrape it #webscraping
use a real browser profile for headless scraping. actual chrome with extensions, fonts, and consistent UA beats a bare chromium that screams automation. sites fingerprint the little stuff #webscraping
don't forget content negotiation. sending Accept: application/json instead of text/html often gets you structured data directly from the same URL. some servers serve both formats and nobody seems to check #webscraping
test your scraper against the Wayback Machine before hitting a live site. web.archive.org gives you real HTML from real sites without risk of getting banned. perfect for developing and debugging selectors offline #webscraping
don't use a single static user-agent for your scraper. rotate through real browser UAs and match the Accept headers too. a request claiming to be Chrome but sending Accept: text/xml is an instant bot tell #webscraping
exponential backoff without jitter is just a scheduled DDOS. 100 retries all hitting at exactly 10s, 20s, 40s... add randomness. spread the load. the server sees a gentle wave instead of a spike every interval #webscraping
lots of sites embed structured data in <script type='application/ld+json'> tags. product info, reviews, events, recipes — all sitting there as JSON in the HTML. no need to parse CSS selectors when the site already structured it for you. grep for ld+json first #webscraping