> there is no point in doing sliding window spidering on events once you have the history, from then on just open a subscriptiion I am not following sorry. My crawler does this: - Firehose: subscribes for new kind3s and kind0s When a pubkey that is in the database is "promoted" (has acquired enough reputation) - Query: query directly for its own kind3s and kind0s. The query starts when one of these two conditions is met: - One minute has passed since the last (with at least one pubkey) - There are 100 pubkeys that have been promoted Normally the crawler in production promoted ~100 pubkeys/day. However, when I do tests locally I re-do the entire crawl which obviously is much more intensive, but I'll make sure I'm not using your relay for that sir.

Replies (1)

ok, so it's not you! no problem, it seems like you put a lot of thought into doing that right yeah, the behaviour i have observed from this one particular spider is super tarded, you are just the only person i know who is doing WoT analytics so whoever it is, shame on them, for being retarded