You need to use an outbox model, so check for activity based on the relays each individual user set in their relaylist, you should probably cache a map of npub -> relay i do use exclusively small relays, i bet you will find that @fiatjaf also shows as inactive often