Thread

I'm excited to introduce Nostr Social Duck, or just NSD, a library I've been crafting over the last two weekends that brings sophisticated social graph analysis, even to hardware that doesn't have the luxury of abundant memory. This library leverages DuckDB to perform graph operations over Nostr social graphs built from Contact List events (kind 3), and it's designed to run on all sorts of hardware, from low-end devices to powerful servers. The inspiration came from limitations I encountered with nostr:npub1g53mukxnjkcmr94fhryzkqutdz2ukq4ks0gvy5af25rgmwsl4ngq43drvk excellent nostr-social-graph library. While powerful, loading entire social graphs into memory becomes challenging, specially on resource constrained devices like older computers, mobile devices, Raspberry Pis, and similar environments. Once the graph grows, memory constraints can cause performance issues or outright failures. The recent launch of Relatr and its inclusion in the Umbrel community store (thanks to nostr:npub1ye5ptcxfyyxl5vjvdjar2ua3f0hynkjzpx552mu5snj3qmx5pzjscpknpr ) got us thinking: how could we enable anyone to run this without being constrained by their hardware? Since disk storage is cheaper than memory, how we can balance the tradeoffs 🤔... That's when DuckDB emerged as the perfect solution DuckDB is quite a cool piece of software that's been around for some time with an amazing team committed to open source. It's an embedded, in-process database designed to run on all sorts of hardware. What makes DuckDB particularly compelling for our use case are its advanced memory management capabilities. When complex queries require more memory than available, DuckDB automatically spills to disk, ensuring queries complete even if there is not enough memory, rather than crash. No memory, no problem. This feature alone makes DuckDB a strong choice, but there's more. Its advanced analytic SQL capabilities make graph analysis operations possible and performant. NSD already provides methods to get the shortest distance between pubkeys, along with a complementary method to find the shortest path that returns both the path and distance. You can define a root pubkey which creates a temporary table with all distances precomputed, making subsequent queries much faster. The library also includes methods for social graph analysis, like, get the degree of a pubkey, how many inbound and outbound connections it has, which helps determine the weight of a pubkey in the graph. There are convenient methods to check if one pubkey follows another or if two pubkeys are mutual follows. As well thanks to DuckDB in the future we could use parquet files to distribute social graph data. You can find all the details in the repo. We're currently refactoring Relatr to use NSD, and the results are impressive. By replacing the previous combination of nostr-social-graph library and SQLite database with DuckDB and NSD, we've eliminated the inefficiencies of having separate data sources. Complex queries like profile search now benefit from analytical SQL directly in the search algorithm, reducing data transfer between the program and database and returning more relevant results efficiently. I did some naive benchmarking to understand how it performs compared to the nostr-social-graph library, and it behaves pretty well. Both libraries are pretty close in performance when there's a root pubkey set in NSD. This effort aligns with the ongoing #WoTathon organized by nostr:npub1healthsx3swcgtknff7zwpg8aj2q7h49zecul5rz490f6z2zp59qnfvp8p , as we believe NSD provides fundamental primitives for performing graph operations on everyone's hardware without sacrificing simplicity of use and deployment. The library is available now and can be integrated into any js project: ```bash bun add nostr-social-duck # or npm install nostr-social-duck ``` As we finalize the Relatr refactor, we'll share detailed insights about the improvements and performance gains. The combination of Nostr's decentralized social protocol with DuckDB's efficient analytics creates a powerful foundation for the next generation of social applications. The library's repository: https://github.com/gzuuus/nostr-social-duck Related DuckDD's blog posts: - https://duckdb.org/2025/01/17/raspberryi-pi-tpch - https://duckdb.org/2024/03/29/external-aggregation - https://duckdb.org/2024/12/06/duckdb-tpch-sf100-on-mobile If you like the project please consider supporting our work by zapping, or contribute to it's development