oh, i think that would turn out to be not so great as you think. i'd recommend instead using snappy or zstd on a low setting as a way to mitigate this. the database can enable this, in fact i think it already is. yes, it uses zstd at level 1 already.
btw, it creates a fulltext index for all words, it's not a reverse index though, and i hadn't thought to enable to disable it. but there is no way that even if i bound those words into some kind of lexicon and reference scheme, that i could come close to the compression or performance of ZSTD.
also, i think you WANT that fulltext search capability, and i'm reluctant to enable to disable it, even. but i could, it's a trivial switch and just a few bits of logic to disable index creation and reporting in the nip-11 that nip-50 (iirc) is not available.
i think with just the graph table and the author/p/e replaced with 6 bytes (flag and event/pubkey serial) will have a huge effect on the storage all by itself, even before the basic but fast compression scans blocks of events to deduplicate segments of it.
btw, the compression does not just merely compress single events, it compresses, i'm not sure... lemme see what claude says:
1. Badger LSM Tree Compression (Storage Layer)
Located in pkg/database/database.go:158-160:
opts.Compression = options.ZSTD
opts.ZSTDCompressionLevel = 1 // Fast compression (500+ MB/s)
★ Insight ─────────────────────────────────────
- What it compresses: SST (Sorted String Table) files on disk — the actual database files
- When it runs: During compaction, when Badger merges and rewrites LSM tree levels
- Trade-off chosen: Level 1 prioritizes write throughput over compression ratio
─────────────────────────────────────────────────
Available compression levels for Badger:
| Option | Description |
|----------------|--------------------------------|
| options.None | No compression |
| options.Snappy | Fast, moderate ratio (~2-4x) |
| options.ZSTD | Best ratio, configurable speed |
ZSTD Levels (when using options.ZSTD):
- Level 1 (current): ~500+ MB/s compression, ~4-6x ratio
- Level 3 (default): ~300 MB/s, ~5-7x ratio
- Level 9: ~50 MB/s, ~8-10x ratio
- Level 19: ~10 MB/s, maximum ratio
-----
ah yes, it has a 512mb, zstd level nine compressed in-memory hot cache with 5 minute TTL, also, that returns the already encoded JSON if the normalized filter matches the normalized filter in the cache. this has a potential 1.6gb of hot events ready to go without encoding, or any iteration of the database
there is also inlining of small events (under 1kb) in the key table (a unique feature of badger) which avoids a second iteration and is there for more common use cases where there is a lot of mutating of the values rather than, as we have with nostr, pretty much write once, maybe later delete.
yes, all that inlining is configurable, that's just what is default.
