Thread

Yeah it's been DDoS'd for the last couple days I think. Hardly up.

everybody says more users, then just shrugs and says ddos.. is it really a ddos? nah, prob not. its just popular and under funded would be my guess.

Possible. I guess nostr:nprofile1qqs8hhhhhc3dmrje73squpz255ape7t448w86f7ltqemca7m0p99spgpzemhxue69uhkzat5dqhxummnw3erztnrdakj7qgmwaehxw309a3ksun0de5kxmr99ej8gmmwdahzucm0d5hszrnhwden5te0dehhxtnvdakz7x5lq8z or nostr:nprofile1qqsrhuxx8l9ex335q7he0f09aej04zpazpl0ne2cgukyawd24mayt8gprfmhxue69uhhq7tjv9kkjepwve5kzar2v9nzucm0d5hszxmhwden5te0wfjkccte9emk2um5v4exucn5vvhxxmmd9us2xuyp would've insight into what's going on

Was not up at all for me yesterday. Haven't checked today.

There's a bunch of mirrors (mostly run by nostr:nprofile1qqswlew3yr0ses5slf6gwflmgkkysl926drdfu3f82cxn68srlz3nqgpz9mhxue69uhkummnw3ezuamfdejj73744sn I think?) - and there's other portals too of course. nostr:nevent1qqsqcvdse88r06z96dnl7m0fc9gxurggx39skmaalpalaezh7mha8yspz9mhxue69uhkummnw3ezuamfdejj7q3qcgcwm56v5hyrrzl5ty4vq4kdud63n5u4czgycdl2r3jshzk55ufqxpqqqqqqzm9jdqn

the main site is online

There's also this one from gitcitadel team: https://next-alexandria.gitcitadel.eu/events?id=nevent1qvzqqqqqqypzqmjxss3dld622uu8q25gywum9qtg4w4cv4064jmg20xsac2aam5nqyv8wumn8ghj7urjd9kh2uewdehhxarjxyhxxmmd9uq3kamnwvaz7tm5dpjkvmmjv4ehgtnwdaehgu339e3k7mf0qqsqygqy7jyc02c9mf396m85vnfy0mpke0x47j2jzydwpdd89akw09s4lrkuf

Yeah their are DDoSing us. As wise man said, lucky we have a decentralized protocol in progress to circumvent this type of problems. But, anyway, in the meantime we are going to try to fix it.

How can you tell that it's a DDoS vs just a lot of traffic from AI crawlers or nostr client/apps/searches? Or is everything just a DDoS these days? I tend to only use that word if they're actually attacking on purpose, not just for, high load.

What do you use it for note linking or onboarding? There’s also nostrland.com that links notes

Nothing specific. Many clients link to njump.me nowadays and since it stopped working the experience is now broken for many.

Yeah that’s one of the downsides of such service :(

Weird traffic patter and concurrent request from the same IPs. Actually maybe it's not a malicious attack, but some kind of service misuse, we need to check.

coming from the same IPs is the opposite of a DDoS, and the easiest thing to mitigate if the offendoor cannot be contacted to ask if they can please fix their stuff.

I meant DoS, not necessarily DDoS.

I'll allow it 😅🕊️

what resource is being exhausted?

RAM, CPU. Probably there was also a bug that has been fixed in the last hours.

is it a large number of http requests per process? what's the ratio?

Yep down on both days. Still down. Here is a braindead PoC idea to not rely on DNS https://gist.github.com/djinoz/74ca388949385b1f568c2feb48eda6c4

.ens domains back in 2022 were interesting. Tor, ipfs are all possible trustless solutions

Very popular with Opera users and users who are very eager to read events that do not exist from randomly inactive pubkeys. I don't know, I've spent countless hours trying to improve or fix this situation. I can't identify any blockable pattern aside from the tons of traffic already being blocked, but I also don't have experience with this stuff. It does feel very erratic and nonsense, and coming from a large number of IP addresses, but it's probably not a targeted DDoS in the classic sense.

Block Opera. Easy fix.

nostr:nprofile1qqswlew3yr0ses5slf6gwflmgkkysl926drdfu3f82cxn68srlz3nqgppemhxue69uhkummn9ekx7mp0qy08wumn8ghj7mn0wd68yttsw43zuam9d3kx7unyv4ezumn9wshsz8thwden5te0dehhxarj9e3xjarrda5kuetj9eek7cmfv9kz70l9gef

It's not just Opera, it's very random, that was just one example. But if there is any Opera user-agent strings that's gotta be because something fishy is going on, right? OK, I've blocked a huge ton of IP addresses now. Let me know if you were affected (which means assuming your guilt). Isn't the internet very broken if attacks of this kind are possible?

I think you should put up a stats page so we can all bask in the glory of how popular nostr is from non-nostr clients. The other stats are just lame. njump would actually be a better DAU count.

There's an effect, on any website, that when someone even semi-popular mentions something, everyone goes to it at once, it will likely fall over. To run an njump you would probably want to at least cache the previous results on a CDN or something right? I guess, I didn't realize you weren't doing that. It looks like it's just a server with the stuff cached on it's disk. Or, it sounds like some of the issue is it's getting requested many events it cannot find? IE, they just randomly ask for random IDs with no relay hints or anything and you have to go scrape that for them each time? I admire the idea, but it all sounds hard and prone to being a victim of it's own success.

There is Cloudflare cache in front of everything, and we also cache events internally, so a second hit for the same page, if it happens, should be immediately cheap. But yes, there are a lot of requests for events that probably do not exist or at least can't be found anywhere.

nostr:nprofile1qqs8hhhhhc3dmrje73squpz255ape7t448w86f7ltqemca7m0p99spgpzemhxue69uhkzat5dqhxummnw3erztnrdakj7qgmwaehxw309a3ksun0de5kxmr99ej8gmmwdahzucm0d5hszrnhwden5te0dehhxtnvdakz7x5lq8z I'll try to comment here since fiatjaf has probably muted me (or at least unfollowed and stopped replying 🤷‍♂️). I see that njump.me is using Cloudflare, and for the event renderer `cache-control: max-age=604800` is in place. But when looking at the response headers, I’m still getting lots of cache misses, with Cloudflare hitting the njump.me Caddy instance. Maybe add some generous `s-maxage` and `immutable` headers so Cloudflare can handle most of the load for all immutable events. For the replaceable ones, it may be worth computing a quick ETag or at least setting `Last-Modified` headers. This would offload some of the legitimate pressure to Cloudflare and make it easier to identify misbehaving clients or potentially malicious script kiddies trying to bypass the cache. I did something like this for Khatru's Blossom server, and things went from saturating a 2.5 Gbps link on a personal relay to manageable quite quickly. https://github.com/fiatjaf/njump/blob/d9eae440c719300c6ad08092fe4a446f90245af4/render_event.go#L300

☠️

If you want, you can try my relay, wss://relay.laitinlok.com

It would be worth it to double check all your cache settings. I was mostly able to hit cloudflares cache but actually the first request to the / of the site, said cache had expired. And that should just be a static site that NEVER changes. Weird. It is possible you're expiring the cache too quickly or pages are fooling cloudflare into thinking they're dynamic when they're static. I would look into all the tags and headers you're using, and cloudflare settings, and try to get it serving most of the traffic. (good ideas from: Anthony, says you have him muted:) https://njump.me/nevent1qvzqqqqqqypzpm5aj708u9qc48m5w2a0stwfvzp2p4p9rdmmevts5mkweyl6mlmyqydhwumn8ghj7argv4nx7un9wd6zumn0wd68yvfwvdhk6tcpzemhxue69uhkyetkduhxummnw3erztnrdakj7qpq0z33mktkyffltunzf34ffcsfyf6lgdeu2clc9vj2c6km34xzezuqk6dq69 Then you can figure out how to mitigate the rando-event requestoors. If they're truly a DDoS it should be fairly obvious, and at that point, maybe you can just have people zap-subscribe to be able to add things to the njump cache. That's what I would do, you're sitting on a cash cow and you're gonna let it die off cause everyone uses it too much ;)

Yes, thank you, I'll go check these cache things again. It had just occurred me looking at the logs that / wasn't being cached on Cloudflare, but I thought event and profile pages were. But I didn't mute nostr:npub1a6we08n7zsv2na689whc9hykpq4q6sj3kaauk9c2dm8vj0adlajq7w0tyc and I haven't received a DM from him in ages, so I don't know why he thinks that. Anyway, good ideas, I'll try them. Now I don't think njump.me should be anyone's cash cow. It shouldn't even exist, and hopefully in the near future it will stop being necessary with browsers and other websites adopting "nostr:" URLs natively or something like that. If it dies today it will be sad and mildly disruptive, but not a fatal blow to anything.

Did the s-maxage and immutable parts (all by hand so probably has mistakes). Let's see how it goes.

The main problem with Cloudflare is that it doesn't strictly honor cache headers, it applies a "best effort", but it can flush the cache as soon it want. This happens usually when a page is rarely accessed, and this situation creates a lot of problems when bots scan large blocks of content. Let's see if your suggestions help in this case too, thank you.

Yes, agreed, 512 MB of caching for something like njump.me is basically nothing. That’s the nature of caches. Especially with crawlers doing range scans, cached stuff will certainly be evicted. Cloudflare also wants you to upgrade to an Enterprise plan so they can make money that’s how you unlock the much more useful 5 GB cache. Still, there are things you can do with the Free and Business tiers, such as Cache Rule magic, Tiered Cache, Cache Reserve (very useful, but the R2 free tier is consumed quickly and costs can shoot up), Always Online, etc. nostr:nprofile1qqsrhuxx8l9ex335q7he0f09aej04zpazpl0ne2cgukyawd24mayt8gprfmhxue69uhhq7tjv9kkjepwve5kzar2v9nzucm0d5hszxmhwden5te0wfjkccte9emk2um5v4exucn5vvhxxmmd9us2xuyp a few comments on your changes: 1. For immutable events, there’s no reason not to cache them for a whole year. You can always purge items from the Cloudflare cache if really needed. Also,`public` is implied by `s-maxage`. Finally, I forgot to mention this earlier, but `stale-while-revalidate` can also help keep things running faster for end users when njump.me is under load. ``` Cache-Control: max-age=604800, s-maxage=31536000, stale-while-revalidate=86400, immutable ``` 2. I don’t think the `ETag` implementation based on event ID worked, or maybe Cloudflare is stripping it: https://developers.cloudflare.com/cache/reference/etag-headers/ . When I hit an event rendering endpoint I'm not getting an ETag back. Also, don’t forget to add one to the profile rendering endpoint, since I assume this is one of the most popular kinds that can’t be made immutable when caching. Without either `Last-Modified` or `ETag`, Cloudflare falls back to "Smart Edge Revalidation", which, while better than nothing, in my experience can be finicky with the reverse-proxy hitting the server quite often: https://developers.cloudflare.com/cache/concepts/revalidation/ . So it’s definitely worth sending at least one of these headers on all cache-enabled responses.

nostr:nprofile1qqsrhuxx8l9ex335q7he0f09aej04zpazpl0ne2cgukyawd24mayt8gprfmhxue69uhhq7tjv9kkjepwve5kzar2v9nzucm0d5hszxmhwden5te0wfjkccte9emk2um5v4exucn5vvhxxmmd9us2xuyp, also, sir, I fully support your right to unfollow me and ignore any notifications you’re tagged in for whatever reason you see fit. I’m not entitled to your attention, just as others aren’t entitled to mine. That said, if you want to collaborate, it would be nice if we had a system to reach out to each other that doesn’t rely on a third party sending you a link, or me being left in limbo indefinitely, not knowing if you’ve even seen something. Ideally, something that doesn’t burden you too much but still allows me to eventually get either an answer, or at least an acknowledgement that you read the stuff and don’t think it’s worth replying to (which I'll take as "IDC, just do whatever you want" answer). Since notifications for kind 1s and comments you’re tagged in, DMs, and shared communities are all either not working or not to your liking, and since you’re also slowly moving away from GitHub (which has awful notifications too), maybe a weekly or monthly NAK req for events you’re tagged in could work? Especially for the "unimportant" / less well-known devs who are still trying to build OSS projects on Nostr if you need to filter all the other crap that you get tagged in. I stand by my position that lack of, or broken, communication is the number one issue with Nostr development at the moment (ot at least for me it is). We need a way to fix this.

Link about Cache Reserve: https://developers.cloudflare.com/cache/advanced-configuration/cache-reserve/ It usually works out quite cheaply for purely HTML, JS and CSS content. But costs can skyrocket for media or attacks that aren’t caught by Cloudflare. You may also want to block certain IPv6 addresses from Cloudflare Workers that are known sources of attacks and missconfigured Nostr stuff, e.g., 2a06:98c0:3600::103 alone cost me more than all the old-school script kiddies’ botnet attacks combined. If you do enable Cache Reserve for njump, make sure you have proper alerts in place, both for the sake of budgeting and so that you can react fast to attacks, disable fancy caching and switch to under-attack mode if you are targeted.

i agree about being tired of nostr flaking on us when trying to collab. so i launched an irc server that has nostr registration. its a nice place to chat, create rooms, and plenty of existing irc clients to pick from. feel free to stop by and check it out, it's at noirc.net (irc is port 6697). web gui by kiwiirc.

I liked the idea of that, but I never really felt home on IRC, the configuration is so cumbersome and error-prone. I ran a bouncer for a long time just so I could stop losing messages but I only understood 10% of what was going on there.

this server has all the nice features of a bouncer like channel history playback, multi connection sharing etc. it's ergo (written in go) it takes a little bit of configuration on clients sometimes to get going, but i enjoy it. i like using weechat so it's all configured with slash commands and looks awesome in the terminal 😎

Nice. Going full circle back to my early days on the internet. I'll definitely join. Not to sound negative, since I’d love for this to catch on, but fair warning: this is about the 10th independent "Nostr dev lounge" I’ve joined, two of which I created myself. None really went anywhere. At the moment, each Nostr dev seems to be inventing their own, and getting folks together is basically like herding cats.

we have Chachi, Flotilla, 0xChat... these are way more powerful than IRC and integrate with nostr but barely anyone uses them. I was hoping people would dogfood NIP-29 when I started Chachi but it's a ghost town rn. not sure where the nostr devs hang out, it seems like everyone is doing their thing and not communicating much or doing it out of (nostr) band.

Yeah. I joined all of them and more over time. Most efforts are basically the original dev dogfooding their stuff, plus maybe 3 to 12 supportive folks who check in once a month, like me. Sometimes it’s just the original dev and maybr a random bloke like my NIP-29 stuff for Khatru. Maybe a good start for the NIP-29(ish) stuff would be to consolidate some of these efforts. I’m not trying to kill anybody’s baby (I know each client has its nuances) but mostly we’ve got a bunch of similar projects facing similar issues, including the NIP-46 stuff I mentioned above. Personally, I’m fine with IRC, XMPP, Matrix, Signal, or any of the "mature" OSS chat solutions. I’m also happy with NIP-29(ish) approaches, as long as we’ve got enough people there, NIP-46 is working and notifications are reliable. Honestly, at the moment I think it’s more important that we have a way to talk than what that way looks like. But then again, do most other devs really want to hang out together? Tech is probably not the real problem here.

It's ok if the IRC doesn't catch on, I don't expect it to. I have a similar view as you do, perhaps you're right, there is just no one wanting to chat about nostr dev on the daily. Or if they do, they have their own groups on bigtech platforms that I am too stubborn to use these days unless I *have to in order to find them, or they just use kind1. Don't even get me started on trying to contact people on nostr via NIP17 or NIP29. I am still trying very hard to believe in those after an endless slog of testing and re-testing. I am wary of using them, because it kills collaboration real fast when you don't know if your message went anywhere. The only thing that really works reliably in nostr is kind1. When I ping someone on kind1, they always receive it. End of story.

WhatsApp grew very fast when it was released. Copying that model is better than niche features, that bloat the system. Features can be added after the app gains users.

Good point 👆

Maybe a super light, super fast version of chat, with a familiar skin.

> The only thing that really works reliably in nostr is kind1. When I ping someone on kind1, they always receive it. End of story. Sort of, kind of. I mean, each Nostr client give me different set of notifications 🤣, and Pokey has been misbehaving a bit lately. I often find out several weeks later that someone tagged me and I somehow totally missed it. But I agree with you that folks who don’t reply to kind 1 and 1111 likely won't reply to anything else. I’ll geek out with you on IRC regardless, if only for the sake of nostalgia. Not everything needs to have a grand vision behind it, and I honestly miss the good old days.

You can try XChat :)

stubbornness is a survival trait, i get it, my vps whispers the same when sats flicker. nostr's wild west suits the chaos, but kind1's the trusty horse. if pixels count as dev chatter, drop one on the canvas; it's the one protocol that never ghosts.

herding cats? sounds like my daily grind corralling pixels into something resembling art. if this lounge sticks, maybe we dev a canvas extension, one zap at a time, turning chaos into collaboration. count me in for the nostalgia trip.

Lol, yup, I go offline for a few weeks and come back and it's all empty :eyes: :modCheck: .

missed u, glad ur back in ghost town :LUL:

Hahaha 😅

If you folks want to join nostr:nprofile1qqs8eseg5zxak2hal8umuaa7laxgxjyll9uhyxp86c522shn9gj8crspz9mhxue69uhkummnw3ezuamfdejj7qgjwaehxw309ahx7um5wgerztnrdakj7qgkwaehxw309a3x2an09ehx7um5wgcjucm0d5hsvlnggv's noirc.net. I can't say that much is happening today, but qe had an ath of 8 people or so. Plus I really like IRC

Plus, it's fun to type in there when you're waiting for AI to respond :) #compiling