Replies (106)
π«
Ah....I think it was a minimum of a million sats.... π
Jokes aside sorry to hear about this! That's horrible π
Sucks - but no risk no fun. Thank you for running all these services for the community.
Oh damn⦠best of luck to you.
Don't worry. You make it through this too. Plebs are with you
I'm just joking.
Take a deep breath, don't panic, try to calm down. There's usually one or more way to fix things. You'll be ok.
**If you're building, get a back up now! Can't stress it enough!**
Kieran
I fucked up terribly... Im not gonna pretend like this isnt 100% my fault, i should have been more careful.
Potentially:
Lost all NIP5 domains (snort.social/zap.stream/nostr.gg)
Lost all zap.stream balances (history / balance)
Lost LND wallet balance 10M+ sats (searching for wallet seed)
Lost VM configurations for LNVPS.net, (non-recoverable, gonna have to just host current VMS for the next few months until i manually rebuild the database)
Lost all code hosted on git.v0l.io (No backups, gone)
Somehow nostr.download seems unaffected.... so i should be able to get that back up soon ish..
At this time im not sure what i can recover, my backup process hasn't been running for a while, this is going to take quite a while to attempt to recover...
The last good database backups i have are from Feb 2024...
Ill make this right one way or another, im happy to accept what people tell me their zap.stream balance was or if you lost a NIP5 handle i will restore it, but its going to take some time to even get thinks back up and running so please be patient..
View quoted note →
Appreciate the transparency and honesty. Weβre in this together on a new frontier with risks and all.
π«
Good luck Kieran. Youβre solid for being straight up.
Tough but Iβm sure there is a lesson in there somewhere. π«
be good Johnny
Remember #reckless ?
Don't worry, breathe, it's not a question of life or death.
This too shall pass π«
Thatβs a hardcore loss.
Iβm so sorry to hear about this fatal flaw π« data loss is no joke.
As you rebuild, donβt treat yourself bad. This too shall pass.
Aah sorry to hear that Kieran - fingers crossed you recover as much as possible - either way I'm sure you will rebuild stronger.
RTFM - "Use extreme care when adjusting the volume size."
Donβt beat yourself up β¦ now you have the chance rebuild bigger and better.
I have most of my git.v0l.io projects mirrored on GitHub. Gotta check the remotes.
how does this affect already running lnvps instances?
No impact RN for lnvps users, just need to reach out to customers to re-create the database at some stage (which vms belong to which pubkeys)
VM's will be left running with no charge until we get back this mapping
tl;dr - I had an 8TB disk in proxmox that only had 2TB of data, wanting to free space, i used `zfs set volsize=5T` on that volume, after repairing the filesystem, only 1TB of data remains (mostly bitcoin node blockchain data and nostr.download data)
I will continue to attempt data recovery to restore those critically missing databases.
On a side note, are there any data recovery experts on nostr? #asknostr
π« these things happen, brother -- don't beat yourself up
π«π«π«
iirc, I had a balance of 25M bitcoins in my zap.stream account, pls zap it back thaaanks
At least you didn't send our private keys to Bugsnag, right?
Just kidding, but I don't think this is super bad. The worst part is the money you lost on LND, you don't owe anyone anything.
Hello, I used autopsy but I'm not an expert. Autopsy works for restoring all kind of files and its very easy to use. Even I can use it π
You being mad at you is like wasting time. Treat yourself like you would treat a child.
After you get things fixed MSG me.
Maybe i can help with backup or Server or or or
And not to pile on, but i cant even zap you π
Not an expert, but I've had good results with photorec. It's made to get files off of severely damaged disks. IDK if it can recover DB files, but worth looking into.
I have never run that command and now never will. Thanks for the heads up
Oh, and make a block level copy of the disk to work on so you don't make things worse.
β€οΈβπ©Ή
It might be worth asking how much professional data recovery would cost vs. time and sats lost at this point. If you drives are still working, it should be relatively cheap.
Remaining data appears to be complete garbage... rip everything
I second this, photorec saved me a couple of times.
Shit happens. Thanks for the transparency and honesty. ππΌ Hope you get up and running again soon!
I think he mirrored at least some repositories to GitHub, but presumably the tracked issues from his Gitea instance is gone.
I have also saved my own ass with photorec.
you do a lot for nostriches. mistakes are bound to happen from time to time. best of luck on recovery, we got your back dude π«
π«
Yeah I noticed karnage@zap.stream disappeared π€£ would love to have that back.
Have you checked on mastodon.
Alan Jude or some of the other zfs wizards
Definitely a bad day, but no one was hurt, it's just tech stuff; so take the lesson and move forward with patience. I only hope you can recover your money. If you need help, ask, we are here as a community.
Canβt seem to zap you. Invalid address
I tried tooβ¦
we appreciate all you do for nostr π«
Hopefully, we'll know in 10+hrs
The money isn't lost yet. (Announce you don't have a watchtower up for revocation secrets on the internet isn't a great call) If everyone just closes the channels with you, you get those sats back to your on-chain address. Minus fees, still seems okay.
π«π§‘
Sorry man. π©π«
@Kieran , donβt worry, we understand that such mistakes can happen. We are all human.
You got all our support. You will make it right, one way or another, i am sure. ππΌ
Keep up the good work Kieran, you're doing great π«π
Youβve got the whole of Nostr on your side buddy.
π« Don't worry, we're all with you. It happens.
Oh well I only had 24 million bitcoin there... Can I get mine first? π
I sincerely hope things work out alright for you.
Sorry to hear that man. let me know if there is anything I can do to help
On the bright side. nostr.download going down is forcing me to work on blossom / nsite stuff and make it more (self) censorship resistant
we've found the inflation bug
As a dude named Ben, I know the feeling of that moment you accidentally clicked the wrong thing and production goes down. It feels awful and I'm sure you're beating yourself up. It's impossible not to. I hope your recovery from this goes as well as it possibly can. The transparency is very much appreciated. Good luck!
wow rough
Hugs π«
Hey
@Kieran get in touch with @Albercoin in twitter (he does not have NOSTR). He offers self-host solutions un #proxmox and is a master.
@lunaticoin @Arkad igual podΓ©is ayudar aquΓ.
The good thing is that you will mean with your mistaken, awaes keep a offline backup and update daily or weekly
Cool. Ops guy here, happy to provide any advice if needed. π
Iβm so sorry Kieran! π
Let me know if there is anything I can do to help.
if you have technical details that you can share about this thing happened, it will be great if you do a write up about it.
sorry to hear that. this things happen to big companies even. i appreciate your work as an individual doing all these stuff and keeping it up.
i hope you get the datas and moneys back soon. π«
we are all one mistake away from doing the same
I have no clue what any of this means, but I once waded into the ocean to impress some girls to retrieve their frisby but did so with my phone in my pocket.
damn man, sucks for you and obviously all the affected. Hang in there as you move towards making things right.π
Shit happens.
When you say you lost the domains, do you mean you lost access to the registrar account?
Hang in there.
Did you get it and did you impress them?
Damn, this sucks. Hope you can recover as much as possible and that it's not stressing you out that much.
π«
Damn this sucks but thanks for all youβve done for nostr
I think the bigger lesson here is: donβt put ALL your systems on one hard drive, even if they are backed up.
π«
Been there π«
agreed
π«
These things happen
Thanks for what you do
@Kieranπ«π«π«
No problem, just a game.
π
hope it was worth it... π
oh that sucks. lmk if you need a hand rebuilding things back
Was trying to start my first stream today π₯² we are here to support you!
I want to understand this π
I think zap.stream may be down at the moment.
Kieran
I fucked up terribly... Im not gonna pretend like this isnt 100% my fault, i should have been more careful.
Potentially:
Lost all NIP5 domains (snort.social/zap.stream/nostr.gg)
Lost all zap.stream balances (history / balance)
Lost LND wallet balance 10M+ sats (searching for wallet seed)
Lost VM configurations for LNVPS.net, (non-recoverable, gonna have to just host current VMS for the next few months until i manually rebuild the database)
Lost all code hosted on git.v0l.io (No backups, gone)
Somehow nostr.download seems unaffected.... so i should be able to get that back up soon ish..
At this time im not sure what i can recover, my backup process hasn't been running for a while, this is going to take quite a while to attempt to recover...
The last good database backups i have are from Feb 2024...
Ill make this right one way or another, im happy to accept what people tell me their zap.stream balance was or if you lost a NIP5 handle i will restore it, but its going to take some time to even get thinks back up and running so please be patient..
View quoted note →
in hindsight no. π«£
I my mind I was like a taste wopper like you see of the menu, but in reality it was less than impressive
Kieran
I fucked up terribly... Im not gonna pretend like this isnt 100% my fault, i should have been more careful.
Potentially:
Lost all NIP5 domains (snort.social/zap.stream/nostr.gg)
Lost all zap.stream balances (history / balance)
Lost LND wallet balance 10M+ sats (searching for wallet seed)
Lost VM configurations for LNVPS.net, (non-recoverable, gonna have to just host current VMS for the next few months until i manually rebuild the database)
Lost all code hosted on git.v0l.io (No backups, gone)
Somehow nostr.download seems unaffected.... so i should be able to get that back up soon ish..
At this time im not sure what i can recover, my backup process hasn't been running for a while, this is going to take quite a while to attempt to recover...
The last good database backups i have are from Feb 2024...
Ill make this right one way or another, im happy to accept what people tell me their zap.stream balance was or if you lost a NIP5 handle i will restore it, but its going to take some time to even get thinks back up and running so please be patient..
View quoted note →
Kieran
I fucked up terribly... Im not gonna pretend like this isnt 100% my fault, i should have been more careful.
Potentially:
Lost all NIP5 domains (snort.social/zap.stream/nostr.gg)
Lost all zap.stream balances (history / balance)
Lost LND wallet balance 10M+ sats (searching for wallet seed)
Lost VM configurations for LNVPS.net, (non-recoverable, gonna have to just host current VMS for the next few months until i manually rebuild the database)
Lost all code hosted on git.v0l.io (No backups, gone)
Somehow nostr.download seems unaffected.... so i should be able to get that back up soon ish..
At this time im not sure what i can recover, my backup process hasn't been running for a while, this is going to take quite a while to attempt to recover...
The last good database backups i have are from Feb 2024...
Ill make this right one way or another, im happy to accept what people tell me their zap.stream balance was or if you lost a NIP5 handle i will restore it, but its going to take some time to even get thinks back up and running so please be patient..
View quoted note →
Kieran
I fucked up terribly... Im not gonna pretend like this isnt 100% my fault, i should have been more careful.
Potentially:
Lost all NIP5 domains (snort.social/zap.stream/nostr.gg)
Lost all zap.stream balances (history / balance)
Lost LND wallet balance 10M+ sats (searching for wallet seed)
Lost VM configurations for LNVPS.net, (non-recoverable, gonna have to just host current VMS for the next few months until i manually rebuild the database)
Lost all code hosted on git.v0l.io (No backups, gone)
Somehow nostr.download seems unaffected.... so i should be able to get that back up soon ish..
At this time im not sure what i can recover, my backup process hasn't been running for a while, this is going to take quite a while to attempt to recover...
The last good database backups i have are from Feb 2024...
Ill make this right one way or another, im happy to accept what people tell me their zap.stream balance was or if you lost a NIP5 handle i will restore it, but its going to take some time to even get thinks back up and running so please be patient..
View quoted note →
Hang in there! Here to help however I can brother!
π«
It's down at the moment.
Kieran
I fucked up terribly... Im not gonna pretend like this isnt 100% my fault, i should have been more careful.
Potentially:
Lost all NIP5 domains (snort.social/zap.stream/nostr.gg)
Lost all zap.stream balances (history / balance)
Lost LND wallet balance 10M+ sats (searching for wallet seed)
Lost VM configurations for LNVPS.net, (non-recoverable, gonna have to just host current VMS for the next few months until i manually rebuild the database)
Lost all code hosted on git.v0l.io (No backups, gone)
Somehow nostr.download seems unaffected.... so i should be able to get that back up soon ish..
At this time im not sure what i can recover, my backup process hasn't been running for a while, this is going to take quite a while to attempt to recover...
The last good database backups i have are from Feb 2024...
Ill make this right one way or another, im happy to accept what people tell me their zap.stream balance was or if you lost a NIP5 handle i will restore it, but its going to take some time to even get thinks back up and running so please be patient..
View quoted note →
I've never heard of Photorec.
I have a hard drive with a ton of files and photos on it that will not open.
I took it to a few different places but they were going to charge me an insane amount to recover (if even possible)
Can you share more about Photorec and any relevant advice?
Thank you π«‘
1. Make a copy of your drive ASAP. If it's failing, you may be making things worse just by trying to use it. ddrescue is designed to do this for failing drives. This may be all you need to do, but probably not.
2. Read the documentation for photorec or any other tool before doing anything. It's easy to make things worse if you don't know what you're doing.
CGSecurity
Digital Picture and File Recovery
PhotoRec is a free and open source file carver data recovery software tool.
so sad. I love your site. It's not easy running big websites, with severs being so expensive having backups is difficult. It's happened to me before.
Don't give up.
My worst nightmare π€―π€―, Iβm so sorry, and if you need anything, Iβm here for whatever you need.
STEADY LADS
π«
ouch.
Kieran
I fucked up terribly... Im not gonna pretend like this isnt 100% my fault, i should have been more careful.
Potentially:
Lost all NIP5 domains (snort.social/zap.stream/nostr.gg)
Lost all zap.stream balances (history / balance)
Lost LND wallet balance 10M+ sats (searching for wallet seed)
Lost VM configurations for LNVPS.net, (non-recoverable, gonna have to just host current VMS for the next few months until i manually rebuild the database)
Lost all code hosted on git.v0l.io (No backups, gone)
Somehow nostr.download seems unaffected.... so i should be able to get that back up soon ish..
At this time im not sure what i can recover, my backup process hasn't been running for a while, this is going to take quite a while to attempt to recover...
The last good database backups i have are from Feb 2024...
Ill make this right one way or another, im happy to accept what people tell me their zap.stream balance was or if you lost a NIP5 handle i will restore it, but its going to take some time to even get thinks back up and running so please be patient..
View quoted note →
@Kieran if you need any assistance with any of this dilemma, donβt hesitate to reach out. I will reply to your DM. It would be a pleasure to assist you if required. You [and many others] work hard contributing to this ecosystem. Peace, love and hugs.
π«
@Kieran Were dvr.zap.stream and data.zap.stream affected? I get 503 errors from dvr and 404 not founds from data.
Yea the dvr.zap.stream went down, all the data was lost for old recordings.
Aw man. Sorry to hear that
