I've become a dev again. I have been building something in stealth mode up till now.
PageHold is a distributed version of the Wayback machine (Web Archive).
Archive search - PageHold
Holding page for now, it only contains a copy of mikehardcastle.com
The wayback machine (web archive) is a great historical resource for the web, giving snapshots of the web going back to 1996.
It suffers, however, from capacity and funding issues. As well as being under regular attacks, both legally and technically.
So I figured it was time to decentralise and build an app that allowed anybody to take web snapshots to store locally and share them globally.
I have just published a standalone version of this tool on GitHub:
GitHub
GitHub - snurge/pagehold: A self-hosted website archiving service for private, local snapshots.
A self-hosted website archiving service for private, local snapshots. - snurge/pagehold
Feel free to download, play and contribute however you like.
The next stage will be to launch a global federated service that allows individuals to add their personal archives to a global, searchable resource and provide a public search engine.
I have already built this infrastructure, but I wanted to test the standalone service first before launching a fully federated service. Apart from the technical challenges, there are many legal, copyright and other issues surrounding a public service.
I am considering building in revenue stream options;
Either for myself, by offering pre-configured VPS versions that can be spun up and run as a paid for service.
Or by using a high search usage charging model, charging individuals or companies fees for bulk archive searching. This would collect and distribute fees among content contributors.
These are all options ready to go, depending on the feedback.
Before you ask, this isn’t a Bitcoin or NOSTR model, but I expect it will evolve into both over time.
Replies (28)
I want to ask how this could become uncensorable but then I think about *that* side of nostr and some things need to be censored. Le sigh.
Bravo!
Not every project on here has to tick every toxic maxi checkbox 😂
I'm migrating my node back to Core from Knots right now, so I changed my address to my WoS, blame them 😂
View quoted note →I just hate things leaving the internet because they get a call from somebody.
Ah, that, yes. I see.
This may help solve this in time.
Your profile will need updating for this new role.
😂🤣😂
I've only just updated it to show
Chief Ferry Officer - (CFO) 😂
I noted that 😂🤣😂
Fantastic project 💥
This is a very noble cause!
Just as way back machine is part of archive, this could make a good companion to the #alexandria project. Maybe?
@LaeserinWe have spent some time internally discussing wikis and archiving in general as part of alexandria and the nostr data model for them. Mostly around what long term storage of large datasets is. It turns out I was totally wrong in terms of size of text media. All of wikipedia and other archives are round about a dozen GB each in nostr nkbip json format.
The short of this
@mike is storing data on relays in the nkbip kind 30040 and 30041 event formats. Then all supported clients like the ones
@Laeserin hosts on imwald, and #alexandria support them :)
We already have Wikepedia, Project Gutenberg, and Open Library on Nostr relays.
Yeah, just checked. Average is 15 GB per, 45 GB altogether.
The hard part will be where to get all the storage from and having said storage still function 20 years from now. But I really like this idea, archive.org is immensely valuable but also very vulnerable to legal issues.
I'm not so sure. Me storing 3 websites and you storing 5 doesn't require much storage at all, even if we take snapshots once a month.
There are, however, other difficulties. Aside from copyright and other legal issues, the main problem is how to store dynamic or user generated content, or not so much how, but what.
Take Google Maps for example. I may want to scrape a small area, say my home town, but I'm not going to scrape much else. But if you scrape your home town and others do the same, then we build an incomplete, but constantly improving capture.
How does this get indexed by the search interface and what happens to the missing parts?
Then we have stuff like our personal stuff, like Facebook (I know I said a swear word on here, but hear me out). There's nothing to stop me capturing my feed on a certain day, but I want to keep that private. That's an option within the capture settings, but what if I don't keep that private?
There's all sorts of nuance that my agents are working through right now.
Just to be clear, to quote the last line of my post:
"Before you ask, this isn’t a Bitcoin or NOSTR model, but I expect it will evolve into both over time."
What she said :D
The principles match already, just not the functionality right now.
I read the last line, but there are versions of this built on nostr already, just an fyi.
Yes and no is the answer.
There are also several traditional capture services.
There are also two major public capture tools, one of which has been open sourced by the Wayback machine itself. I am incorporating both of these into this service.
N.B. Capturing the modern web is an extremely difficult problem and not possible to solve fully.
This is an attempt to do something different.
The differences are so broad, I don't know where to begin and if I try, the temptation would be to push back and argue individual points, which would miss the whole picture.
Yes, I read everything you wrote, everything all the other projects (NOSTR and traditional wrote) and everything about capture engines, federation systems, the law across 195 countries, including copyright and indemnity.
Remember that feeling when your non Bitcoin friends tell you about Bitcoin and they don't understand the things you tell them and they don't think you understand the things they are telling you 😂
Then I don't understand how this is different. If the event capturing the signal is on a relay, and the archive itself is saved on blossom, you have federation a semblance of federation without the additional requirements.
I think you've forked one tool within the services I've built and like Luke, you're wondering why I'm not following your branch 😂
I think you completely misunderstood my original quote:
"Before you ask, this isn’t a Bitcoin or NOSTR model, but I expect it will evolve into both over time."
Ignore the second half and you will be fine.
There are lots of tools being built around this area, including some NOSTR tools such as the Android app you indicated, as well as several larger projects focusing on specific areas of the web, or specific types of web content.
There are also lots of tools that are being developed to address specific issues that the wayback machine face.
I am using some of them and I've rewritten many of them to include in my solution.
But I'm not trying to build tools, I've built a solution that allows organic growth and can, over time, replace, or at least augment the wayback machine.
It would be like you asking Satoshi why he didn't use RSA encryption instead of SHA-256, while not thinking about what Bitcoin is or does.
Keep trying, you Manhattan Project deviant.
🫶🏻
