Webrecorder is by a former Internet Archive engineer, Ilya Kreymer, who now captures online performance art for an art museum. What he's doing with capture and playback of Javascript, web video, streaming content, etc. is state of the art as far as I know.
(Disclaimer - I use bits of Webrecorder for my own archive, perma.cc.)
For OP, I would say consider building on and contributing back to Webrecorder -- or alternatively figure out what Webrecorder is good at and make sure you're good at something different. It's a crazy hard problem to do well and it's great to have more ideas in the mix.
(Disclaimer: I also do personal archiving stuff with getkumbu)
Thank you for seconding the newly updated WAIL. I am the maintainer/creator of the newly update WAIL (the Electron version) https://github.com/N0taN3rd/wail
I was unable to attend IIPC Web Archiving Conference (WAC) but the original creator of WAIL(Python) Mat Kelly did attend (we both are apart of the same research group WSDL).
If you or anyone else have any questions about WAIL I am more than happy to answer them.
That would allow constant archival of every webpage a user ever visits -- an immutable record over the user's years of crawling the web.
This is usually solved by using a proxy: http://netpreserve.org/projects/live-archiving-http-proxy/
I thought I’d add a few notes here, as there’s a few ways you can use Webrecorder and related tools.
First, Webrecorder supports two distinct modes:
- Native recording mode — http/s traffic goes to through the browser and is rewritten to point to the Webrecorder server (This is the default).
- Remote browser mode — Webrecorder launches a browser in Docker container, and streams the screen to your browser (using noVNC). The traffic is either recorded or replayed depending on the mode, but the operation is the same (we call this ‘symmetrical archiving’) This gives you a recording proxy w/o having to configure your browser or install any plugins.
You can choose this mode by clicking the dropdown to choose a browser (currently Chrome and FF) This is essentially a remote browser configured via HTTP/S proxy, and allows us to record things like Flash and even Java applets, and other technologies that may become obsolete.
- We also have a desktop player app, Webrecorder Player, available for download from: https://github.com/webrecorder/webrecorderplayer-electron/re...
This is an app that plays back WARCs files (created by Webrecorder and elsewhere), and allows browsing any WARC file offline.
Another way to create a web archive (for developers): You can use the devtools in the browser to export HAR files, and Webrecorder and Webrecorder Player will convert them on the fly and play them back. Unfortunately, this option is sort of limited for developers, but you can actually create a fairly good archive locally using HAR export (available in Chrome and Firefox at least). The conversion is done using this tool: https://github.com/webrecorder/har2warc
- If you use webrecorder.io, you can register for an account or use it anonymously. If you register for an account, we provide 5GB storage and you have a permanent url for your archive. You can also upload existing WARCs (or HARs)
- You can also run Webrecorder on your own! The main codebase is at: https://github.com/webrecorder/webrecorder and the remote browser system is actually a separate component and was first used for oldweb.today and lives at https://github.com/oldweb-today
Finally, the core replay/recording tech is actually a separate component, an advanced ‘wayback machine’ being developed in https://github.com/ikreymer/pywb
There’s a lot of different components here, and we would definitely appreciate help to any and all parts of the stack if anyone is interested! All our work is open-source and we are a non-profit, so any help is appreciated.
I'm not sure how much this would interfere with normal browsing -- it's not a typical usecase.
(Yes, yes, `wget --convert-links`, I know. Not quite as convenient, though.)
Do you think being able to download the archive locally would be useful?
Firefox: https://addons.mozilla.org/en-US/firefox/addon/wayback-machi...
Chrome: https://chrome.google.com/webstore/detail/wayback-machine/fp...
Safari: https://safari-extensions.apple.com/details/?id=archive.org....
Android: https://play.google.com/store/apps/details?id=com.archive.wa...
iOS: https://itunes.apple.com/us/app/wayback-machine/id1201888313
As it is now, I personally wouldn't use it (but it's a cool project, definitely please keep working on this idea!).
I agree with GP here, that anything billed as "My own internet archive" should be run on my computer. Not some one elses.
You can do just that via https://chrome.google.com/webstore/detail/warcreate/kenncghf... http://warcreate.com.
I am a core contributor to this project on github (https://github.com/machawk1/warcreate) and the maintainer/creator of the latest version of WAIL. So I am not biased in anyway ;)
I believe the only way to incentivise participation in such a system is by paying for timestamp'ed signatures, eg. "some subset of downloaded [content] from [url] at [time] hashed to [hash]" all tucked into a Bitcoin transaction or something. There are services that will do this with user-provided content[1]; I am looking for something that will pull a url and timestamp the content.
This would also be a way to detect when different users are being served different content at the same url, thus the need for a global network of validators.
Is it possible to prove something was done in the past? All I can think of is some sort of scheme involving destroyed information.
My focus is on the something much more so than the when. I can do my own doctoring of any data, or use some service to make something that looks real[1]. Getting some proof that this fake data existed is not what I'm after.
Instead, I want multiple, completely separate (and ideally as independent and diverse as possible) attestations that something was out there online, as proof that some person or organization intended for it to be seen by everyone as their content. Being able to prove that irrefutably seems nearly impossible today even for the present time, particularly against insider threats.
Your question regarding proving something in the past is going far beyond what I'm hoping for; it will take me quite a while to come up with anything that might be helpful for such a situation. I assume most would hit up the various archive sites, but my gut feeling is that it winds up being a probability based on how well forensics holds up / are not falsifiable.
[1] simitator.com - not linking because ads felt a bit extra-sketch!
The only way to do it should be to sync your observation to other observers as soon as the observation is made. The other observers can confirm the time then by knowing when they received the information.
Block chain with comments.
I can prove I had today's papers today, but once I've seen it I can prove it any day. So you can say "this information existed at day X or earlier".
There's also the issue of proving the content you hashed actually came from the place you say it came from. To me it seems that would require proofs of authenticity, by the source itself; not something that's easy to come by.
If you really want to create your own archive, set up a Live Archiving HTTP Proxy[1], run SquidMan [2] or check out WWWOFFLE[3].
If you want something simpler, have a look at Webrecorder[4] or a paid Pinboard account with the “Bookmark Archive”[5].
[1] http://netpreserve.org/projects/live-archiving-http-proxy/
[2] http://squidman.net/squidman/index.html
You're right, for now it's a single rate-limited HTML form and you'll have to manually collate the links to the archives you create. I'll be adding specialty features (with accounts) next. :)
Print and store pages as PDFs.
Download and save entire pages as webarchives (Safari, wget)
[1]: http://blog.archive.is/post/72136308644/how-much-does-it-cos...
I like the look. Very clean. I like how fast it's responding; better than archive.org (though, obviously, they have different scaling problems).
"Your own internet archive" might be overselling it, as other commenters have pointed out; the "Your" feels a bit misleading. I think "Save a copy of any webpage." gives a better impression, which you use on the site itself.
The "Archive!" link probably shouldn't work if there's nothing in the URL box. It just gives me an archive link that errors. Example: [1]
Using it on news.YC as a test gave me errors with the CSS & JS [2]. This might be due to the fact that HN uses query parameters in their CSS and JS, which repeat in the tesoro URL, which you may not be parsing correctly.
Maybe have something in addition to an email link for submitting error reports like the above, just cause I'd be more likely to file a GitHub issue (even if the repo is empty) than send a stranger an email.
As other commenters have pointed out, archive.is also does this, and their longevity helps me feel confident that they'll still be around. Perhaps, if you wish to differentiate, offer some way for me to "own" the copy of the page, like downloading it or emailing it to myself or sharing it with another site (like Google Docs or Imgur) to leverage redundancy, or something like that. Just a thought.
All in all, nice Show HN.
EDIT: You also may want to adjust the header to work properly on mobile devices. Still though, nice job. Sorry if I'm sounding critical.
[1] https://archive.tesoro.io/320b55cc9b78e271c94716ee23554da8
[2] https://archive.tesoro.io/a7bf03e247224bc3b4e5a7c1f2ad42b1
I know a lot of these sites have archiving features, but want something centralised and automatic.
They will love it!
This got me thinking about how a decentralized p2p internet archive could solve the trust problem that exists in centralized internet archives. Such solution could also increase the capacity of archived pages and the frequency at which archived pages are updated.
It is true that keeping the entire history of the internet on your local drive is likely impossible, but a solution similar to what Sia is doing could solve this problem: split each page to 20 pieces and distribute each piece to 10 peers such that every y pieces can recover the original page. So, you only have to trust that 10 peers out of 20 that store a page are still alive to get the complete page.
The main problem I can see right now would be lack of motivation to contribute to the system -- why would people run nodes? Just because it would feature a yet another cryptocurrency? Sure, this could hold now, but when the cryptocurrency craze quiets down and people stop buying random cryptocurrencies just for the sake of trading them, what then? Who would run the nodes and why?
[0]: https://ipfs.io/ [1]: https://filecoin.io/
extensions: Firefox "Print Edit" Addon / Firefox Scrapbook X / Chrome Falcon / Firefox Recoll
open source: Zotero / WorldBrain / Wallabag
commercial: Pinboard / InstaPaper / Pocket / Evernote / Mochimarks / Diigo / PageDash / URL Manager Pro / Save to Google / OneNote / Stash / Fetching
public: http://web.archive.org / https://archive.is/
I (obviously) think personal archives are a great idea, but republishing is a hornets' nest.
If I want my own archive, Ctrl+S in Firefox usually works fine for me.
As is it, while it's a nice service, it's still got all the issues of other archive ones:
1. It's online only, so one failed domain renewal or hosting payment takes everything offline.
2. It being online also means I can't access any saved pages if my connection goes down or has issues.
3. The whole thing is wide open to having content taken down by websites wanting to cover their tracks. I mean, what do you do if someone tells you to remove a page? What about with a DMCA notice?
It's a nice alternative to archive.is, but still doesn't really do what the title suggests if you ask me.
Instead of hosting this directly on my computer, it would be interesting to have a setup where the archiving is done via the service and I would just provide somewhere a storage space where the content would end up being mirrored (just to guarantee that my valuable things are saved at least somewhere, should the the other nodes decide to remove the content).
I would prefer this setup, because it would be easily accessible for me from any device and I would not need to worry about running some always available system. With some suitable P2P setup my storage node would have less strict uptime requirements.
Have you heard of InterPlanetary Wayback (ipwb)? https://github.com/oduwsdl/ipwb
InterPlanetary Wayback (ipwb) facilitates permanence and collaboration in web archives by disseminating the contents of WARC files into the IPFS network.
[1] https://chrome.google.com/webstore/detail/cmmlgikpahieigpccl...
Yup, API and chrome extension are next on the feature list. :)
I also second the need for user accounts. If I am to use your site as my personal archive, then I would need to log in and create a collection of my own archived sites.
Briefly: Sites are archived using a system written in Golang and uploaded to a Google Cloud bucket.
More: The system downloads the remote HTML, parses it to extract the relevant dependencies (<script>, <link>, <img> etc) and then downloads these as well. Tesoro is even parsing CSS files to extract the url('...') file dependencies from here as well, meaning most background images and fonts should continue to work. All dependencies (even those hosted at remote domains) are downloaded and hosted with the archive, meaning the src attributes on the original page tags are wrangled to support the new location.
The whole thing is hosted on GCP Container Engine and I deploy with Kubernetes.
I'll write up a more comprehensive blog post in some time, which portion of this would you like to hear more about?
How can you pay for this if it's free? It's unreliable unless its financially viable.
The page URI is a bit obscure though. I think a tresoro.io/example.tld/page/foobar/timestamp would look good.
What about big media content and/or small differences between them ?
I'm confused. It looks like image sources in "archived" pages on Tesoro still point back to the origin domain.
Edit: it works as expected. I just didn't notice the relative paths.
The site will rewrite absolute image URLs as relative ones pointing to Tesoro. For example, in the Chicken Teryaki example on the homepage, the main image is sourced from the relative location "static01.nyt.com/.../28COOKING-CHICKEN-TERIYAKI1-articleLarge.jpg", which looks like it's coming from nytimes.com, but you can check in the Chrome dev console that it isn't.
Have you found an example where it isn't working correctly? If so would you mind posting it here and I'll fix it :).
Here is an example from archiving nytimes home page:
https://archive.tesoro.io/665dbeab57a4d57d8140f89cfedc69b5
If you look at network traffic (domain in devtools), you'll see that only a small % is coming from archive.tesoro.io -- the rest of the content is loaded from the live web. This can be misleading and possibly a security risk as well.
Not to discourage you, but this is a hard problem and I've been working on for years now. This area is a moving target, but we think live leaks are mostly eliminated in Webrecorder and pywb, although there are lots of areas to work on to maintain high-fidelity preservation.
If you want chat about possible solutions or want to collaborate (we're always looking for contributors!), feel free to reach out to us at support [at] webrecorder.io or find my contact on GH.
I wonder what this site uses
If a person in the UK uses your service you're committing contributory infringement for commercial purposes, AFAICT.
Moreover, the ECD has different protections than DMCA. Particularly getting a takedown notice isn't required.
>Maybe the crawler allows for spoofing the user-agent. //
As a tort you only need to get a preponderance of evidence. IP of the crawler that made the copy puts the owner of that IP in court for contributory infringement, no?
If you make copies of parts of BBC sites and serve those copies from your server how is that not copyright infringement by you??
FWIW I like the service and do not like the copyright regime as it stands, particularly how UK law lacks the breadth of liberties of Fair Use.
