Saving 172 BBC websites with BitTorrent

A recent budget cut at the BBC meant that as many as 200 websites were going to be shut down. However, an individual named Ben Metcalfe crawled 172 of the websites before they were deleted and has made them available via BitTorrent. I don't think Metcalfe really needed to expend the effort since it's likely the Internet Archive has archived the sites or will shortly, but it's nice to see an individual being pro-active in ensuring the sites would not be lost. The loss of the sites is a big downer for some who have contributed significant content to the sites in the past.

Memento wins Digital Preservation Award 2010

Congratulations to Herbert Van De Sompel and Michael Nelson for being awarded the Digital Preservation Coalition's Digital Preservation Award 2010 for the development of Memento.

"‘Memento offers an elegant and easily deployed method that reunites web archives with their home on the live web,’ explained Richard Ovenden, chair of the Digital Preservation Coalition. ‘It opens web archives to tens of millions of new users and signals a dramatic change in the way we use and perceive digital archives.’"

I've been working with Herbert and Michael on the development of the Memento Browser for Android. It's great to see these guys being recognized for their hard work.

Memento Browser for Android is available

I've just created a home for the Memento Browser for Android, a project I started working on this past summer. The free Android app allows you to view older versions of web pages by merely selecting a date. The browser uses the Memento protocol to find archived versions of the page and displays whatever page is closest to the requested date.

For example, the screenshot below shows the browser viewing cnn.com:



If you wanted to see what this page looked like on Sept 7, 2007, you could select that date, and in a few seconds be looking at this archived page from WebCite:



Note that the page displayed is actually one day later than the requested date. That's because the browser was not able to find an archived copy on the exact date requested. The browser is only displaying archived copies from Internet Archive, WebCite, and a few other archives. While they have a huge amount of the web archived, they certainly don't have everything archived.

You can download Memento Browser here. I am working on an iPhone version of the app with a colleague of mine, but I don't have an ETA for it yet.

If you don't have an Android device, you can still download the MementoFox add-on for the Firefox browser which does the same thing.

Finally, you can watch a demo of the browser in action here.

Memento: Adding time capabilities to the Web

This summer I'm working on a research project adding Memento support to the Android platform. I'll talk more about my project at a later date, but first I want to provide a quick overview of Memento.

Memento is an architecture which allows a web browser to seamlessly access older versions of web pages. It allows you to "time-travel" on the Web.

The best way to explain this is with an example. If you were to access cnn.com, you would be presented with today's version of the page. But what if you wanted to see how it looked one year ago? You would need to visit the Internet Archive's Wayback Machine to find a list of old copies of the page they had archived, and you would need to click on one of the links. And if IA didn't have the page archived, you would have to search other web archives for the archived version. This is potentially a lot of work.

Memento makes access to archived versions of a page transparent to the user. Using a web browser that supports Memento, you would only need to visit the URL as you normally would and supply a desired date... the browser would automatically locate the archived page from that date and display it to you without the need to manually search through multiple archives.

You can see this in action right now by using the Memento Firefox add-on. Below is a screen shot using the add-on to browse cnn.com as it appeared on July 9, 2009. I actually told the add-on to show the June 21, 2009, version, but the July page (from the European National Archives) is the closest page that was found in any archive. This is not a failing of Memento... it's a limitation of web archiving in general.


Memento uses HTTP content negotiation to add this time dimension to the Web. Instead of discussing the technical details here, I'll instead point you to the Memento Guide Intro if you're interested. Ideally, all web browsers and web servers in the future will support the Memento HTTP headers, and no special add-on will be necessary.

Memento is the brain-child of Michael Nelson (Old Dominion Univ) and Herbert Van de Sompel (LANL). It's made a quite a stir in the past year with a write-up in New Scientist, a paper at the Linked Data on the Web workshop (LDOW2010), and some significant funding from the Library of Congress. Tim Berners-Lee said this about Memento: "This is neat; there is a real need for this."

When Twitter is gone, your Tweets will live on

The Library of Congress announced on Wednesday that

"Every public tweet, ever, since Twitter’s inception in March 2006, will be archived digitally at the Library of Congress."

That's right... every thoughtless, trivial, and crass remark you ever tweeted is now going to be made available for future generations (your tax money at work ).

This is actually a very positive development because this corpus of short messages will provide invaluable to researchers and historians. My guess is some research on this corpus will likely be used to improve web search. I will certainly have my web IR course in the spring do some analysis on the corpus.

My hope is that some day the LoC will also archive all of Facebook. This will prove much more problematic as Facebook data is inherently private, and access to the archive will likely need to be restricted. But losing this treasure chest of bytes would, in my estimation, be far more of a loss to society and future researchers than losing a few tweets.

Facebook: Memorialize the deceased

In a blog post on Monday, Facebook brought attention to a previously existing feature intended to "memorialize" Facebook users who have deceased. You can submit a "Deceased" form (pictured below) that notifies Facebook about "dead" accounts. Once Facebook determines that the account owner is indeed deceased and flips the switch, no one can log into the account anymore, and the person's face no longer appears in friend recommendations or Suggestions. However, you can still post message's on the departed's Wall.



The problem with memorializing the account is that the user's family or friends, if they had the user's password, can no longer access the user's Messages or other personal data. (This could be good or bad thing.) For anyone in this predicament, I highly recommend you archive the deceased's account using ArchiveFacebook before their account is memorialized. Then you will always have a snapshot of the person's Facebook account on your own hard drive.

I'm giving a talk about the ArchiveFacebook Firefox add-on tomorrow afternoon in a Harding University Computing Seminar. If you're in Arkansas Thurs, feel free to stop by at 4:00 pm in Science 113.

P.S. This issue of "what happens to my data now that I'm gone?" is going to continue being more relevant as more of our data is stored in the clouds.

Archive spam

So that's why Hanzo:web has been down...


Spam: A tragedy of the commons. Looks like they are not the only web archiving site to be plagued by spam; Spurl.net has had this notice posted on their website for months which apologizes for offering "reduced functionality due to heavy spam attacks":



Update 7-1-08:

I received an email from Mark Middleton this morning informing me that Hanzo:web's free archiving service has been discontinued. There's really no other free web archiving service to take Hanzo:web's place- WebCite is mainly for academic citations, Furl and Spurl only archive since HTML pages and don't make archived materials publicly available, and Archive-It charges a subscription fee.

What happens when your archive goes down?

I think it's only temporary, but what if you were relying on Hanzo:web to archive your treasured websites, only to have them phase-out their service or go out of business? Think it unlikely?

(Hanzo:web has been around for at least 3 years, but they're still in beta... sounds like they're pulling a Google.)


Update on 7/1/2008:

No, they're down for good.

IWAW 2007

The International Web Archiving Workshop was held at JCDL this year (normally it's held at ECDL). Our research group at ODU was well represented with 3 presentations. I presented our new Warrick queueing system called Brass (slides are available here). This was the second time I attended, the first time being in Vienna 2005.


There were some excellent presentations by a number of people including the Internet Archive, UNC, CDL, and others. One that I found really interesting was about Preserving 2008 Presidential Election Videos (work from VidArch) where they have been archiving YouTube videos using their TubeKit. Overall I really enjoyed the workshop and hope to attend again.

I'll be heading home tomorrow!

Web Curator Tool, standardizing PDF, and orphaned works

Some notable events in the world of digital preservation:

• The National Library of New Zealand and the British Library have collaborated to produce the Web Curator Tool (WCT), a tool that allows non-technical users to archive websites in a simplified manner. It’s essentially a wrapper around the Heritrix web crawler with numerous management functions added on. In a recent article, Philip Beresford from the British Library discusses the history of WCT and shows how it can be used to crawl and archive a website.
  
  
• In an effort to convince the world that the PDF format is ideal for long-term storage, Adobe is submitting it to ISO for standardization. Microsoft has also submitted their Ecma-approved Office Open XML for standardization to ISO, a radical departure from the "secret-sauce" mentality Microsoft has held for years. Governments and other organizations are slowly becoming aware of the problems created by storing their data on closed formats that change over time, and Microsoft and Adobe don’t want to be dropped from their largest customers. By standardizing these formats, interoperability should be much less of an issue in the future.
  
  
• Brewster Kahle, co-founder of the Internet Archive, recently lost a U.S. appeals decision in Kahle v. Gonzales. Kahle, along with several notable companies like Google, MSN, and Yahoo, are trying to get orphaned works (copyrighted work whose owner cannot be reached) into the public domain in order to remove legal barriers that prohibit the scanning and digital distribution of those works. Kahle rightly blames Disney for the mess:
  

  What happened is that some overzealous copyright laws got passed with heavy lobbying from folks like Disney and these are screwing things up... Instead of keeping just Mickey Mouse or just the profitable works under copyright for longer, they fundamentally changed the structure of copyright.

  
  
  

hanzo:web = Internet Archive + Furl

I just discovered a new web archiving service called hanzo:web. It is similar to Furl except that they allow you to archive an entire website, not just a single web page. From the site:

We have observed important and beautiful websites emerge and disappear from the web everyday. We believe that archiving the content of all sites is a social necessity and needs to take place now! To this effect we intend to archive all sites, pages and links that come through Hanzo and allow free access to this collection forever.

This sounds a lot like the Internet Archive’s mission:

The Internet Archive is working to prevent the Internet — a new medium with major historical significance — and other "born-digital" materials from disappearing into the past. Collaborating with institutions including the Library of Congress and the Smithsonian, we are working to preserve a record for generations to come.

It's difficult to say when the Hanzo service came online. I found some archived content going back to Nov 2005. They’ll be talking about it at the O'Reilly’s Emerging Technology conference in March and will be launching a public beta of their API.

The contents of the hanzo:web archive are apparently accessible to web crawlers. I couldn’t find a robotos.txt file, and if you do a Google search for "warrick reconstruct" you'll get my Warrick page first and the archived version from hanzo:web second! (I don’t know who archived my page, but thanks!)

Notice the Internet Archive-looking URL:

http://hanzoweb.com/archive/20060104132758/http://www.cs.odu.edu/~fmccown/research/lazy/warrick.html

At the bottom of the page they insert some JavaScript to redirect links back to Hanzo:

<SCRIPT language="Javascript">
<!--

// FILE ARCHIVED ON 20060104132758 AND RETRIEVED FROM
// HANZO:WEB ON 2006-02-21 18:47:35.004450.
// JAVASCRIPT APPENDED BY HANZO:WEB.
// ALL OTHER CONTENT MAY ALSO BE PROTECTED BY COPYRIGHT
var archiveurl = "http://www.hanzoweb.com/archive/20060104132758/";

function rewriteURL(aCollection, sProp) {
var i = 0;
for(i = 0; i < aCollection.length; i++)
if (aCollection[i][sProp].indexOf("mailto:") == -1 &&
aCollection[i][sProp].indexOf("javascript:") == -1)
aCollection[i][sProp] = archiveurl + aCollection[i][sProp];
}

if (document.links) rewriteURL(document.links, "href");
if (document.images) rewriteURL(document.images, "src");
if (document.embeds) rewriteURL(document.embeds, "src");

if (document.body && document.body.background)
document.body.background = archiveurl + document.body.background;

//-->
</SCRIPT>

This is similar to what Internet Archive does with archived pages.

Hanzo allows pages to be tagged. My Warrick page was tagged with “webarchiving”. Below is a screen shot when accessing my Warrick page from the search interface. This is using frames, so the metadata is shown in the upper frame and the page on the bottom.



Not only did they have this page already archived, they also had archived several other pages from my website. I can’t tell if there is a way to list all pages they have archived I can search for “fmccown” or “www.cs.odu.edu/~fmccown” using their search interface, and all that shows up is my Warrick page. I assume in the next few months they’ll be adding more info about how to find archived pages.