Saving 172 BBC websites with BitTorrent

A recent budget cut at the BBC meant that as many as 200 websites were going to be shut down. However, an individual named Ben Metcalfe crawled 172 of the websites before they were deleted and has made them available via BitTorrent. I don't think Metcalfe really needed to expend the effort since it's likely the Internet Archive has archived the sites or will shortly, but it's nice to see an individual being pro-active in ensuring the sites would not be lost. The loss of the sites is a big downer for some who have contributed significant content to the sites in the past.

Thousands of websites about to bite the dust...

Yahoo announced a month ago that it was pulling the plug on GeoCities, one of the Web's first free web-hosting services. There doesn't appear to be any plan to migrate the thousands (millions?) of websites this will affect to other services. If you don't act by the end of the summer, you're Geocities website will disappear.

That is unless the Internet Archive has grabbed a copy, but they aren't likely to have many pages from each Geocities website archived. I've been conversing with someone who lost a backup of her Geocities website years ago, and IA only had a handful of pages archived. This is likely going to be a recurring story in the years ahead.

My first website was on Geocities. In fact, that's how I first learned how to use HTML in 1997. I'm so embarrased by that first website that I'm keeping the address a secret. I fear the day the Internet Archive's Wayback Machine has full-text search, because someone's going to pull it up and post it on Facebook or something. That's one stream of bites I'm not afraid of losing.

The Web in a box

The Internet Archive and Sun Microsystems have just announced the launching of a new data center that stores IA's entire web archive and serves the Wayback Machine. According to Brewster Kahle:

This 3 Petabyte (3 million gigabyte) datacenter will handle the 500 requests per second as it takes over the full Wayback load.

My visit to the Internet Archive

After my visit to the Googleplex, I drove up to San Francisco to meet Kris Carpenter of the Internet Archive. My dissertation work had a lot to do with Google and Internet Archive, it was exciting to visit both places in the same day.

IA's main headquarters is located in the Presidio, an ex-Army base near the base of the Golden Gate Bridge (in an incredibly beautiful setting). As the photo shows, IA is housed in a few buildings that were originally built in the late 1800s. It's a very quaint setting and appropriate for an organization that is attempting to archive the Internet for future generations. The data center is housed in downtown San Francisco which I didn't see.

Before getting a tour of the facilities, Kris, Gordon, Brad, and I met and discussed some upcoming opportunities for undergraduate CS students to contribute to the Wayback Machine and other projects. I'm hoping some of my students will get involved this summer or next semester. We also talked about doing another URL overlap study like the one I did a year ago. I brought up the subject of archiving Facebook and other social networks, and apparently other researchers from places like Stanford have already approached them about the subject. This is something I'd like to get involved in.

I also met several staffers and ran into Brewster Kahle, the Internet Archive's founder. It was really neat meeting the guy that was responsible for what I think is one of the most worthwhile organizations out there.

Pictured below: Machine from the data center (left), machine which can print out a paper copy of a book in several minutes (center), and guys scanning in a book (right). Other photos are available on Flickr.

Fav5

This one is coming a little early, because I'm heading out to California in the morning. I'm attending a program committee meeting at Stanford University over the weekend. This is my first trip out there, so I'm pretty excited. I'll be touring the Googleplex and the Internet Archive on Monday.

1. Abilene Christian University, one of our sister schools, has decided to give all incoming freshmen next year an iPod. Is this a gimmick or the future of secondary education? Let's hope the move helps boost their enrollment and shrink their $3 million dollar budget deficit.


2. Google Sites has just launched as part of Google Apps. Essentially it allows you to create a web page or website via wiki tools that anyone in your group or organization can modify. I haven't tried it yet, but it looks promising.


3. Matt Cutts always has some interesting items to read: Subscribed links, how search engines should deal with NOINDEX, and Blogger Play.
   


4. Good news for those of us that work in IT: salaries are expected to grow in 2008. Will more money start getting freshmen to major in CS?


5. Java or .NET? According to the Tiobe Programming Community Index, Java is still ranked number 1. But in a recent survey by Info-Tech of 1,900 companies, 12% of enterprises focus exclusively on .NET , and 49% center primarily on .NET. This compares to 3% and 20% of companies that are focused exclusively or primarily on Java.

Internet Archive and censorship

Just yesterday, an anonymous Wikipedia editor using the IP address 211.7.98.52 claimed that the Internet Archive had "censored" pages from around the 9/11 time frame. To prove their point, they provided a link to a snapshot showing the gaps in the Archive. They cited no other external source that discusses the issue.


I'm sure there is a good explanation for the pages not appearing in the Archive... most likely Alexa (who supplies a majority of the archived pages to IA) just didn't crawl those sites on those dates. The Web is a very large place, and Alexa doesn't crawl every website on the same schedule.

I've changed the wording to read "missing pages" instead of "censored", but I suspect 211.7.98.52 will be reverting the edits back sometime today. This anonymous editor obviously thinks there is something sinister going on.


Update on 1/30/2008:

Sure enough, the conspiracy theory will not die. Someone at 61.204.240.194 added further "evidence" that IA has yanked the pages, and 211.7.98.52 apparently agrees.

211.7.98.52 is apparently Victoria Sachs, or at least that's the name given in this Internet Archive forum posting which asks about the missing pages. Brewster Kahle, founder of the IA, responded to her post:

There was nothing special happening before 9/11/2001-- the attempt is to crawl every site ever 2 months. sometimes things would be more frequent, but mostly 2 months.

with 9/11 events, the crawler team tried to archive things much more frequently. the news sites had trouble staying up, so that record is a bit spotty.

I hope this helps.

-brewster

Sorry, Brewster, but that explanation will not appease Victoria and 61.204.240.194. There are other websites that were archived in the weeks before 9/11, so it must be a huge conspiracy to keep the world in the dark. Victoria also wrote:

Yet the Internet Archive's records clearly show the major sites listed above have never before experienced such an enormous 'missing cache' gap in the history of the Internet Archive.

Of course this is not true. Here's one counter example where Time.com is missing all of July 2000, a 50+ day gap. I could give number other counter examples, but alas I have students to teach. (BTW, Victoria, the IA doesn't cache pages, they archive them. There is a difference. )

Victoria's Wikipedia article edit concludes:

Unfortunately, future generations will never be able to read what Newsweek.com, Reuters.com, Altnet.org, ABC.com, Time.com, MSNBC.com, ABCnews.com, Nasdaq.com, Bloomberg.com, LAtimes.com, Timesofindia.com, CNN.com, UAL.com, CBSnews.com, and NYtimes.com, published during the weeks preceding September 11th, 2001.

With this I do agree... it is too bad IA doesn't have those pages archived. But I seriously doubt they purposely removed those pages from their Archive; it completely runs contrary to what they are attempting to do.

I am visiting with some folks at IA in San Francisco next month. I'll bring up this matter with them and report back what they say.

Update on 2/12/08:

I think the controversy has finally subsided.

On Feb 10, a user from 211.7.100.61 (Victoria?) removed their posting and signed off with

OK, censorship wins.

The Talk page basically outlines the rambling (and unsigned) thoughts of 211.7.100.61 and how other editors tried to reason with him/her. It makes for some interesting reading. Even Mr. Wikipedia got into the action:

This guy has been emailing me as well. It is an unsubstantiated crackpot conspiracy theory that doesn't pass the basic sniff test. Even the links he sends to show he is right, thoroughly refute his claims. (As said above, the gaps, while regrettable, are normal for that era of the archive.)--Jimbo Wales (talk) 17:39, 6 February 2008 (UTC)

I think that about sums it up.

Fav5

TGIF! My pick of the week's top 5 events, articles, or items of interest:

1. The Internet Archive just increased their web archive (the Wayback Machine) by 25%. Brad Tofel, an archivist at IA, told me that from here on out the Wayback Machine should be 3 months out-of-date rather than 6-12 months like they have been in the past.


2. It's a great time to be a talented college student looking for an internship.


3. Some brilliant blogging advice from one of the most prolific bloggers at Google:
   

   Don't post when you're angry.

   These and other wise sayings from Matt Cutts were in response to a Google employee's blogging about her dislike of the new movie Sicko. I'll add to this...
   

   Don't post a response to someone else's blog posting when you're angry.

   I've learned this lessons the hard way.


4. Dell is warning their customers about moving from XP to Vista. I recently made the move a few weeks ago when I purchased a new desktop for home use. It hasn't exactly been the smoothest transition, but I'll blog more about that later. I would certainly advise anyone making the leap to be on their guard.


5. If you haven't played the Wii yet, find a friend who owes one and give it a shot! On the 4th our friends Brad and Katie threw my family a going away party (we're heading back to Arkansas in Aug). Brad owns a Wii, and it must have been the most popular activity at the party! No wonder the Wii is currently out-selling the PS3 six-to-one.
   

JCDL 2007 - day 1

I'm in Vancouver at JCDL 2007. I flew in yesterday with Martin, and the rest of the ODU posse flew in today. Between the 3 of us Ph.D. students, we'll be making 6 presentations at JCDL and IWAW this week.

This is my first trip to Vancouver, and I must say it's one of the most beautiful cities I've been to. Just wish I had Becky and the Bean here with me to enjoy the beauty. The photo on the right shows what the view looks like from my room at the Westin Bayshore hotel.

This morning I joined Kris Carpenter and Brad Tofel of the Internet Archive in a tutorial about researching the Internet Archive. I shared some of my research with Warrick and a recent study analyzing IA overlap with search engine caches. (You can see my slides here.)

I learned a lot about IA from Kris and Brad. Here are just a few items of note:

• The IA currently has archived around 96 billion resources (html, pdf, images, etc.), or about 1.9 petabytes of data, 51% of which is unique.
  
  
• Although the Archive's holdings are 6-12 months out-of-date, beginning July 1, the Archive will receive updates on the first of each month and will only be 2 months out-of-date.
  
  
• The Archive is currently working on adding full-text search to its contents from 1996-2000 in a project called 20th Century Find. (No URL yet.)

Tomorrow morning I'll be making a presentation about a Warrick experiment, and Joan will be presenting a short paper after me. Looks like I'll be burning the midnight oil getting prepared for tomorrow...

Web Curator Tool, standardizing PDF, and orphaned works

Some notable events in the world of digital preservation:

• The National Library of New Zealand and the British Library have collaborated to produce the Web Curator Tool (WCT), a tool that allows non-technical users to archive websites in a simplified manner. It’s essentially a wrapper around the Heritrix web crawler with numerous management functions added on. In a recent article, Philip Beresford from the British Library discusses the history of WCT and shows how it can be used to crawl and archive a website.
  
  
• In an effort to convince the world that the PDF format is ideal for long-term storage, Adobe is submitting it to ISO for standardization. Microsoft has also submitted their Ecma-approved Office Open XML for standardization to ISO, a radical departure from the "secret-sauce" mentality Microsoft has held for years. Governments and other organizations are slowly becoming aware of the problems created by storing their data on closed formats that change over time, and Microsoft and Adobe don’t want to be dropped from their largest customers. By standardizing these formats, interoperability should be much less of an issue in the future.
  
  
• Brewster Kahle, co-founder of the Internet Archive, recently lost a U.S. appeals decision in Kahle v. Gonzales. Kahle, along with several notable companies like Google, MSN, and Yahoo, are trying to get orphaned works (copyrighted work whose owner cannot be reached) into the public domain in order to remove legal barriers that prohibit the scanning and digital distribution of those works. Kahle rightly blames Disney for the mess:
  

  What happened is that some overzealous copyright laws got passed with heavy lobbying from folks like Disney and these are screwing things up... Instead of keeping just Mickey Mouse or just the profitable works under copyright for longer, they fundamentally changed the structure of copyright.

  
  
  

Google Archive?

In September Google apparently registered for multiple domain names that implied a Google Archive (or what I call Internet Archive Part Deux) is in the works. Garett Rogers of ZDNet was the first to break the story. It wouldn't be surprising if Google decided to quit throwing away their cached copies of the Web and allow users to search the Web through time. This functionality is something the folks at the Internet Archive have been working on for quite some time. Personally I'm glad they don't provide an archive search- I would be very embarrassed if people could see the first website I created back in 1997. Perhaps with Google's deep pockets we'll see a searchable (and more up-to-date) Internet archive before the year is out.

Ghostsites

Steve Baldwin has a really nice blog called Ghostsites which is dedicated to really old web pages and sites. A few entries that caught my eye:

• Is This the World’s Oldest Active Web Page?
• This Site Proudly Optimized for Netscape 2.0!
• The Museum of E-Failure

Baldwin also pointed out John Maeda’s (MIT) side-by-side comparison of Yahoo and Google’s root pages from 1996-2005 as obtained from the Internet Archive. The comparison points out the simplicity of the Google page when compared to the complex Yahoo page.

hanzo:web = Internet Archive + Furl

I just discovered a new web archiving service called hanzo:web. It is similar to Furl except that they allow you to archive an entire website, not just a single web page. From the site:

We have observed important and beautiful websites emerge and disappear from the web everyday. We believe that archiving the content of all sites is a social necessity and needs to take place now! To this effect we intend to archive all sites, pages and links that come through Hanzo and allow free access to this collection forever.

This sounds a lot like the Internet Archive’s mission:

The Internet Archive is working to prevent the Internet — a new medium with major historical significance — and other "born-digital" materials from disappearing into the past. Collaborating with institutions including the Library of Congress and the Smithsonian, we are working to preserve a record for generations to come.

It's difficult to say when the Hanzo service came online. I found some archived content going back to Nov 2005. They’ll be talking about it at the O'Reilly’s Emerging Technology conference in March and will be launching a public beta of their API.

The contents of the hanzo:web archive are apparently accessible to web crawlers. I couldn’t find a robotos.txt file, and if you do a Google search for "warrick reconstruct" you'll get my Warrick page first and the archived version from hanzo:web second! (I don’t know who archived my page, but thanks!)

Notice the Internet Archive-looking URL:

http://hanzoweb.com/archive/20060104132758/http://www.cs.odu.edu/~fmccown/research/lazy/warrick.html

At the bottom of the page they insert some JavaScript to redirect links back to Hanzo:

<SCRIPT language="Javascript">
<!--

// FILE ARCHIVED ON 20060104132758 AND RETRIEVED FROM
// HANZO:WEB ON 2006-02-21 18:47:35.004450.
// JAVASCRIPT APPENDED BY HANZO:WEB.
// ALL OTHER CONTENT MAY ALSO BE PROTECTED BY COPYRIGHT
var archiveurl = "http://www.hanzoweb.com/archive/20060104132758/";

function rewriteURL(aCollection, sProp) {
var i = 0;
for(i = 0; i < aCollection.length; i++)
if (aCollection[i][sProp].indexOf("mailto:") == -1 &&
aCollection[i][sProp].indexOf("javascript:") == -1)
aCollection[i][sProp] = archiveurl + aCollection[i][sProp];
}

if (document.links) rewriteURL(document.links, "href");
if (document.images) rewriteURL(document.images, "src");
if (document.embeds) rewriteURL(document.embeds, "src");

if (document.body && document.body.background)
document.body.background = archiveurl + document.body.background;

//-->
</SCRIPT>

This is similar to what Internet Archive does with archived pages.

Hanzo allows pages to be tagged. My Warrick page was tagged with “webarchiving”. Below is a screen shot when accessing my Warrick page from the search interface. This is using frames, so the metadata is shown in the upper frame and the page on the bottom.



Not only did they have this page already archived, they also had archived several other pages from my website. I can’t tell if there is a way to list all pages they have archived I can search for “fmccown” or “www.cs.odu.edu/~fmccown” using their search interface, and all that shows up is my Warrick page. I assume in the next few months they’ll be adding more info about how to find archived pages.

arcget is a little too late

Gordon Mohr from the Internet Archive told me about a program called arcget that essentially does the same thing as Warrick but only works with the Internet Archive. Aaron Swartz apparently wrote it during his Christmas break last Dec. It’s too bad he didn’t know about Warrick. That seems to be the problem in general with creating a new piece of software. How do you know if it already exists so you don't waste your time duplicating someone else's efforts? All you can do is search the Web with some carefully chosen words and see what pops up.