End of the Google 502 errors?

Google users have sporadically seen Google 502 (bad gateway) errors the last several years. The errors appear momentarily and then disappear. I’ve linked to a few postings about it according to date:

Mar 2003
July 2003
June 2005
Sept 2005
Nov 2005
Feb 2006
May 2006

Google API users have seen the 502 errors much more frequently:

Nov 2005, and another
Dec 2005
Jan 2006
Feb 2006, and another
May 2006

From my investigations, it looks like Nov 2005 is when the problems began. I have personally dealt with the problem ever since Mar 2006 when I integrated the Google API into Warrick. I had to add some logic to sleep for 15-20 seconds when encountering the error and then re-try.

In late May I started a new experiment which uses the Google API, and I’ve been monitoring it daily to see how many 502 errors I was receiving. From late May to June 6, I consistently received a 502 error for about 30% of my requests. On June 7, the number of 502s went down to zero. I have only received an occasional 502 out of hundreds of requests made daily.

Someone at Google finally got sick of the bad press and made some changes, and I’m thankful for it. :)

JCDL 2006 - day 3

I’m back from JCDL. Overall I really enjoyed the conference, and I’m really hoping to attend next year’s conference in Vancouver, British Columbia. Mon and Tues were packed with activities, but everything was wrapped up on Wed morning. A few highlights from the remainder of the conference:

• The dinner Tues night at the Ferrington Village was fantastic- I haven’t eaten that much pork in a long time. Everyone seemed to have a good time conversing. Johan took the top poster award, and Lagoze et al. took the top paper award.
  


• Wed morning I enjoyed Johan’s presentation on aggregating and analyzing scholarly usage data although I had seen the presentation before at ODU. Some of the other presentations were well done, but I admittedly zoned-out a lot of it and went through my emails. I could tell many others around me were doing the same, just giving partial attention to the presentations. That’s got to be annoying to the presenters.
  


• After the conference I was hoping to do a quick run over to see Duke’s campus, but it was pouring down rain. So Michael, Martin, and I headed back to Norfolk right after lunch. It was nice getting back and seeing Becky. Although the Carolina Inn was fantastic, there’s really no substitute for home.

Technorati Tag: jcdl2006

JCDL 2006 - day 1 and 2

I’m at JCDL 2006, hosted at UNC in Chapel Hill, NC. Michael brought Joan, Martin, and me down for the conference, and so far I’ve really enjoyed it. UNC has a really nice campus, and the Carolina Inn is deluxe. ;) Joan and I presented our dissertation abstracts at the Doctoral Consortium on Sunday, and I was able to get a few helpful suggestions. Ray Larson and the other faculty treated us to a fantastic dinner at Top of the Hill that evening. Since Michael is a co-chair of JCDL, we were unable to submit a paper, but that just means I can relax and enjoy the conference.

Here are a few highlights so far:

• In the opening talk Monday morning, Daniel Clancy, Engineering Director of the Google Book Search, talked about Google’s efforts to digitize and index books from the G5, the five libraries that are cooperating with the digitization process. It was a very informative talk, and I certainly applaud Google for taking on such a massive and important project.


• Andrew McCallum presented a paper about leveraging topic analysis and introduced rexa.info, a website like Google Scholar that displays published papers. The cool thing is how they also show co-authorship, authors that you site, and authors that cite you. They just had 2 of my papers indexed, but I guess that isn’t bad for a research project.
  


• Carl Lagoze presented a paper that honestly addressed some of the shortcomings of the “low barrier” implementation of the NSDL. Turns out the implementation is rather people-intensive: problems include content providers unwilling to prove quality metadata and improperly implementing OAI-PMH. There was one notable absence from the references. At least one of the audience members publicly admitted being depressed at the current situation. I also do wonder about the future of a digital library that can’t scale without an enormous amount of people-intensive work. How do you build a DL that in many ways is competing with Google?
  


• Johan gave a very in-your-face poster presentation: “Have any of you wondered about your funky JCDL reviews from last year?” Johan’s poster showed how the reviewers from last year’s JCDL were not reviewing papers based solely on their expertise. So why were non-experts judging papers that weren’t in their domain?
  


• Bill Arms introduced me to Andreas Paepcke, a researcher at Stanford who works with WebBase/WebVac. Looks like they are making all their crawls available to other researchers who want them, but they won’t work for my website reconstruction research since it depends on real-time search engine content.
  


• I talked some with Alesia Zuccala who presented her work with LexiURL, a piece of software written by Mike Thelwall. LexiURL uses the Yahoo API to report backlinks for a set of URLs. I really enjoy reading Thelwall's papers and hope to meet him at some point.
  


• This morning Jonathan Zittrain gave a very entertaining and informative presentation about redaction, restriction, and removal of open information. It was one of the best presentations that I’ve seen, and his PowerPoint presentation was a fantastic example of how to put together a presentation. Even Tufte would have approved. Once of the most memorable slides showed the accidental grouping of two books on Amazon.com: a children’s book with “American Jihad”.

Tonight we’re being bussed out to Fearrington Village, home of the “oreo cows” for a pig pickin’. Yum.

Technorati Tag: jcdl2006

Yahoo - Error 999

Yesterday I finally received the coveted “Error 999” page from Yahoo:

Sorry, Unable to process request at this time -- error 999.
Unfortunately we are unable to process your request at this time. This error is usually temporary. Please try again later.
If you continue to experience this error, it may be caused by one of the following:

1. You may want to scan your system for spyware and viruses, as they may interfere with your ability to connect to Yahoo!. For detailed information on spyware and virus protection, please visit the Yahoo! Security Center.


2. This problem may be due to unusual network activity coming from your Internet Service Provider. We recommend that you report this problem to them.

While this error is usually temporary, if it continues and the above solutions don't resolve your problem, please let us know.

Just like Google, Yahoo appears to also be monitoring for high volume traffic/automated requests and denying access for a period of time from infringing IP addresses.

I have a couple of scripts that make 300 queries per day to Yahoo using their web interface. 126 of my queries received the error yesterday, and 125 today. The scripts ran for 11 days before being detected. You’d think 300 queries wouldn’t be enough to trigger the response! I’m also making the 300 same queries using the API to see what the difference is in their responses.

I’ve seen others complain of the 999 error dating back to April 2004, but this is the first time I have personally experienced it. Murray Moffatt shares his experience with the error and some possible fixes. Basically all you can do if you are running a script and encounter the page is to sleep for several minutes and try again.

Update: 6/9/06

Today I increased the wait time between each query to a random number of seconds between 3-8. I also ran the script at 8:00 am EST instead of 2:00 am EST to see if blending into the croud helped at all. Today I received 133 error 999s. Not good. Possibly I'm being punished because I'm making requests at a high volume time. Next: increase the wait time to 15-20 seconds between each query.

Windows Live Academic Search

Microsoft launched Windows Live Academic Search (what I call Live Academic for short), a competitor for Google Scholar, a couple of months age (Apr 11, 2006 to be precise). According to their FAQ, they are harvesting material from open archives (like arXiv) using OAI-PMH. This is a different strategy than Google’s; Google is mainly indexing papers found on the Web.A rather detailed article by Barbara Quint about Live Academic which discusses how Microsoft learned from Google’s experiences and how Google is not feeling threatened by this newest entry in the search webosphere. Quint was impressed by the “very polished look” of Live Academic. I gave it a try, and here’s what I have to say about it:

Things I liked:

• Display of the abstract and other metadata for the article on the right side of the screen.


• The ability to click on an author’s name to search for other works by that author.


• Support for BibTeX and EndNote.


• The ability to sort by author, date, journal, and conference.

Things I did not like:

• The attempt to produce a “snazzy” interface (using Ajax) which has a scroll bar that jumps around from time to time with no apparent explanation. Also if I used IE, it was almost impossible to highlight and copy text. Surprisingly Firefox on Windows had no such problem.


• No advanced search. You can’t limit the search results to just computer science or search by author, journal, title, etc.


• When using IA, the Back button on the browser frequently does not return to the previous page. Firefox on Windows sometimes also exhibited this behavior.


• Intermittent problems with searching. For example, searching for "mod_oai" results in nothing being found. But it I search for “apache module for metadata harvesting” the paper with “mod_oai” appears in the title. But if I search for “apache module for”. (Correction: these problems appear to have been fixed overnight.)


• Searching for authors with a middle initial can be problematic. A search for "michael l. nelson" (with quotes) seems to accurately locate many of Michael's publications. But if you click on "Michael L. Nelson" in one of the results, a search is made for authors matching "Nelson, M" which produces many false-positives.
  


• I could not find a single one of my publications even though several of them are in arXiv. (Correction: this morning several of them now appear to have been indexed including my thesis.)
  

Overall I'd say stick to Google Scholar for now. But as Microsoft appears to be making some major improvements (literally overnight), my list of “didn’t likes” are bound to get much shorter.

Graphs in R

The last week or so I’ve been trying to learn the R programming language. The language was named R after the authors’ names, a really poor choice since it makes it almost impossible to search the Web for R-related web pages. One of my colleagues once stated that the R stands for “razor” as in what you feel like using on your wrists when trying to learn R! I have to agree- the intro material they supply is ok for learning a few basics, but I have yet to come across anything that shows all the basics of producing simple line and bar graphs. And I’m amazed by the poor examples in the user-contributed documents that give a little code with no pictures or pictures so small you have to magnify the image x10 to see anything. Therefore I have created my own intro to producing simple graphs in R. It’s by no means complete, but it’s much better than anything I’ve found.

Getting external backlinks from Google, Yahoo, and MSN

It’s often useful to know how many external backlinks are pointing to a particular URL. This metric can be used to partly determine a page’s popularity on the Web. A good tool for automating this process is the Backlink Analyzer which uses the Google, MSN, and Yahoo APIs using the "link:" command. The software allows a user to specify sites to ignore in the backlink counts, a useful function since the link: command returns backlinks from external and internal links for all three search engines.

I don’t know for sure if Backlink Analyzer is doing this or not, but for Yahoo and MSN, it is possible to perform a single command to give only external backlinks using a combination of link: and -site: parameters. For example, the following query will show all the pages pointing to my Warrick page for Yahoo and MSN:

link:http://www.cs.odu.edu/~fmccown/research/lazy/warrick.html -site:www.cs.odu.edu

Google will not handle the –site: parameter successfully in this query although it does handle it in other types of queries.

Sacred Marriage

This weekend Becky took a trip out to Memphis to visit her family, and Sara came down to visit from DC, bringing along a friend. We hit the beach and did some camping, i.e., I got absolutely no work done. It was really nice to spend some time with my sister (although I can't wait to see my wife!).

What I did have time to do was read about half of Gary Thomas’ Sacred Marriage. Becky had read it recently and recommended I also read it. So far I have to say it’s a really great book for someone who is considering marriage or has been married for some time. Instead of looking at marriage as a “what can I get out of it?” point-of-view, Thomas tells his readers that marriage is an opportunity to improve who we are and love the way God loves us. Rather than make us happy, marriage is designed by God to make us more holy.

Here are a couple of passages I underlined this weekend:

Everything I am to say and do in my life is to be supportive of this gospel ministry of reconciliation, and that commitment begins by displaying reconciliation in my personal relationships, especially in my marriage.

Christians can command attention simply by staying married.

We can never love somebody “too much.” Our problem is that typically we love God too little.

As Betsy and Gary Riucci point out, “Honor isn’t passive; it’s active. We honor our wives by demonstrating our esteem and respect: complimenting them in public, affirming their gifts, abilities, and accomplishments; and declaring our appreciation for all they do. Honor not expressed is not honor.”

It is guaranteed that your spouse will sin against you, disappoint you, and have physical limitations that will frustrate and sadden you… This is a fallen world… You will never find a spouse who is not affected in some way by the reality of the fall.

I wouldn’t be surprised if many marriages end in divorce largely because one or both partners are running from their own revealed weaknesses as much as they are running from something they can’t tolerate in their spouse.

I seriously recommend this book to men and women who want to see more clearly what their marriage is all about. If we took these lessons to heart, it would make a huge difference in the lives of many.

OA debate - Eysenbach and Harnad

I’ve been following a rather lively debate on the American Scientist Open Access Forum between Gunther Eysenbach (a professor at the University of Toronto and editor-in-chief of JMIR), and Stevan Harnad (a professor at the University of Southampton and Open Archives "archivangelist"). Eysenbach published an article that showed the citation benefits of OA publishing: OA articles (articles which are freely accessible to the public) in Proceedings of the National Academy of Sciences (PNAS) were more than twice as likely to be cited one year later than non-OA articles (articles that must be paid for to access) published in PNAS.

Although Eysenbach and Harnad are both OA proponents, what appears to have stirred up the trouble was that Eysenbach’s article criticized several of the studies that Harnad was involved in (and failed to point to two recent studies), pointing out that they lacked a certain amount of statistical rigor and had some inherent fallacies. Eysenbach gives a detailed account on his website about the methodology of his paper which used multivariate analysis to account for known confounders (variables which are strongly associated with the outcome of interest) like the number of co-authors of a paper. Eysenbach argues that if a paper has multiple authors, it is more likely to be self-archived (green OA- see below). This is intuitively true (my paper on search engine coverage of the OAI-PMH corpus was self-archived by Xiaoming before I even gave it a second thought). But a paper is also more likely to be cited if it has more authors since each author is vested in citing their work. It’s also possible papers with multiple authors are of higher caliber (and hence will get cited more often) since there were more heads looking at the problem. Other factors like this one definitely need to be considered when trying to determine if OA is causing the increase in citations or not.

A big part of the argument stems around what is OA. There are two different flavors:
1. green OA - articles (including dissertations and preprints) are published in closed-access journals but are self-archived in an OA repository/archive or personal website. Green journals explicitly allow authors to self-archive their work.
2. gold OA – articles are published in OA journals where they are immediately accessible to the public for free. A gold journal may make all articles freely accessible or make only certain articles freely accessible by charging a fee to the author (which is usually paid by the author's institution or research foundation).

Although green OA is currently the most popular form of OA (5% gold, 90% green), it is sometimes difficult to test for since it’s possible an author will make their article publicly accessible the day it is accepted for publication or months after its been published. Gold OA is easier to test since the status is determined the first day it is published. Eysenbach tested for gold vs. green to see if papers that were self-archived but had closed access were any more likely to be cited than articles that were gold OA (it’s not clear how he discovered if a paper was self-archived; maybe he searched Google or maybe there was a way for an author to indicate if the paper was self-archived). He found that “self-archiving OA status did not remain a significant predictor for being cited.” This point appears to have also really bothered Harnad about the study.

I’ve learned a lot about OA from this debate. I just wish there was a little less animosity (zealousness?) from both sides. It’s a he-said/I-didn’t-say exchange which is now well documented on a public email forum which is archived on the Web, a blog, and in a letter to the editor: a prime example of how scientists air their differences today.

By the way, I just came across a really cool slide illustrating the access-impact problem between the Harvards and the have-nots (nice pun!) is on page 4 of Leslie Chin’s slides.

Google limiting researchers to 1000 queries

I recently read a poster from ISSI 2005 entitled “Google Web APIs - an Instrument for Webometric Analyses?” The poster was written by Philipp Mayr and Fabio Tosques to introduce the Google API to webometric researchers. They ran several experiments to demonstrate that the API was useful. One experiment queried Google’s web interface and API with the term “webometrics” over 240 days. Their results showed a huge difference between the web interface and the API which made me wonder how you can consider an API useful if it gives you far different responses from what the rest of the world is seeing.

In their conclusion, Mayr and Tosques reported a limit of 10,000 requests per day. Google only allows 1000, so I emailed Mayr to see why they reported 10,000. He replied that Google would give researchers more queries, but when I emailed api-support@google.com requesting a bump up, they replied with this:

Due to overwhelming demand, we are no longer accepting requests for additional queries or for commercial use permission.

So researchers are in a quandary: use Google’s public web interface to perform searches which frequently (in my experience) leads to being blacklisted for hours at a time (even when less than 1000 daily queries are being made), or use the buggy (502 errors are common) API with only 1000 daily query limit which returns very different results than those obtained through the web interface.

Inspired by this dilemma, I have decided to put the APIs from Google, MSN, and Yahoo to the test. I am running a series of experiments comparing what the APIs return to what the web interfaces return. I’m hoping this will result in something that will give researchers a little more information on how to go about using search engines in their experiments and what to expect when using the APIs. Now if I can just find a free server that I can use to make requests for a few months…

MSN malware error screen

Looks like MSN is being targeted by hackers. I got the message below when I tried searching MSN for link:http://forums.absoft.com/viewtopic.php?pid=1932

We are seeing an increased volume of traffic by some malware software. In order to protect our customers from damage from that malware, we are blocking your query. A few legitimate queries may get flagged, and for that we apologize. Please be assured that we are hard at work on this problem and hope to get it resolved even better as soon as possible.
If you are using phpBB, please check out the phpBB downloads site http://www.phpbb.com/downloads.php and make sure you are not vulnerable.
- MSN Search Team

I did a search on Google to find out more, and apparently this has been seen by others:

Jan 2006:
http://www.emailbattles.com/archive/battles/vuln_aacgfbgdcb_jd/
http://forums.digitalpoint.com/showthread.php?t=47620
http://www.webmasterworld.com/forum97/716-3-10.htm

Feb 2006:
http://forum.abestweb.com/showthread.php?t=69268

May 2006:
http://www.webproworld.com/viewtopic.php?t=63478

I reported about this problem previously with Google. Hopefully MSN is not going to get as aggressive as Google about denying service to automated queries.

Server encoding caching experiment

To determine if my server-side component encodings could be inserted into indexable/cacheable HTML files, I ran a little experiment. I created 3 HTML files that contained encoded chunks in HTML comments at the base of each file:

html_encoded1.html - 2 KB
html_encoded2.html - 45 KB
html_encoded3.html - 99 KB

If you view the source of the pages, you’ll see something like this at the end:

<!-- BEGIN_FILERECOVERY
chunks = 4
filename = xor.o
recover = 2
orig_size = 1105
block_size = 554
block_num = 3

fY/xaGQn0V5MOOpLnM1WIsIUMirrVBQ2XNhidvc5yjL9tEyKTmNjNPjcrJzcPWvs INxxHl1Gt5lKQAYoNi1DXOhFI5ExBm15Nxx1T/hFCwVvsyaHsQQdd3lcqWJl+WTw BTlkiI8yWcPPoy38dqgTVnc4aSNd+0YQWW0bDl67/6XTnych3rSXn5YEYhVMU2eS LCR/0N4pAhKgeMb7SXtdJNQ6WykqDXYJAjtTOIrT2CLaPNRdKbU/ydsvUSDenSt+

Etc…
END_FILERECOVERY -->

I placed these files in my public_html folder on April 19, and linked to them from my index.html page. Today I checked Google, MSN, Yahoo, and Ask to see if any of them were cached. Here’s the results:

Google – cached all three
MSN – cached 1 and 2
Yahoo – indexed 2 only (not available in their cache)
Ask – nada

To see if Google can handle any more, I have created 4 new files of 150, 200, 250, and 300 KB. Looks like 99 KB is too large for MSN. Yahoo’s cache is really inconsistent- maybe 2 is in there, maybe it’s not. Why didn’t they grab 1?

I’ll check back in a couple of weeks and see if anything else has been cached.

Update: 6/20/06

Google and MSN have cached all files that range up to 300 KB. Yahoo has only indexed the first 3 (none are cached), and Ask has nothing.

Now I'm going to create a 400 KB, 500 KB, and 1 MB file and see what happens.

Update: 2/21/07

The cache limits for the search engines appear to be the following: Google - 977 KB, Yahoo - 214 KB, and MSN - 1 MB. I still cannot tell for sure what Ask's limit is, but I ran an experiment where I found 984 KB cached for a document that was 1.6 MB. Google's limit has been confirmed by others.

Yahoo Site Explorer

I just discovered Yahoo’s Site Explorer which was apparently released in September 2005. The tool allows you to see which pages of a site are currently indexed by Yahoo and the inlinks to a particular page. For example, I can see that Yahoo currently has around 1400 URLs indexed from my ODU website, and there are 19 inlinks pointing to the Warrick page. There is an API for accessing the service so page scraping is unnecessary. Now if only we can get Google to provide a similar service!

shiri-maimon.org is hacked

This weekend I was contacted by a fan of Shiri Maimon, an Israeli pop singer, who wanted to reconstruct shiri-maimon.org. This site was hacked recently, and all the files were deleted. The webmaster placed this explanation on the website:

April 20, 2006

This website has been hacked by someone. They have deleted everything, and I have decided that the website will NOT be back online. The reason for this is, that the people who did this, they hacked in just to delete everything. Which means, if I got everything back up and running - they could delete it the day after again. And I don't want to waste my time on that. Besides, the most important stuff, such as the forum, gallery and news is lost - and can't be restored. My host refuse to help me - even though they have the back-up files. So tomorrow I will cancel the domain. They say that they have the back-up files, but they can only re-upload everything if the files were lost during a server crash :-s They're practically writing to me, as if I deleted everything myself. They don't seem to get, that someone freakin' hacked the site! Like I would delete everything myself anyway :-s

You must all know by now, that I have spent endless hours - even weeks and months on this website. I'm very sorry to end the website I loved the most this way. It honestly breaks my heart. I feel really bad for both Shiri and the fans. I only tried to show my appreciation and wanted to spread the word about her. Apparently someone couldn't take that, and decided to ruin it for all of us. And they call themselves fans. Hah! Thanks a lot, whoever you are. I would like to thank all of you who kept visiting and coming back. It really meant a lot to me. Keep supporting Shiri out there ~ don't let the silence remain!

~ Camilla

I’m really surprised the hosting company would not recover the files for her. I’d let everyone know of my disappointment with the company. Looks like many of the pages are still in Google’s cache. I am glad Warrick will help get the site back.

It's becoming very apparent to me that third-party reconstruction is one of the primary things Warrick is useful for. If you don't personally own a backup, it's the only way you are going to get a site back.

iclnet.org is back up!

I just noticed that ICLnet is back up. This is the first website that I recovered with Warrick that used the recovered resources to go back on-line. The WWW 2006 conference website www2006.org was the first truly "lost" website that I reconstructed, but I don't know if they actually used the recovered files to bring the website back on-line. I noted a few weeks ago there was another website that was recovered using Warrick, but I didn't assist with that recovery. There still may be someone else that has used Warrick without my knowledge to recover their site. The Warrick download log shows quite a few downloads over the past few months.

WebCite

This week I was reading about link rot (Wikipedia) when I stumbled across a tool called WebCite for creating archives of web pages that are cited in an academic work. From their website:

WebCite® is an archiving system for webreferences (cited webpages and websites), which can be used by authors, editors, and publishers of scholarly papers and books, to ensure that cited webmaterial will remain available to readers in the future.

The canonical reference for WebCite appears to be a 2005 article in the Journal of Medical Internet Research (JMIR) entitled “Going, Going, Still There: Using the WebCite Service to Permanently Archive Cited Web Pages” by Eysenbach and Trudel. I also found a 2003 poster about the service, so it appears to have been around for a while.

WebCite is a great idea for combating link rot, although other archiving services like Spurl.net and Hanzo:web could also be used. The advantage to WebCite is that they also provide "impact statistics" on cited web pages.

I did a search for “WebCite” in Google Scholar to see if WebCite had been widely adopted since I had never seen it used before. The only articles I could find that used the system were from JMIR which I assume has a policy that enforces use of WebCite for all their articles. Here’s an example of a WebCite URL:

http://www.webcitation.org/426


I may use WebCite the next time I write an article. The only thing I’m concerned about is the long-term survival of WebCite. For several days I was unable to access their website. If their service is not entirely stable, it makes me wonder how long they’ll be around.

Candidacy Exam is over

Last Friday (Apr 7) I completed my Candidacy Exam (proposal defense) without any difficulties. It was nice having my wife in attendance along with a few other friends/PhD students. Now I’m ABD there’s nothing keeping me from graduating except that measly dissertation. A few days before the exam I had my hopes dashed when one of my papers was rejected from a conference. I won’t go into another rant but will keep shopping the paper around until it finds a good home. Gotta keep positive: maybe Google will unexpectedly stumble across my work like they did with Ori Allen and offer me millions for Warrick. ;) In the meantime, Becky and I are heading to the OBX for some R&R.

Bill Arms, who served on my committee, gave a really great talk after my exam about the Cornell Web Library. He published an article about it in D-Lib Magazine (same issue as our paper on crawler activity) and has a more technical paper about it accepted to JCDL 06. The library is based on the collections from the Internet Archive, and it will give researchers the ability to perform Web research much easier than it is today. We may be able to use the library to perform some work with Warrick since it contains a number of lost websites.

URL Canonicalization

The term URL canonicalization (also frequently called URL normalization) refers to the process that is performed on a URL to make it easier to tell if two syntactically different URLs are the same. For example, the URL

http://www.Harding.edu/USER/dsteil/www/abc/../index.htm

could be normalized to produce the canonical URL:

http://www.harding.edu/user/dsteil/www/

Search engines typically use different URL canonicalization policies which makes it difficult for Warrick to tell if URL x from MSN is the same as URL y from Google. I’ve noted some peculiarities in my blog here, here and here. Matt Cutts at Google also discussed some of their canonicalization policies back in Jan 2006.

I have not found much work in the literature about URL canonicalization/normalization. RFC 3986 has some standard normalization procedures that should be done. Pant et al. (2004) has a section about it in their chapter Crawling the Web from the book Web Dynamics. The first paper I’ve seen that deals with the issue head-on is by Sang Ho Lee et al. (2005) "On URL normalization".

I also checked Wikipedia and didn’t find anything about URL canonicalization. I decided to create a page about it and added a reference to it from the web crawler page. That was the first page I ever created on Wikipedia. Proverbs 25:2 – “It is the glory of God to conceal a thing; but the glory of kings is to search out a matter.” I’m no king, but I think God actually delights in our effort to learn about the great world He has created, and I appreciate Wikipedia providing a unique resource for us to consolidate and share our learning.

Warrick reconstructs JaysRomanHistory.com

On Mar 31, I received an email from an individual who had successfully used Warrick to reconstruct JaysRomanHistory.com. The cool thing was that he was able to reconstruct the site without getting any help from me. I have reconstructed a couple of websites on behalf of others, but this is the first third site I am aware of where someone ran Warrick on their own to reconstruct a website.

A couple of quotes from their site:

Welcome! This website has been put back on the Internet by friends of Jay King, the original author, who died unexpectedly in 2005. We didn't want Jay's excellent reference site to be lost forever because it no longer had a home on the Internet at SJSU.

and

This site has been selected as a valuable educational Internet resource for Discovery Channel School.

Update on 4/30/06:

This week I received an email from a Carter R., a webmaster who had used Warrick back in Jan 2006 to reconstruct two of his sites when the hard drive of his personally-maintained web server crashed:

http://dckickball.org/
http://cubanlinks.org/

He writes about using Warrick in his blog entries:

http://cubanlinks.org/blog/articles/2006/01/17/im-back-sort-of
http://cubanlinks.org/blog/articles/2006/01/20/getting-there

From Carter's blog:

One bright spot has been the recovery of my content via a tool called Warrick that uses various caching services and APIs from Google, Yahoo, the Internet Archive and others to reconstruct lost websites. So far, I’ve recovered posts for Cubanlinks going all the way back to its first post in 2002...

... I’ll describe the rebuilding process in more detail as I go along. The main point that I want to get across is this: BACK UP YOUR DATA!. The shock of losing a year’s worth of blood and sweat (regarding the code that powered DCKickball) still has yet to fully sink in. Don’t pull a Carter.

Although Warrick wasn't able to recover all of Carter's websites, he seemed pretty thankful for what he was able to get back:

It’s unclear how many posts never got recovered with Warrick in the first place. Eyeballing it, I’d say I have at least 80% of my posts. And you know what? I’ll take that.

These sites are definitely the first to be reconstructed with Warrick without my help.

Wikipedia, the study aid

In six days I’ll be defending my Ph.D. proposal. In our department, it’s called a Candidacy Exam, and it requires me to not only show that I am fully knowledgeable in my research area but also very knowledgeable in every area of computer science. The committee gets to ask me any question they want and expects a well-informed response.

In preparation for the exam, I have come to realize just how invaluable Wikipedia is for a study tool. I’ve also become somewhat addicted to updating resources in my field of study. I recently updated entries on digital libraries, OAI-PMH, and digital preservation. I also found a comprehensive section on the Churches of Christ; I’m a member of this church and learned some things I never even knew about it!

A recent study published in Nature showed that the accuracy of information found in Wikipedia is nearly equal to information found in Britannica. Britannica responded to the findings with many criticisms, pointing out that many articles in Wikipedia are poorly written and give too much attention to controversial scientific theories. I have personally found many Wikipedia articles to be very readable and somewhat complete, at least in the areas of computer science. Considering the accuracy level is not too far off from Britannica, I consider it invaluable for any student needing a crash course in the field.