Link rot in CACM

I was really surprised this afternoon to see an article in the Communications of the ACM that cited a cached URL from the MSN search engine in place of a missing web page:

The link to the U.S. Secret Service “Operation 4-1-9” report at www.secretservice.gov/alert419.htm appeared to be broken when this column was written, but cached copies remain available (for example, cc.msnscache.com/cache.aspx?q=3910458378891〈=en-US).
Communications of the ACM, Volume 49, Number 12 (2006), Page 18.

The editors of CACM may not be aware of this, but search engines do not keep cached copies of pages long. In fact, they will often purge their caches of any web page that returns a 404 when crawling. (You can read my paper on an experiment which illustrates this.) Citing a cached page from a search engine should never be done in academic writing. Instead of citing just one broken URL, CACM has now cited two.

If you are interested in learning more about link rot and how to combat it, check out this Wikipedia article that I contribute to.

Speaking of link rot, Baden Hughes of the University of Melbourne has recently published a study entitled Link? Rot. URI Citation Durability in 10 Years of AusWeb Proceedings. (Not sure why he used a question mark in his title.) He used many of the methodologies that I used when examining link rot in D-Lib Magazine last year. Turns out AusWeb URLs have a much lower half-life (6 years) than D-Lib article URLs (10 years). This is probably because authors of D-Lib articles are more aware of link rot than authors in other professions.

I’m curious if any other on-line magazine or journal can beat D-Lib’s 10 year half-life. I have a suspicion JMIR articles could since many of them use WebCite.

Google's cached date = last request date

Vanessa Fox, a member of the crawl team at Google, announced on Tuesday that Google would start posting the last request date on their cached pages. Google used to only indicate the date that the page was last retrieved, so if Google made an If-Modified-Since request and the web server responded with a 304 (not modified) response, the cached date would be left unchanged. Now the cached date will indicate the date of the 200 or 304 response. Matt Cutts also discussed this change and even made a little video for those that needed a visual explanation.

Frankly, I was very surprised to learn that Google’s cache date worked this way. In effect, it’s was much like Yahoo’s Last-Modified date… it was really just an indication as to when they noticed the page changed. I have crawl data from 2005 that indicates Google would periodically issue regular HTTP GET requests, possibly just to verify that the content had indeed not been changing.

I’m not totally sure what MSN’s cache date is indicating. From my 2005 crawl data, MSN apparently never issued an If-Modified-Since request. If they are still operating with the same crawl policy, then they are storing the time they last contacted the web server, so their cache date would indicate the same thing as Google’s.

What this means for Warrick: Google will more frequently now have the most recent version of a page. Therefore Google’s overall percentage of contributed resources will likely increase in the reconstructions I’ve been performing the last few weeks.

On a side note, someone asked Matt Cutts why Google does not post the cached date of PDFs, and Matt said he was going to ask the crawl team about it.

That’s not a cache... that’s an archive!

This morning I stumbled across a March 2006 blog posting by Danny Sullivan entitled “25 Things I Hate About Google”. Sullivan’s opinions carry a lot of weight in the search engine world, and so I started to sweat when I saw number 9 on his list:

9. Stop caching pages: I was all for opt-out with cached pages until a court gave you far more right to reprint anything than anyone could have expected. Now you've got to make it opt-in. You helped create the caching mess by just assuming it was legal to reprint web pages online without asking, using opt-out as your cover. Now you've had that backed up legally, but that doesn't make it less evil.

Sullivan doesn’t agree with the January 2006 Nevada federal court ruling that declared Google’s cached pages did not constitute copyright infringement, thereby okaying the opt-out policy used by search engines using the noarchive meta-tag. Sullivan and others make some good points in the forum discussing the ruling, showing where the ruling may have some flaws.

One the arguments opponents of the ruling make is that a search engine cache is hardly a cache in the traditional sense because pages are cached long after they are changed or deleted from a web server. One of the posts by mcanerin gives an example of a web page that had been cached for almost 2 years (the example is no longer accessible). In mcanerin’s words: “That's not a cache, it's an archive.”

The fact that the cache is more like an archive is exactly what makes it beneficial to most Web users, and that’s why I think the court’s judgment was fair. Search engine caches are a huge public good. Caching is not evil. Yes, there may be a few scenarios where caching may not work to everyone’s benefit, but in most cases the good far outweighs the bad. As long as search engines provide a mechanism to keep crawled content from being cached and to remove cached content immediately if needed, then there is no really compelling reason to force search engines to use an opt-in policy. (Yes, I know it can be a real pain to manually remove entries from many search engines, but how often does anyone really need to do that?)

My research on digital preservation of websites relies heavily on the wide-spread use of search engine caching, and if caching turns from an opt-out to opt-in, I am going to be in serious trouble, and so are users of Warrick. I’ll be keeping my eye on this…