Bing is "copying" Google's search results?

Earlier this week, Google revealed the results of a sting operation that appeared to catch Bing red-handed. Microsoft has apparently been using Internet Explorer to determine which queries and search results their users were clicking on when using Google, and they incorporated that information into Bing's search results. Although Google says this is unfair copying of search results, Microsoft is claiming that the information is just a small part of their overall formula for ranking search results.

The debate has received a lot of attention. Even Colbert had something to say about it: "Evidently, hiybbprqag is a word meaning 'You got serverd!'"

This debate has been good fodder for my Web Science course, and it was even relevant to my Seminar class which discussed ethics and intellectual property this morning. Search engine results are intellectual property, so is Microsoft's use of clickstream data fair use, or did they cross the line?

Update:

This post generated some interest on my Facebook account. One of the comments included this link to Danny Sullivan's article that gives more analysis of the situation.

Apple vs. Google vs. Microsoft War Chart

This infographic by Shane Snow beautifully illustrates the epic battle that Apple, Google, and Microsoft are engaged in. From ebooks and phones to office and email, all three giants are battling for more users.

Although many of these services are free to the end users, there are billions of dollars in revenue at stake. Click on the map to zoom-in.

Google: We're sorry...

I tried to access my school email account this morning, and I got this error screen:



It says:

"We're sorry... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now."

Google is sorry again that their automated query detector has been tripped. At least they aren't accusing me of having a virus this time.

Anyone else seeing this? Apparently yes.

Google Squared & Wolfram Alpha

Structuring the world's unstructured data... this is the future of search. These last few weeks have seen some impressive attempts to do just this by Wolfram Alpha and Google Squared.

Wolfram Alpha, which launched on May 18, is pulling results from their highly curated, massive database which is likely built atop massive (possibly unstructured) data sets. Google Squared, launched on May 12, is pulling results straight from the unstructured Web. These two approaches are complementary, but they are also competitive.

I'll provide just a couple of examples.

Below is Wolfram Alpha's answer to the query passing touchdowns Dallas Cowboys, Denver Broncos. Wolfram Alpha is providing a graph of data they probably acquired from a trusted source (they give some source information, but nothing specific).



The same query against Google Squared won't produce a very useful result. But a query for NFL teams results in a table of results pulled from a variety of websites. The data making up the first row is from www.detroitlions.com, a travel website, Wikipedia. Why they are not just taking information from a single trusted site like NFL.com is anyone's guess... it likely has to do with making their search algorithms more generic.


Give these search engines a try and let me know what you think.

Nutch, Sitemaps, and Google's findings

My search engine class is winding down, but our final project is to implement a Sitemap Protocol parser for Nutch, a popular open-source search engine. I mentioned a while back that Nutch is not for wimps... my students would certainly vouch for the huge learning curve to making code modifications. I've even had to scale back how much work my students do because of the complexity of changes required. I'm going to do the difficult part of integrating their code with the innards of Nutch sometime in the next few weeks.

The reason I mention our Sitemap project is that WWW 2009 is meeting in Madrid this week, and a paper entitled Sitemaps: Above and Beyond the Crawl of Duty is being presented today by Uri Schonfeld (UCLA) and Narayanan Shivakumar (Google). This is the first paper to report on widespread usage of Sitemaps in the Web using Google's crawling history.

Schonfeld & Shivakumar report that Sitemaps were used by approximately 35 million websites in late 2008, exposing several billion URLs. 58% of the URLs included last modification dates, 7% included change frequency, and 61% a priority. About 76.8% of Sitemaps used XML formatting, and only 3.4% used plain text. Interestingly, 17.5% of Sitemaps are formatted incorrectly.

The figure below represents how many URLs Google discovered via Sitemaps (red) vs. regular crawling (green) for cnn.com. Notice that on any given day, more URLs could normally be discovered via Sitemaps.



Another interesting figure (below) shows when a URL was discovered via Sitemaps vs. regular web crawling for cnn.com. In most cases URLs were discovered at the same rate, but there are a number of them (dots below the line) that were discovered via Sitemaps much earlier than web crawling.


CNN's website is not typical. Schonfeld & Shivakumar report that in a dataset of 5 billion+ URLs, 78% were discovered via Sitemaps first compared to 22% via web crawling.

The paper also describes an algorithm that can be used by search engines to prioritize URLs discovered via web crawling and Sitemaps as well. I've covered the high-lights, but I recommend you read the paper if you're interested in some of the finer details.

My first knol

As I mentioned yesterday, I just wrote my first knol entitled Introduction to Web Search Engines. This article was originally meant for my Internet Development class. I wanted to have them understand some of the technical issues of how search engines work because it affects how a website should be developed to make it Google-friendly. At the same time, I didn't want the article to be overly technical... it needed to convey just enough technical information so my students would get the big picture. Whether I hit that sweet-spot or not is debatable.

I couldn't find any similar articles on the Web (this is close), so I thought it would be useful to put one out there, and I've been eager to try out Google's new knol service. Despite my many frustrations yesterday, it wasn't too bad. I didn't find myself needing to manipulate the raw HTML too much, and the ability to add and manage references was very intuitive.

Yes, the amazing graphics are my own. I know I got skillz.

Why I (sometimes) hate cloud computing

Cloud computing offers a lot of positive benefits, namely access to data from anywhere.

But today I hate it.

I realize "hate" is a strong word, and I rarely break it out, but today I must.

I decided to write my first Knol yesterday. Google's Knol system uses an online editor and saves your data in their systems (i.e., in the cloud). After working on my article for some time, the system started having problems saving, but a warning message at the top of the screen warned that Knol would be down for about an hour. So after waiting the hour, I returned to the system and was able to make a lot of progress.

Or so I thought.

This morning when I returned to my article, only the first two paragraphs remained. I accessed the revision system to see if there were previous versions that contained my complete article, but every revision was the same: just two paragraphs. This really jolted me because I had repeatedly saved every so often just so something like this wouldn't happen. I never received a single error message after pressing Save.

Thankfully I had been smart and saved a copy of my article in Microsoft Word... my previous experience with cloud computing told me such a move would be smart. So I didn't loose my cool too much... I just copy and pasted my stuff back into Knol and then worked on some formatting issues.

About 10 minutes after I started editing, I got this error message warning me that my session had expired:


The message suggested I refresh the page. Knowing what I know, this is usually not a good way to repair an "expired" session. But there was nothing else I could do. Sure enough, after refreshing the page, my content was all gone. Back to two paragraphs.

Now I'm hot.

After taking a little walk to calm down, I returned to my office and decided to persist. I think I'm almost done with my Knol now, but I'm still feeling raw. If this is what cloud computing is going to be like, I'd rather stay on the ground. I've never had Word lose my document because my session expired.

Now you may be thinking this is an isolated issue with Knol, but I have had similar losses with Blogger (losing entire blog posts when their system was temporarily inaccessible) and Google Calendar (losing a number of appointments I had typed in but were apparently never saved). Maybe this is more a Google problem than a cloud problem, but if Google can't seem to get it right, who will?

Specifying canonical URLs

Last week the big three search engines (Google, Yahoo, and Live Search) announced their support for a new HTML attribute value which will help prevent search engines from indexing duplicate content. Search engines naturally want to avoid crawling and indexing duplicate content because it lessens the quality of search result pages. Google's Webmaster Central Blog has a good write-up about the new rel="canonical" attribute value.

Essentially, the new attribute value will allow a webmaster to tell a web crawler to ignore a page if it is accessible from another URL. So if a I have a single page that is accessible at URLs A, B, and C, I can tell the web crawler that URLs B and C are pointing to the same content as A by placing the following code in the head element of the page:

<link rel="canonical" href="http://foo.com/A" />

When the web crawler grabs the pages using URLs B or C, it will find the given canonical URL A in the header and therefore ignore the contents of the pages since they duplicate page A.

Of course the entire mechanism requires a willing and competent webmaster to implement it. Webmasters who are very concerned about SEO are likely to use it since it will help bolster the PageRank of certain pages. But the rest of us who aren't concerned about our rankings can safely ignore this new functionality.

See also rel="nofollow".

Fav5 - Jan 30, 2009

My pick of the week's top 5 items of interest:

1. Researchers at the University of Washington are developing a system for "digitally preserving and authenticating first-hand accounts of war crimes, atrocities and genocide." The system computes a hash of the video so that any changes to the video content will render it un-authentic.
   
   


2. Is the GDrive (Google Web Drive) for real? Scott Gilbertson argues that problems with security and large media files will hinder the GDrive from becoming a killer-app. Not everyone agrees.
   
   


3. April 23,2008: The highest volume of email spam caught by Google on a single day. Google blocked an average of 194 spam messages per user that day.
   
   


4. I just made available a paper on arXiv that I co-authored with Michael Nelson and Herbert Van de Sompel entitled Everyone is a Curator: Human-Assisted Preservation for ORE Aggregations. I'll be presenting this paper at DigCCurr 2009 in April.
   
   


5. This year's Super Bowl pits the Pittsburgh Steelers vs. the Arizona Cardinals. Although I really like Big Ben, I'll be cheering for Kurt Warner and the underdog Cardinals. I'll also be watching the commercials, hoping they're a lot better than last year's.
   

Fav5

My pick of the week's top 5 items of interest:

1. A typical Google search generates 7g of CO2. Or maybe not. (Thanks, Dustin.)
   
   


2. CWE/SANS just released a report of the Top 25 Most Dangerous Programming Errors. This is going out to all my programming classes.
   
   


3. The NY Times had this to say about the R programming language:
   

   "But R has also quickly found a following because statisticians, engineers and scientists without computer programming skills find it easy to use."

   Sorry, but I do have computer programming skills and still find it quite difficult to use, primarily because the documentation is lacking useful examples. That's why I created an R tutorial to help myself and others struggling with the language.
   
   


4. Looks like Warrick, the service I developed a few years ago, is still getting some good press. (Thanks, Burt.)
   
   


5. I found this on Matt Cutt's blog: Mac vs PC. What initially looks like a low-budget production suddenly transforms into footage that looks like it's from a multimillion dollar high-tech Hollywood production.
   
   
   

Fav5

My pick of the week's top 5 items of interest:

1. What will the Internet look like in 2020? Pew experts think the mobile device will be the primary means of using the Internet, the Internet will not increase social "tolerance", interaction in virtual worlds will be more common, and people will be more open to sharing private information and opinions on the Internet. Any surprises here?


2. Several new digital billboards in Japan will have cameras mounted to determine how many people are looking at the billboards. It uses software that matches faces to a average Japanese face to determine how large the audience is.


3. Google Image Search now allows you to search by several types: Photographic, clip art, and line drawing.


4. Google is now serving search results optimized for the iPhone and Android phones. The results load faster and don't require using the horizontal scrollbar.


5. Congratulations to the 446 Harding students who will be graduating on Saturday morning - a record December graduation.

Fav5

My pick of the week's top 5 items of interest:

1. Google has released its list of popular searches in their 2008 Google Zeitgeist. The most popular "what is..." query? What is love. Baby, don't hurt me.


2. According to J.L. Needham, Google's manager of public-sector content partnerships, approximately 1,000 federal government websites are inaccessible to search engines -- that is, they lie in the Deep Web.


3. How well are we preserving our video game heritage? The paper 'Grand Theft Archive': A quantitative Analysis of the State of Computer Game Preservation attempts to answer this question. Their findings aren't very positive.


4. Although this is three years old, I just came across it this week. Charles Petzold asks, Does Visual Studio Rot the Mind? Petzold's Windows programming books have been a staple of my GUI class since 1997.


5. Some Harding students have developed this very creative and entertaining Christmas card. Enjoy.

Google for football scores

Very impressive... a Google search for denver broncos shows last weekend's score as the first result (which I've highlighted in red).

Google unveils SearchWiki

On Thursday, Google unveiled their newest attempt to deliver the most relevant search results. SearchWiki allows a logged-in user to move results up and down, delete results they don't like, and even leave comments attached to results.

The screenshot below shows a search for "google searchwiki". I've highlighted the Promote, Remove, and Comment icons that appear next to each result. Note that the news result (ranked first) doesn't have any icons- Google always wants that result to be at the top.



I decided to leave a comment on the YouTube video, and as I entered the comment, Google warned me it would be visible to everyone. I also chose to promote the result. Now I can see my comment as well as 16 people who clicked to promote the result and 3 who wanted it removed:



This ability to give personal input to search results is only available to logged-in users. If you are not logged in (you don't see a "Sign out" link in the upper-right corner of the results page), you won't be able to vote.

It's not clear how Google will use personal votes to promote or degrade results, but you have to believe it will be a factor when ranking results in the future, a Human PageRank if you will. It's certainly something other search engines have been thinking about.

Fav5

My pick of the week's top five items of interest:

1. Google Flu Trends is a new tool from Google that can predict flu outbreaks by examining flu-like search terms that people google when they become sick. The Google data can be days ahead of reports released by the U.S. Centers for Disease Control and Prevention (CDC).
   
   


2. Researchers at the Graphics Lab at the University of Southern California have developed a display box using spinning mirrors that creates a beautifully defined 3D image. Make sure you watch the video.
   
   


3. Didn't I see this urine-drinking system thing in a Kevin Costner film?
   
   


4. "When can I destroy humanity?" Another nightmarish robot, this one named Jules.
   
   


5. RIP Elwin "Preacher" Roe, Harding alum (1935-38) and pitcher for the World Series Champions, the Brooklyn Dodgers (1951), who passed away last week at the age of 92.

Fav5

Happy Halloween! It's been a busy few weeks since my last Fav5. Here they are, my pick of the week's top 5 items of interest:

1. I've heard a lot of people complaining that the Web is making people dumb, but some new research shows that surfing the Web may actually help stimulate your brain. Of course, any activity that challenges your mind will accomplish the same thing.  
   
   


2. A study by browser maker Opera (based on 3.5 million crawled web pages) shows that only 4.3% of the Web is using standards-compliant XHTML, and half of the sites that display a standards-compliant badge are actually not standards-compliant. Other findings:
   
   
   
   • Adobe Flash is used on roughly 35% of all websites
   
   
   • Flash is most popular in China (used on 67% of Chinese websites)
   
   
   • Ajax is used on 3.2% of websites
   
   
   • JavaScript is used on 75% of websites
   
   
   • CSS is used on 80% of websites
     
     
   
   
3. Those of you interested in GUIs and HCI might want to read an interesting blog post on pie menus. 
   
   


4. The Deep Web gets even smaller: Google is now indexing the text in scanned PDFs.
   
   


5. Matt Cutts points out a scary change to Google's robots.txt file.
   
   

Fav5

My pick of the week's top 5 items of interest:

1. Technorati has released their State of the Blogosphere 2008 report. Some interesting findings:
   
   
   
   • 1.5 million blog posts are created each week.
   
   
   • 57% of US bloggers are male, 58% are at least 35 years old, 26% are single, and 74% have a college degree.
   
   
   • The average US blogger has been blogging for 35 months (I've been blogging for 33 months).
     
     
   • The average blog has 18,000 unique visitors a month.
   
   
   • 83% of female and 76% of male bloggers (globally) have a personal blog.


2. A recent study finds college students ignoring potential malware popup dialog boxes. Why am I not surprised.


3. Dan Waters, a Microsoft Academic Developer Evangelist, writes an informative article for getting a job in the game industry. It's a little like going pro after college... it's very difficult to do, and only a few make it.


4. Google is sponsoring Project 10^100. If you have an idea that you think could help a large number of people, submit it before Oct. 20. Google will be giving $10 million to the projects they think are the most deserving.


5. Amazon is now selling a book entitled Lazy Preservation: Reconstructing Websites from the Web Infrastructure. The book has an uncanny resemblance to my dissertation.

Fav5

My pick of the week's top 5 items of interest:

1. Google recently launched Google Audio Indexing (GAudi) which converts video to text and allows the videos to be searched. I wonder how long before Google begins to crawl mp3s and video on the Web and serve it up in their search result pages?


2. A new study by Pew Research finds that, believe it or not, almost all teenagers play video games.
   

   Fully 99% of boys and 94% of girls report playing video games... Older girls are the least "enthusiastic" players of video games, though more than half of them play. Some 65% of daily gamers are male; 35% are female.

   The most popular game teens play is Guitar Hero followed by Halo 3.
   


3. Ten top Google scientists weigh-in on what the next ten years will bring on the Internet. None of them predicted that Sergey would actually start a blog.


4. Sara Palin's Yahoo email account was hacked this week. Was she using her pet's name as her password? There doesn't appear to be any serious damage done except to her pride.


5. Happy Talk Like a Pirate Day! Now you can even search Google using pirate language.

Fav5

My pick of the week's top 5 items of interest:

1. A new version of iTunes is out, and it contains Genius, a feature which magically creates playlists for your songs. Apparently some people don't think the feature works that well. I've only had a few minutes to play with it and haven't formed an opinion yet. I do wonder why Apple would name such a feature "Genius", thus putting a clear target on its head when it doesn't perform perfectly.
   
   
2. ComputerWorld has a nice article summarizing Google's current dabble with enabling the masses to influence search results.
   
   
3. Google is expanding their digitization efforts to news archives.
   
   
4. Some researchers have built a "Facebot" application, a Facebook app which appears to be harmless but secretly performs a denial-of-service attack on an unsuspecting web server.
   
   
5. If you've never seen Pirates of Silicon Valley, you can view the entire made-for-TV movie on YouTube, at least for a little while. This entertaining movie from 1999 shows how a young Bill Gates and Steve Jobs transformed the computer world, mostly by stealing ideas. The acting may not always be stellar, but if you're interested in how Apple and Microsoft got their start, you'll love this movie.
   
   Jobs: "You're stealing from us!"
   Gates: "Steve, we're not stealing from you. So don't tell me that..."
   Jobs: "What are you doing then? What is this that I keep hearing about you developing this... what do you call it... Windows?... to compete with us?!"
   
   

Fav5

My pick of the week's top 5 items of interest:

1. Google Chrome. It's fast. It's (mostly) secure. It's open source. It's all people are talking about.
   
   Personally, I really like it. Especially the Internets easter egg.
   
   The only thing I'm not fond of is how the Omnibar is used for entering URLs and search terms. While this creates a spartan interface, it also erases my search term after I enter it. Sometimes it's helpful to have it remain visible so I can quickly perform the same search in another tab.
   
   Also it mixes my search terms with my URLs while I'm typing... that is going to take some getting used to.
   
   
2. When you write a paper, you want others to cite it, especially if you trying to get tenured. And it turns out there are some not-so-obvious things you can do to increase your chances, at least in the field of astro-physics. My guess is this is true of other fields too. 1) If your paper appears in a mailing list, make sure it's near the top. 2) Write longer papers.
   
   
3. Trivia: Which web browser was the first to display an image?
   
   
4. Interesting article on cyber warfare.
   
   
5. This week I updated my Introduction to PHP for C++ Programmers. It now includes regular expressions, exception handling, and OOP. Let me know if you see something I should change and/or add.