Get flag images from Wikipedia

I needed a large number of national flag images at a certain resolution for a project my Web Development course was working on. By examining some flag images I saw on Wikipedia, I noticed that they were creating flag images on the fly.

For example, to create the United Kingdom's flag that is 100 pixels wide, you can access this URL:

http://upload.wikimedia.org/wikipedia/commons/thumb/4/45/Flag_of_the_United_Kingdom.svg/100px-the_United_Kingdom.png

which produces this flag:

So I developed a Perl script that would automate this process for me. I've included it below for anyone else who might need flag images. Note that I had to set the user agent string, or Wikipedia would not respond properly to the http request. If you use this script to download a lot of images, please be nice throttle your requests with the sleep() command.

#!/usr/bin/perl

# This script will attempt to download the national flag
# produced by Wikipedia using the $flag country name and 
# $image_size as the image width.  By Frank McCown.

use LWP::Simple;
use HTTP::Response;
use strict;

# Width of the image
my $image_size = 100;

# Country's name
my $flag = 'the United Kingdom';

my $filename = lc $flag;
$filename =~ s/\s/_/g;
$filename = $filename . "_" . $image_size . ".png";

my $url_filename = $flag;
$url_filename =~ s/\s/_/g;

my $img_url = "http://upload.wikimedia.org/wikipedia/commons/thumb/4/45/Flag_of_" .
$url_filename . ".svg/" . $image_size . "px-" . $url_filename . ".png";

print "Getting $img_url\n";

my $ua = LWP::UserAgent->new;
$ua->agent('Mozilla/5.0 Firefox 5.6');
$ua->from('your@email.com');

my $response = $ua->get($img_url);

if ($response->is_success) {
   print "Writing to $filename\n";

   open(IMG, ">$filename");
   binmode(IMG);
   print IMG $response->content;
   close IMG;
}
else {
   print "ERROR: Could not download.\n";
}

Why I left Wikipedia

An article in this week's Newsweek reports that Wikipedia has been floundering since the spring: "Thousands of volunteer editors, the loyal Wikipedians who actually write, fact-check, and update all those articles, logged off-- many for good." The WSJ first reported the fallout almost a year ago when it was discovered that 49,000 English editors left Wikipedia during the first three months of 2009 compared to a loss of 4,900 during the same period in 2008.

Update: As one of the comments below states, the WSJ article was hasty in their conclusions. It all hinges on what you call an "editor", and a more balanced definition suggests that editors are not leaving Wikipedia in droves.

As the Newsweek article points out, there are a number of reasons why Wikipedia may be stagnating. There are so many articles already present that there is little new ground to break. Some may be scared away or frustrated by overly aggressive editors. Or perhaps "most people simply don't want to work for free."

Some research at Georgia Tech shows that editing a Wikipedia article is very challenging for computing newbies; the "Editing this way will cause your IP address to be recorded publicly" message causes lots of confusion, and this certainly prevents many from joining the ranks of Wikipedia editors.

I have always been a Wikipedia fan. I first started making serious contributions in 2004 when I was beginning my PhD research and discovered that many of the new concepts I was being introduced to simply didn't exist in Wikipedia.

I wrote a number of articles from scratch like web archiving, web search query, adversarial information retrieval, and URL normalization and made a significant number of edits on other technical topics. I was motivated in part by being the first to write the articles and the fact that I would likely refer back to them as reference material as I continued my research.

However, I found that keeping vandalism at bay and fighting poor edits was quite time-consuming. Some articles that I valued quite highly like web crawler needed tons of work, and although the desire was there, I just didn't have the time... I was trying to complete my PhD, and maintaining Wikipedia articles was not paying the bills.

I had an ah-ha moment at a conference a few years ago when someone quoted from Wikipedia's article on digital preservation, and I could have sworn I had been the sole author of the quoted piece. Wikipedia was given credit as the source, not me. That didn't bother me all that much, but it did make me realize that contributing to Wikipedia is often not in the interests of academics who are often judged by the amount of citable material they produce. Someone citing what you wrote in Wikipedia doesn't "count" like someone citing what you wrote in a journal article.

Over the past year or so, I just have lacked the motivation necessary to put time into an anonymous forum. My time is expensive, and Wikipedia is not paying. It's hard enough just to find time to edit my blog!

I still think Wikipedia is extremely valuable, and I hope it never goes away. I regularly send my students there and encourage them to make a serious contribution.

Have you seen The Book of Eli? At the end of the movie, a group of people are attempting to restore some of the greatest literary works of mankind. They are quite happy to have nearly a complete set of Britannica encyclopedias. No mention is made about the remnants of Wikipedia. :-(

Fav5

My pick of this week's top 5 items of interest:

1. It's official: Google's Knol, a Wikipedia-like source of user-contributed information, is now available to the public. Lots of people are talking about it. I found the interface a little lacking... there's a search box and a list of some randomly selected Knols, but I can't figure out how to browse by subject (of course it's difficult to do this in Wikipedia as well). Also I can't tell how many knols there are yet; a search for "a" shows about 80 knols, and most appear to be health-related. My guess is the tech guys may stick with Wikipedia.
   


2. At this week's SIGIR'08 conference, Microsoft Research presented a paper called BrowseRank: Letting Web Users Vote for Page Importance. The paper introduces a new relevance ranking algorithm called BrowseRank. Instead of relying on the web graph to assign web page importance as PageRank does, BrowseRank assigns importance based on users' browsing behavior. A good summary of the paper is at CNET News.
   


3. Facebook will soon be using Microsoft's web search technology to give search results and sponsored ads. Currently Google is powering MySpace.
   


4. According to a new report from the antivirus company Sophos, they have detected over 16,000 malicious web pages each day in the first half of 2008, most using SQL-injection techniques. Blogspot.com hosts the largest number of malicious web pages, mostly because of how easy it is to setup a blog with this service and to inject
   


5. You'd better hurry: The domain ☼.com is still available! Oh, and Google has discovered at least 1 trillion pages on the Web.
   

Fav5

My pick of the week's top 5 items of interest:

1. Bill Gates has given up using Facebook. Apparently he was getting over 8000 friend requests per day. And I thought my wife was popular...


2. New word for the day: bacn - Email that you've subscribed to that isn't spam because it's not totally unwanted, but it isn't really wanted either. Examples: Facebook friend requests and Amazon recommendations.


3. Bennett Haselton writes his thoughts on Google's new Knol project and how it is likely that many people will copy over content from Wikipedia into Knols (creating a Wikipedia fork) because of the potential to make some money. Haselton also raises some interesting ideas on how Knols will be incorporated into Google web search.


4. Every time I get stopped at a traffic light and there are no cars waiting at the intersection, I get furious. Why can't we figure this problem out? I'm happy to see that researchers from Romania and the US are trying to solve this problem. Of course their solution requires wireless communication between cars and traffic lights, something that is unlikely to happen anytime soon.


5. All Your iFRAMEs Point to Us - an interesting paper from researchers at Google and John Hopkins Univ. From the abstract:
   

   In this paper, we provide a detailed study of the pervasiveness of so-called drive-by downloads on the Internet. Drive-by downloads are caused by URLs that attempt to exploit their visitors and cause malware to be installed and run automatically. Our analysis of billions of URLs over a 10 month period shows that a non-trivial amount, of over 3 million malicious URLs, initiate drive-by downloads. An even more troubling finding is that approximately 1.3% of the incoming search queries to Google’s search engine returned at least one URL labeled as malicious in the results page.

Internet Archive and censorship

Just yesterday, an anonymous Wikipedia editor using the IP address 211.7.98.52 claimed that the Internet Archive had "censored" pages from around the 9/11 time frame. To prove their point, they provided a link to a snapshot showing the gaps in the Archive. They cited no other external source that discusses the issue.


I'm sure there is a good explanation for the pages not appearing in the Archive... most likely Alexa (who supplies a majority of the archived pages to IA) just didn't crawl those sites on those dates. The Web is a very large place, and Alexa doesn't crawl every website on the same schedule.

I've changed the wording to read "missing pages" instead of "censored", but I suspect 211.7.98.52 will be reverting the edits back sometime today. This anonymous editor obviously thinks there is something sinister going on.


Update on 1/30/2008:

Sure enough, the conspiracy theory will not die. Someone at 61.204.240.194 added further "evidence" that IA has yanked the pages, and 211.7.98.52 apparently agrees.

211.7.98.52 is apparently Victoria Sachs, or at least that's the name given in this Internet Archive forum posting which asks about the missing pages. Brewster Kahle, founder of the IA, responded to her post:

There was nothing special happening before 9/11/2001-- the attempt is to crawl every site ever 2 months. sometimes things would be more frequent, but mostly 2 months.

with 9/11 events, the crawler team tried to archive things much more frequently. the news sites had trouble staying up, so that record is a bit spotty.

I hope this helps.

-brewster

Sorry, Brewster, but that explanation will not appease Victoria and 61.204.240.194. There are other websites that were archived in the weeks before 9/11, so it must be a huge conspiracy to keep the world in the dark. Victoria also wrote:

Yet the Internet Archive's records clearly show the major sites listed above have never before experienced such an enormous 'missing cache' gap in the history of the Internet Archive.

Of course this is not true. Here's one counter example where Time.com is missing all of July 2000, a 50+ day gap. I could give number other counter examples, but alas I have students to teach. (BTW, Victoria, the IA doesn't cache pages, they archive them. There is a difference. )

Victoria's Wikipedia article edit concludes:

Unfortunately, future generations will never be able to read what Newsweek.com, Reuters.com, Altnet.org, ABC.com, Time.com, MSNBC.com, ABCnews.com, Nasdaq.com, Bloomberg.com, LAtimes.com, Timesofindia.com, CNN.com, UAL.com, CBSnews.com, and NYtimes.com, published during the weeks preceding September 11th, 2001.

With this I do agree... it is too bad IA doesn't have those pages archived. But I seriously doubt they purposely removed those pages from their Archive; it completely runs contrary to what they are attempting to do.

I am visiting with some folks at IA in San Francisco next month. I'll bring up this matter with them and report back what they say.

Update on 2/12/08:

I think the controversy has finally subsided.

On Feb 10, a user from 211.7.100.61 (Victoria?) removed their posting and signed off with

OK, censorship wins.

The Talk page basically outlines the rambling (and unsigned) thoughts of 211.7.100.61 and how other editors tried to reason with him/her. It makes for some interesting reading. Even Mr. Wikipedia got into the action:

This guy has been emailing me as well. It is an unsubstantiated crackpot conspiracy theory that doesn't pass the basic sniff test. Even the links he sends to show he is right, thoroughly refute his claims. (As said above, the gaps, while regrettable, are normal for that era of the archive.)--Jimbo Wales (talk) 17:39, 6 February 2008 (UTC)

I think that about sums it up.

Fav5

My pick of the week's top 5 items of interest:

1. Jury 101: If you're currently serving on a jury, don't blog about it.


2. Finally, a monitor which doesn't use power when in standby mode. What's the big deal? The British Government estimates 8% percent of all domestic electricity is consumed by devices in standby.


3. Interesting new Wikipedia research at the University of Minnesota: Only one-tenth of 1% of Wikipedia editors account for nearly half the content. And the probability that a user would view an article in a damaged state is 0.37%.


4. Cats: This is just weird. Almost as weird as this.
   Eggs: This is absolutely amazing.


5. Congrats to the Harding Bisons who defeated Arkansas Tech 62-55 at Homecoming and set a new Harding record for points scored in a game.
   

Fav5

My pick of the week's top 5 items of interest:

1. Which Wikipedia articles are trustworthy? A new service called Veropia thinks they know. They are hoping to act as a warehouse for Wikipedia articles that are vetted by "experts."


2. Students at Saginaw Valley State University are building a video game in their spare time. Their role-playing game is a lot more involved than the board games we produce in our CS capstone class; they are enlisting help from 15-60 students and hoping to release their game in Nov 2008.


3. Does NBC's Pres Jeff Zucker really believe "Apple has destroyed the music business"? If anything, Apple should be heralded as the company that finally pulled the music industry out of the Dark Ages. Tone down the rhetoric, Zucker.


4. Google and an alliance of social network parters are releasing OpenSocial APIs, open services that allow social applications to be created on sites besides Facebook. Facebook has so far been excluded.


5. This is Harding's Homecoming weekend. Go Bisons!
   

Fav5

My pick of the week's top 5 items of interest:

1. According to the Commission on Professionals in Science and Technology (CPST), Enrollment in bachelor's programs in computer science have dropped 40% from 2001 to 2006, with many students avoiding the field because of "increased risk of job loss due to offshoring and other issues." Unfortunately, we educators are not doing enough to recruit CS students, and many falsehoods continue to be circulated about job risk. In fact, now is one of the best times to be a CS major. The job market is very strong, and many predict that it will continue. Money Magazine recently ranked the top 10 professions, and 4 of the 10 can be fulfilled by CS majors!
   


2. This is pretty cool: You can now use the Sitemap Protocol to tell Google about source code housed on your website. This allows Google to improve the accuracy of its Google Code Search.


3. Congrats to Microsoft who beat out Google and Yahoo in a partnership deal with Facebook. It's tough to beat Google at anything these days.
   
   
4. Neil L. Waters writes an interesting piece in CACM entitled Why you can't cite Wikipedia in my class. This is something every student should read. You may not have the proper permissions to view the article, so I have temporarily placed it here.
   


5. My Colorado Rockies are down 0-2 in the World Series against the Red Sox. Here's hoping a return to Denver will bring back the magic.
   

Fav5

My pick of the week's top 5 items of interest:

1. When you make a closed system, you are basically inviting every hacker out there to break it open. Just a few days ago, some college-bound kid posted on his blog how to "unlock" the iPhone (use another cell phone provider besides AT&T). Wish Apple would have saved him the trouble in the first place.
   


2. This is a little odd: The Perverted Justice group is calling Wikipedia a "Corporate Sex Offender". Apparently someone from PJ was getting a little out of hand on Wikipedia and got banned and is now overreacting. What in the world is a corporate sex offender anyway?


3. Researchers at MIT have developed a clutter detector, a method for reducing the clutter from visual displays. My wife would love to have one of these for the house.


4. Here's something my wife would really love: an automated essay grading system.


5. Justice is finally served: Linux felon forced to install Windows.
   

Fav5

My pick of the week's top 5 items of interest:

1. Have you heard of MyDeathSpace.com? It archives the MySpace page of those who die. A little morbid, but interesting nonetheless.


2. American Airlines doesn't like it that Google allows others to compete for the words "american airlines" when they do a search. I'm siding with Google on this one.


3. Some Wikipedia entries were edited by users on CIA and FBI computers about topics that the FBI and CIA probably know more about than anyone else. Wikipedia says it violates their neutrality guidelines. I say give me a break.
   


4. There's an interesting article in Information Week about the future of web search technologies.


5. Who would have guessed? Swapping votes over the Internet is legal.
   

Citizendium or Wikipedia?

A few weeks ago I applied to be an editor on Citizendium, a new wiki project intended to be a more accurate and reputable Wikipedia. Citizendium does not allow anonymous postings, and they put new articles under a review process. My application was accepted a few weeks later, and a user account was created for me along with a user page listing my brief CV.

I decided to warm up with an article on digital preservation, a topic which I feel qualified to write about. Rather than start from scratch, I imported the Wikipedia article on digital preservation. Although Citizendium frowns upon importing articles from Wikipedia, I had previously written a large portion of the Wikipedia article, so I didn't feel too bad about doing it. I cleaned up the article by focusing the definition, cleaning up the references, and removing the numerous external links.

I didn't spend a whole lot of time editing the article because I got to thinking... can Citizendium really compete with Wikipedia? Larry Sanger (Citizendium's founder and co-founder of Wikipedia) seems to think so. Although I agree that Citizendium's policies in theory would result in more reputable articles, I don't think Citizendium can possibly scale to Wikipedia's size. First, there are a number of people who want to make a minor contribution to a Wikipedia article, a slight correction or clarification, for example. And since they don't have to register, the barrier to entry is sufficiently low enough for them to contribute.

Second, there are a number of people who, for whatever reason, want to remain anonymous or known by some alias. They are not likely to sign on with Citizendium and convey to the world why they are qualified to write about XYZ.

Third, if someone wants to make a contribution to an article on a particular subject, now they have to decide do they make the contribution just on Wikipedia, or on Citizendium, or both? Do they monitor both sites for changes to an article that is important to them? If a Citizendium article is actually better than the Wikipedia article, what is stopping Wikipedia from just importing the entire Citizendium article?

Forth, choosing the name "Citizendium" was a poor choice. It doesn't exactly roll off the tongue.

I really do hope Citizendium takes off, but I think its going to take a very, very long time before they have anything near the number of articles that Wikipedia has to offer. In the meantime, I would much rather put my efforts into something I know is going to rank high in search results and gets far more page views. I'll be watching Google to see if my Citizendium article will ever beat out Wikipedia's (currently ranked number 8) in the search results for "digital preservation". If it ever does, I'll defect to Citizendium.

Fav5

It's been a few weeks since I've posted a Fav5, but now that I'm through with traveling, I'm back at it.

1. Internet music is about to die. Thursday marked a day of silence in protest of the absurd fees that must soon be paid by Internet music providers. Very sad indeed.
   


2. The iPhone is finally here. Being the traditional late adopter that I am, I don't really give a hoot, but it's entertaining to see all the technophiles go crazy over it.
   
   
3. An interesting article in the NY Times discusses social searching and the new searching paradigms that may challenge Google's lead in the Web search field. Matt Cutts says (unofficially) that Google is not opposed to leveraging the wisdom of the crowds in the future to stay in the lead.


4. Researchers at Northeastern University have demonstrated the minimum number of moves needed to solve a Rubik's Cube is 26.


5. I came across Citizendium this week. It's essentially a Wikipedia where only the educated and pre-approved can edit articles. I submitted my credentials and am waiting to hear back. In theory, Citizendium should be of higher quality than Wikipedia, but I think the barrier-to-entry is just too high to make it a serious contender. Besides, sometimes anonymous contributions can be useful.

Wikipedia: nofollow and noMSedit

Some new news from the world of Wikipedia:

• Minor: All external links from Wikipedia are now using the NOFOLLOW attribute. This attribute tells web crawlers like Google that the link has not been vetted, so it will not be used in their algorithms to artificially bolster the ranking of some pages. Wikipedia’s action will seriously reduce the amount of link spam that currently plagues many entries.
  
  
• Major: Microsoft has attempted to hire Rick Jelliffe, chief technology officer of XML tools company Topologi Pty. Ltd., to “correct” Wikipedia entries on ODF (OpenDocument format) and OOXML (Microsoft Office Open XML). You can see Rick's original post about the offer here. Apparently Wikipedia is keeping Microsoft employees from making the edits themselves, so Microsoft thought a third party could update the entries that apparently shed a negative light on Microsoft’s format. This astroturfing blunder has created quite a few waves.

Use Wikipedia to make your computer smarter

Still on the Wikipedia kick... Researchers from Technion-Israel Institute of Technology are using Wikipedia to give computers context information and make connections between different words. See the article here. For example, when a spam filter encounters a word like “B12” and needs to determine if the email should be marked as spam, the filter currently doesn’t know that B12 is a vitamin (the subject of many spam emails) unless the email also uses the term vitamin. But by examining the Wikipedia article on B12, the spam filter could be smart and deduce that an email with B12 is trying to sell vitamins. The same information could have been obtained by searching for B12 using a search engine, but the results aren’t necessarily vetted. That’s the Wikipedia advantage.

Reading lists from Wikipedia

Alexander D. Wissner-Gross, a physics Ph.D. student at Harvard, presented his paper this summer entitled Preparation of Topical Reading Lists from the Link Structure of Wikipedia at ICALT'06. Wissner-Gross shows how an algorithm based on PageRank can be used to generate background reading lists from Wikipedia. I especially like this paper because it is the solution to a real teaching problem that Wissner-Gross encountered when preparing to teach one of his courses: how can we automate the time-consuming process of generating a quality reading list for a class?

Update on 1/30/07:

Wissner-Gross emailed me this morning with the web address of the reading list engine: http://www.wikiosity.com

I got some interesting results for Digital preservation. Although Digital obsolescence popped up first, some irrelevant results like Vanderbilt University and University of Virginia also popped up. A search for web crawling brought up 2003 as a result. I'm not sure if these lists would be more useful than if I looked directly at the See also section, but it's still an interesting idea.

Wikipedia, the study aid

In six days I’ll be defending my Ph.D. proposal. In our department, it’s called a Candidacy Exam, and it requires me to not only show that I am fully knowledgeable in my research area but also very knowledgeable in every area of computer science. The committee gets to ask me any question they want and expects a well-informed response.

In preparation for the exam, I have come to realize just how invaluable Wikipedia is for a study tool. I’ve also become somewhat addicted to updating resources in my field of study. I recently updated entries on digital libraries, OAI-PMH, and digital preservation. I also found a comprehensive section on the Churches of Christ; I’m a member of this church and learned some things I never even knew about it!

A recent study published in Nature showed that the accuracy of information found in Wikipedia is nearly equal to information found in Britannica. Britannica responded to the findings with many criticisms, pointing out that many articles in Wikipedia are poorly written and give too much attention to controversial scientific theories. I have personally found many Wikipedia articles to be very readable and somewhat complete, at least in the areas of computer science. Considering the accuracy level is not too far off from Britannica, I consider it invaluable for any student needing a crash course in the field.