Report on InDP in D-Lib Magazine

My report on the Innovation in Digital Preservation workshop (InDP 2009) has just been published in D-Lib Magazine. Overall I think the workshop was a success, although we really missed not having Andreas Rauber there. I'm not sure if I'll be the one to lead the 2nd InDP, but I hope there will be one in the future.

Thanks to Spencer Lee (Virginia Tech) who filmed the workshop and created a virtual presence for InDP in Second Life, where the memories of InDP will last forever (or five years, whichever comes first). Below are some screenshots from Second Life that Spencer sent me.

What are you doing this summer?

I've been asked a number of times what I'm doing this summer since I'm faculty and have no classes to teach. Last summer I was doing research in Los Alamos, but this summer has been very different. A lot of my time is spent at home, getting adjusted to life with a newborn and toddler and helping Becky get some extra sleep in the mornings.

Professionally, I've presented a few papers at a conference, co-chaired a workshop, and am working on a paper about my search engine courses.

But most of my working days are spent producing a series of instructional videos for Introduction to Programming with C++ (2nd ed) by Y. Daniel Liang. You can sample a video I made just this week on file I/O. I'm not sure if the videos will be available to book owners only or made freely available on the book's website. I'll hopefully wrap these up by end the end of July and then start on videos for Liang's Introduction to Java (8th ed).

I'll be preparing soon for my Games Programming course. This course has only been offered once at Harding before, and it was taught by Dana Steil who is currently away working on his PhD. I'm excited about teaching this courses, but it's also a lot of work to teach a class for the first time, and it's a little disconcerting that I will likely not get to teach it again since Dana will likely want the course back when he returns.

So that's my summer. What are you doing?

Update:

I'm no longer doing Liang's Java book. I didn't finish the C++ videos until Aug... where does the time go?

Power.com: Give me your Facebook data!

TechCrunch is reporting that Power.com is suing Facebook over their lack of data portability. Power.com is a service which allows you to aggregate your various social networks into a single location, but Facebook's data, as indicated in their Terms of Service, is still off-limits to them. Disregarding the restrictions, Power.com tried using the Facebook API and screen-scraping to get their data until being sued earlier in the year by Facebook.

This is exactly what I've been working on (with a graduate student at ODU) for the last few months. But I'm doing this to preserve the data, not necessarily to aggregate it along with other social networks. However, there's no reason why a preserved Facebook account could not be uploaded into another service.

My guess is my approach won't be looked at kindly by Facebook, but they'll probably leave me alone since I'm only providing a service for individuals to archive their account, and I'm not aggregating the data to my own server.

Email Preservation Parser

Here's an excerpt from an email announcement I received from Riccardo Ferrante (Smithsonian Institution Archives) about a tool for preserving email. It was one of the tools developed by the Collaborative Electronic Records Project (CERP).

The Email Parser migrates an email account and its messages into a single XML file using the Email Account XML Schema developed in collaboration with the North Carolina State Archives and the EMCAP project.

The CERP Email Parser migrates an email account in MBOX format into XML, using the schema to preserve the full body of messages, together with their attachments, and keeps intact the account’s internal organization (e.g., an Inbox containing subfolders labeled Policies, Special Events, and Projects). The CERP team successfully preserved email accounts from a variety of applications including Microsoft Outlook, AppleMail, LotusNotes, and Netscape. All email messages retain their full header content, in contrast to some tools produced in earlier research efforts.

Elrod on Twitter and Iran

I just found out that Harding Professor Mark Elrod was interviewed just a few days ago by Jessica Dean on KATV-7 about Iranians using Twitter (see the video below). Just a few months ago David Adams was interviewed by THV-11 about the history of the flu. Looks like the vast expertise of our history dept is starting to get tapped by the local press.

I'm at JCDL 2009 in Austin

JCDL 2009 is about to wrap up. It's been a good conference with some interesting presentations, and I've enjoyed catching up with old friends. The conference is being held on the UT campus... short on grass but big on buildings. I think the UT football stadium is more impressive than many NFL stadiums I've visited. I guess that's what happens when you win a few national championships.

I especially enjoyed the two panels. The first panel, What should we preserve from a born-digital world?, basically came to the conclusion that everything should be saved. I concur... disk space is cheap, and it's hard to know what will truly be valuable years from now. I also enjoyed hearing about Megan Winget's work in preserving games.

The second panel, Google as Library Redux, discussed the unfortunate conclusion of Google's lawsuit with publishers and authors, agreeing to a settlement instead of pressing the court to settle the bigger questions in regards to copyright, orphaned works, etc. One of the more provocative statements came from Michael Lesk who said JCDL was irrelevant because there were no attendees from Google, Amazon, Microsoft, etc. We are being ignored. Ouch. But he may be right. I see plenty of guys from Google et al. at the WWW and SIGIR conferences.

I gave a couple of talks this year (see my slides below). There was a lot of interest particularly in my Facebook paper, What Happens When Facebook is Gone?, where I discuss the ramifications of having all our data locked-up in the walled garden of Facebook. Carlton Northern, a graduate student at ODU, is currently working on a Facebook archiving add-on for Firefox, and hopefully it will be available soon.




My second paper, A Framework for Describing Web Repositories, is work pulled from my dissertation. In it I discuss how we can view web repositories (everything from a search engine cache to a web archive) in a more abstract manor. I propose some new terminology and an API that web repositories could/should implement to be helpful to clients accessing the repository's contents.




Tomorrow I'll be co-hosting a the InDP 2009 workshop. It's an all-day event, and I'll be flying home late tomorrow night. It'll be good to be back with the family.

I think I'm going to be sick...

No need to blatantly lie to your professor anymore... a new "service" helps students deceive their professors by giving them a corrupted file to turn-in, possibly buying them a few more hours or days to work on their assignment. When the professor goes to access the assignment and notices the submitted file was corrupted, he'll just ask the student to re-submit her file. The student is happy to oblige, and this time she submits the completed assignment to the unsuspecting professor.

I'm not sure if I'm more sickened by the thought of someone developing such a service or the thought that they are likely to be quite successful.



Update on 6/22/09

I thought about this problem a little more, and there's really a simple solution for the technically-inclined.

1. Have the student produce an MD5 hash of the file before it is emailed or submitted to the professor, and have the student email the hash to the professor.
   
   
2. If the received file is corrupted, the professor should produce an MD5 hash of the file. If it matches the hash from the student, he received the correct file, so the student's original file was corrupted. Let him bring in his laptop and show you how his file could be opened successfully on his machine since it won't open on yours. Probably he won't be able to, so give him a zero.
   
   
3. If the submitted file's hash does not match the submitted hash, the file got garbled in transmission or the student did not email the correct hash. The student should just resubmit the file... eventually the received file's hash should match the original hash. If the student is not able to produce a file that matches the original hash, he's either incompetent because he did not properly create the original hash, or he modified the original file (which he shouldn't do if it's finished), or he's trying to cheat. Either way, give him a zero. (Wow, I'm mean!)
   

Tweet this: Manor one of 20 developers to follow

Elijah Manor, one of our Harding CS graduates, was just listed in 20 Developers to Follow on Twitter. Very cool.

Google Squared & Wolfram Alpha

Structuring the world's unstructured data... this is the future of search. These last few weeks have seen some impressive attempts to do just this by Wolfram Alpha and Google Squared.

Wolfram Alpha, which launched on May 18, is pulling results from their highly curated, massive database which is likely built atop massive (possibly unstructured) data sets. Google Squared, launched on May 12, is pulling results straight from the unstructured Web. These two approaches are complementary, but they are also competitive.

I'll provide just a couple of examples.

Below is Wolfram Alpha's answer to the query passing touchdowns Dallas Cowboys, Denver Broncos. Wolfram Alpha is providing a graph of data they probably acquired from a trusted source (they give some source information, but nothing specific).



The same query against Google Squared won't produce a very useful result. But a query for NFL teams results in a table of results pulled from a variety of websites. The data making up the first row is from www.detroitlions.com, a travel website, Wikipedia. Why they are not just taking information from a single trusted site like NFL.com is anyone's guess... it likely has to do with making their search algorithms more generic.


Give these search engines a try and let me know what you think.

Thousands of websites about to bite the dust...

Yahoo announced a month ago that it was pulling the plug on GeoCities, one of the Web's first free web-hosting services. There doesn't appear to be any plan to migrate the thousands (millions?) of websites this will affect to other services. If you don't act by the end of the summer, you're Geocities website will disappear.

That is unless the Internet Archive has grabbed a copy, but they aren't likely to have many pages from each Geocities website archived. I've been conversing with someone who lost a backup of her Geocities website years ago, and IA only had a handful of pages archived. This is likely going to be a recurring story in the years ahead.

My first website was on Geocities. In fact, that's how I first learned how to use HTML in 1997. I'm so embarrased by that first website that I'm keeping the address a secret. I fear the day the Internet Archive's Wayback Machine has full-text search, because someone's going to pull it up and post it on Facebook or something. That's one stream of bites I'm not afraid of losing.

Flight simulator site AVSIM destroyed by hackers

This morning I got a call from an individual who alerted me to the AVSIM tragedy. Apparently this popular flight simulator website with 13 years of articles, forum posts, etc. was not being backed-up properly, and a hacker took them out.




Tom Allensworth, the website's founder, stated:

"The method of the hack makes recovery difficult, if not impossible, to recover from. AVSIM is totally offline at this time and we expect to be so for some time to come. We are not able to predict when we will be back online, if we can come back at all."

It's possible Warrick could recover a significant amount of lost content, but I have not heard from anyone at AVSIM about it. Perhaps they are using it now as we speak.

Braden William McCown has arrived!

Braden made his appearance at 2:38 pm this afternoon. He was 8 pounds, 4 ounces, and 22" long. He had a basketball in one hand and a tennis racket in the other which made the birth very painful , but we were very thankful he decided to come during the day instead of the middle of the night. Ethan was excited to meet Braden and even gave him a couple of kisses. Let's hope they remain good buddies!




Becky and I are very appreciative of all the calls, emails, and Facebook messages we've received. We are excited to introduce you all to the little guy.

God is good!

Java Sitemap Parser

I've just released the Java Sitemap Parser on SourceForge.net. The software is capable of reading Sitemaps in XML, Atom, RSS, and text format. As far as I can tell, this is the first open source Sitemap-parsing software available on the Web.

The Java Sitemap Parser was the final project for my Search Engine Development class. I talked about the project a few weeks ago and how prevalent Sitemaps are becoming. Originally we wanted to add Sitemap support to Nutch, but developing just the parser proved to be quite a task. By releasing it as an independent project, I'm hoping Nutch, Heritrix, and other open-source crawlers will integrate it into their systems.

I love my teacher evaluations

Every semester I usually get evaluated by my students (I just got my results back today). They answer questions like, "How effective has the instructor been in this course?" and "Rate the instructor's command of the subject matter." All responses are anonymous.

This is common practice at most universities, and it often creates terror in the hearts of many faculty. I've known colleagues who have never read their teacher evaluations for fear of what their students might say, and I've known others who can still recite word-for-word some of the cruelest comments made by students over 20 years ago.

I've received my share of poor evaluations, especially when I was a new teacher. It took me a few semesters to get the hang of teaching, and now my evaluations are generally good (not great, but typically a little higher than the average Harding professor).

What I've found over my 10+ years of teaching is that some students give really helpful comments that can help you improve your class next time around. "I wish we could have spent some time discussing how to apply some of the new principles we learned to our project." Some students are going to really like you and let you know it. "The professor had good teaching skills, was responsive and helpful to questions, and was very knowledgeable."

Other students... well... you have to take their comments with a grain of salt. You have to realize that some students are not going to like it if you require them to work hard (many students think they should receive a B just for attending every lecture). Some students are just poor at evaluating others' performance. Others have yet to realize that they are responsible for their own learning. Occasionally a student is going to be having a bad day, and you're anonymous evaluation is going to be the perfect target.

What really helped me was learning how to properly interpret students' remarks and judge whether the criticism has merit or not. I think learning this skill is important to any new faculty member, otherwise you'll be crying yourself to sleep after reading your evaluations.

Here are a few comments I've received over the past couple of years along with my interpretation of said comment and response.

1. Student 1: The projects expected a lot from the students.
   Student 2: Smaller, less-brutal projects would not be a bad idea.
   
   Interpretation: I thought this class was supposed to be easy!
   
   Response: If computer science was easy, we wouldn't be getting paid like we are, and everyone would be doing it. The projects are tough because I'm preparing you for the far more difficult and complex projects you'll encounter when you enter the workforce. You'll thank me later.
   
   


2. Have different projects that we can choose from instead of making everyone do the same project.
   
   Interpretation: I like my classes like my Burger King - my way!
   
   Response: I always entertain ideas for new projects, but it's unreasonable for any teacher to spend hours coming up with a menu of project choices to cater to every whim. In a software development job, you are unlikely to have a boss ask you which project you'd like to work on... you'll work on what needs to be completed.
   
   


3. Instead of making us use the programming language you want us to use, let us use one we are already familiar with.
   
   Interpretation: Learning something new is highly overrated.
   
   Response: If you graduate from Harding being comfortable with only one or two languages, you should get your money back, because we haven't adequately prepared you. You'll need to learn new languages all the time as a working professional.
   
   


4. Disable the Internet on the classroom computers so that we can only access web sites are necessary for class. Remove Solitaire, Minesweeper, Hearts, etc. from the computers.
   
   Interpretation: Save me from myself!
   
   Response: I appreciate this student's honesty. I asked our lab administrator today to remove all games. There's going to be some very disappointed students next Fall.
   
   


5. Student 1: The fast pace of the class made it difficult to fully learn concepts.
   Student 2: It felt like sometimes you paced the classes very slowly.
   
   Interpretation: The pace of the class is perfect!
   
   Response: If roughly the same number of students complain that the pace of the course is too fast and too slow, I know I'm covering it at just the right pace.
   



6. You try to cover too much material for a semester. Your previous classes didn't have to learn as much as we've had to. :-(
   
   Interpretation: Curse you ever-evolving technology!
   
   Response: One of the enigmas of higher education is that the consumers (the students) are often happier to receive less for what they are paying for (education). Can you imagine the same student being upset if McDonald's gave him a large order of fries for the price of a medium? Harding should fire me if I quit trying to keep my classes current and just teach the exact same stuff every semester.
   



7. Don't give us really hard assignments, and don't expect us to have them done by the next class period... we do have other classes and lives!
   
   Interpretation: I'm serious about "me" time.
   
   Response: You should schedule 2-3 hours of outside-class time for each hour you are in class. (This is a universal rule that applies to all your major courses, not just mine.) So if I give a homework assignment on Mon and expect it due Wed, you should have already allocated 2-3 hours (at least) to getting the assignment finished. If your assignments are taking much longer than that to complete on a regular basis, that's a sign that you need to start getting some extra help and adjust your schedule accordingly. Remember that half of the class thinks we're going too slowly (see #5 above).
   
   


8. Weaknesses of the instructor: Calvinism
   
   Interpretation: ???
   
   Response: "Isms in my opinion are not good. A person should not believe in an ism - he should believe in himself. I quote John Lennon: 'I don't believe in Beatles - I just believe in me.' A good point there. Of course, he was the Walrus. I could be the Walrus - I'd still have to bum rides off of people." - Ferris Beuller
   

Update:

Inspired by Jordan's comments, I have added a little to my original post.

Spring semester is over

I wrapped up all my grading today. We have a senior reception tonight and the graduation ceremony tomorrow.

Below is the grade distribution for my Intro to Programming, Internet Development, and Search Engine courses. The average was 79.0, and the median 85.4. If I had time I'd compare this to my past semesters, but I don't think much has changed.


I guess I'm a little guilty of grade creep... the average student is supposed to get a C, right? I think my students would argue with that conclusion. A recent survey found that 30% of college students agree with the statement: "If I show up to every class, I deserve at least a B." Surely that percentage isn't nearly as high at Harding.

Team Digital Preservation

In an effort to bring digital preservation to the masses, DigitalPreservationEurope (DPE) is developing an entertaining series of short animations introducing and explaining digital preservation problems and solutions. Below is their first video. It's a throw-back to animated cartoons of the 1960s, and it is fantastic. Watch as Team Digital Preservation thwarts Team Chaos' plans to disrupt digital information from a nuclear power plant.

"You fiend! It's essential to have long term stable and trusted information on how nuclear power plants are built and what's inside them!" - DigiMan

Future cartoons will be made available on DPE's You Tube Channel.

Improving movie recommendations

If you haven't yet checked out the new CACM blogs, you need to soon. One of the posts that caught my attention was Greg Linden's What is a Good Recommendation Algorithm? Linden wonders if Netflix's one million dollar reward for a better recommendation engine is a little short-sighted. The goal for their recommendation system is to only show people how much they might like a movie. But Linden points out:

However, this might not be what we want. Even in a feature that shows people how much they might like any particular movie, people care a lot more about misses at the extremes. For example, it could be much worse to say that you will be lukewarm (a prediction of 3 1/2 stars) on a movie you love (an actual of 4 1/2 stars) than to say you will be slightly less lukewarm (a prediction of 2 1/2 stars) on a movie you are lukewarm about (an actual of 3 1/2 stars). Moreover, what we often want is not to make a prediction for any movie, but find the best movies. (emphasis mine)

Shifting gears a little, I want talk about a couple of small fixes to an existing movie recommendation system that could make customers a lot happier.

I haven't used Netflix, but I've been using Blockbuster Online for over a year, and I've played with their recommendation feature a lot. I would assume their recommender is on par with Netflix (hint: someone needs to compare the two).

One feature Blockbuster offers allows you to select "Do not show me this movie again", a little icon on the side of each movie's ratings. I've clicked this icon a lot (is it just me, or there's a lot of garbage out there?), hoping Blockbuster would stop recommending these specific movies to me and others like them. However, the screen shot below is what I saw this morning when I logged into my account:


Note how I was recommended "Zack" and "Quarantine" despite having clicked on the no-show icon weeks ago. They also recommend , a movie I've already rated (and therefore obviously seen). But since I didn't rent "Changeling" directly from Blockbuster, they still offer it as a movie I "might have missed."

These movies do not appear in my formal set of recommendations (the screen that results from clicking on the Recommendations link), so my guess is Blockbuster is using a different set of algorithms to populate their might-have-missed list from their formal recommendation list. However, I suggest that the might-have-missed list should take advantage of previous ratings to improve overall customer satisfaction.

This should be common sense: Do not suggest a movie that a user has already marked "do not show me this movie again". Especially not on the first page the user sees when logging into your site.

One more point. Below is a screen shot from the first page of recommendations made by Blockbuster. None of the movies below appeal to me, but I can see how they might have been recommended based on my viewing history and ratings.



But one movie really stands out as a bad recommendation: "Swing" (bottom-left). Note how it has only received two stars on average, equivalent to "I didn't like this movie".

Why would Blockbuster think I would like this movie when most people don't?

I know my taste in movies is probably not typical, but I don't think I've ever given a movie with an average rating of two stars a rating better than two stars. Even if Blockbuster thinks this movie matches my tastes, it would make much more sense to put movies with higher overall ratings on the first result page and bump lower rated movies back a few pages.

My experience in general has been that Blockbuster's recommendations don't really work. I've found one recommended movie in the past year that I thought looked interesting. Then again, I don't often try iffy movie recommendations because I'm not ready to gamble on two hours of a nice evening.

I'm looking forward to a time when the recommendation system really works well, but until then, I'll be consulting with my friends and family who have a much better idea of what I really like to see.

Micah Pate has been found

If you haven't already heard, Micah Pate's body has been found. Micah's husband Thomas is being charged with the killing this morning.

Micah Rine Pate was a Harding University graduate and Searcy native. Her parents are employees of Harding and Harding Academy. As you can imagine, the Searcy community has been rocked with this story. Our prayers go out to the Rine family and to Thomas' family.

The photo on the right is a screen shot of Micah's Facebook page. Many of her friends are posting sad farewells to her and telling her family how much they loved her. Her account will likely remain active as long as Facebook is around. I imagine her family is going to "capture" her Facebook account as well as an artifact of remembrance. I'm presenting a paper on this subject in June at JCDL 2009.


Update:

Two vigils in Searcy were held for Micah and the Pates, one at Harding. KARK 4 News had a news story about it last night. One thing that comes across in the story and interviews is Micah's faith and the positive influence she has had on others.

Upload an image in PHP

I created this function for my Internet Development students which saves a single uploaded image to disk. Example:

// Assuming the web server has write permissions to /mydir
SaveUploadedImage("/mydir/myimage.png");    

The function can easily be modified to handle multiple filenames (change the parameter to accept an array of filenames and modify the final foreach block). Note that this is modified code from the webdeveloper.com forum. If you want to know more about uploading files in PHP, check out the PHP - File Upload tutorial.

// Return empty string if uploaded image is successfully saved as
// $image_filename or an error message. $image_filename should be 
// saved in a directory that the web server can write to.
function SaveUploadedImage($image_filename)
{
   // This function is greatly modified code from
   // http://www.webdeveloper.com/forum/showthread.php?t=101466
   
   
   // Possible PHP upload errors
   $errors = array(1 => 'php.ini max file size exceeded',
                2 => 'html form max file size exceeded',
                3 => 'file upload was only partial',
                4 => 'no file was attached');
    
   // Store nonempty files in the active_keys array
   $active_keys = array();
   foreach ($_FILES as $key => $file)
   {
      if (!empty($file['name']))
         $active_keys[] = $key;
   }

   // Check at least one file was uploaded
   if (count($active_keys) == 0)
      return 'No files were uploaded';
        
   // Check for standard uploading errors
   foreach ($active_keys as $key)
   {
      if ($_FILES[$key]['error'] > 0)
         return $_FILES[$key]['tmp_name'] . ': ' . $errors[$_FILES[$key]['error']];
   }
    
   // See if the file we are working on really was an HTTP upload
   foreach ($active_keys as $key)
   {
      if (!is_uploaded_file($_FILES[$key]['tmp_name']))
         return $_FILES[$key]['tmp_name'] . ' not an HTTP upload';
   }
    
   // Make sure the image uploaded appears to be an actual image
   foreach ($active_keys as $key)
   {
      if (!getimagesize($_FILES[$key]['tmp_name']))
         return $_FILES[$key]['tmp_name'].' is not an image';
   }
       

   // Save every uploaded file to the same filename (normally we'd want to
   // save each file with its own unique name, but we are assuming there
   // is only one file).
   foreach ($active_keys as $key)
   {
      if (!move_uploaded_file($_FILES[$key]['tmp_name'], $image_filename))
         return 'receiving directory (' . $image_filename . ') has insufficient permission';
   }
    
   // If you got this far, everything has worked and the file has been successfully saved.

   return '';
}  

Nutch, Sitemaps, and Google's findings

My search engine class is winding down, but our final project is to implement a Sitemap Protocol parser for Nutch, a popular open-source search engine. I mentioned a while back that Nutch is not for wimps... my students would certainly vouch for the huge learning curve to making code modifications. I've even had to scale back how much work my students do because of the complexity of changes required. I'm going to do the difficult part of integrating their code with the innards of Nutch sometime in the next few weeks.

The reason I mention our Sitemap project is that WWW 2009 is meeting in Madrid this week, and a paper entitled Sitemaps: Above and Beyond the Crawl of Duty is being presented today by Uri Schonfeld (UCLA) and Narayanan Shivakumar (Google). This is the first paper to report on widespread usage of Sitemaps in the Web using Google's crawling history.

Schonfeld & Shivakumar report that Sitemaps were used by approximately 35 million websites in late 2008, exposing several billion URLs. 58% of the URLs included last modification dates, 7% included change frequency, and 61% a priority. About 76.8% of Sitemaps used XML formatting, and only 3.4% used plain text. Interestingly, 17.5% of Sitemaps are formatted incorrectly.

The figure below represents how many URLs Google discovered via Sitemaps (red) vs. regular crawling (green) for cnn.com. Notice that on any given day, more URLs could normally be discovered via Sitemaps.



Another interesting figure (below) shows when a URL was discovered via Sitemaps vs. regular web crawling for cnn.com. In most cases URLs were discovered at the same rate, but there are a number of them (dots below the line) that were discovered via Sitemaps much earlier than web crawling.


CNN's website is not typical. Schonfeld & Shivakumar report that in a dataset of 5 billion+ URLs, 78% were discovered via Sitemaps first compared to 22% via web crawling.

The paper also describes an algorithm that can be used by search engines to prioritize URLs discovered via web crawling and Sitemaps as well. I've covered the high-lights, but I recommend you read the paper if you're interested in some of the finer details.