Wednesday, December 17, 2014

XNA and Kinect 2 hand motion demo

This demo will show you how to write a simple XNA application that reads hand motion from the Kinect v2. The Kinect sensor can detect motion for your entire body, but here I'll focus on just detecting hand motion and whether the hand is open (all fingers out) or closed (in a fist) as shown in the screenshot below.


Prerequisites

You must have the Kinect for Windows 2 correctly installed along with the SDK. There are plenty of online tutorials showing you how to program with the older Kinect; this is for the latest version.

See the Prerequisites section from my previous post on installing the necessary software to code this demo using Visual Studio 2013.


Create an XNA Project

First create an XNA project by selecting FileProject... from the menu. Then select XNA Game Studio 4.0 template under Visual C# and select Windows Game (2.0). Name the project KinectMotionDemo.


Add Kinect Reference

Right-click the project in the Solution Explorer and from the context menu select AddReference.... Type kinect in the dialog box's search box, and check the Microsoft.Kinect reference. Then press OK. You should now see Microsoft.Kinect among the project's References in the Solution Explorer.


Initialization

Add the Kinect namespace to your Game1.cs file:

using Microsoft.Kinect;

Inside the KinectMotionDemo namespace and immediately after the Game1 class, create a class that represents a Hand. We will keep track of whether the hand is left or right, open or closed, and its location to be displayed on the screen.

class Hand
{         
 // HandLeft or HandRight
 public JointType Type { get; set; }
 
 // Open or closed
 public HandState HandState { get; set; }

 // Screen location of hand
 public Vector2 ScreenPosition { get; set; }
}


Add some class-level variables which will be needed elsewhere:

// Active Kinect sensor
private KinectSensor kinectSensor;

// Body frame reader
private BodyFrameReader bodyFrameReader;

// Array for the bodies and hands
private Body[] bodies;
private Hand[] leftHands;
private Hand[] rightHands;

// Sprites used to display hands
private Texture2D leftHandOpenSprite;
private Texture2D leftHandClosedSprite;
private Texture2D rightHandOpenSprite;
private Texture2D rightHandClosedSprite;
Note that a Microsoft.Kinect.Body represents a person's body, and Kinect can track up to six people at the same time.

Add some code in the Initialize method to initialize the sensor, create arrays large enough to track up to six people, and add an event listener so we'll know when body sensor data is available.

protected override void Initialize()
{
 // Allow mouse to be visible when on top of the window
 IsMouseVisible = true;

 // One sensor is currently supported
 kinectSensor = KinectSensor.GetDefault();                                 

 // Determine how many bodies and hands can be tracked
 int totalBodies = kinectSensor.BodyFrameSource.BodyCount;
 bodies = new Body[totalBodies];
 leftHands = new Hand[totalBodies];
 rightHands = new Hand[totalBodies];

 // Open the reader for the body frames
 bodyFrameReader = kinectSensor.BodyFrameSource.OpenReader();

 // Specify handler for frame arrival
 bodyFrameReader.FrameArrived += this.Reader_BodyFrameArrived;
      
 // Open the sensor
 kinectSensor.Open();

 base.Initialize();
}


Override the Game class's OnExiting method to free up Kinect sensor when the game window is being closed.

protected override void OnExiting(object sender, EventArgs args)
{
 if (bodyFrameReader != null)
 {
  bodyFrameReader.Dispose();
  bodyFrameReader = null;
 }

 if (kinectSensor != null)
 {
  kinectSensor.Close();
  kinectSensor = null;
 }

 base.OnExiting(sender, args);
}

Where are the hands?

Now write the event listener for the body frame reader that will obtain the sensor data for the left and right hands of all the bodies that are being tracked. This method will call UpdateHandInfo, a method that will determine if the hand is open or closed and determine where it is located in depth space so it can be accurately mapped to an (x,y) location on the screen.

private void Reader_BodyFrameArrived(object sender, 
    BodyFrameArrivedEventArgs e)
{
 bool dataReceived = false;

 // Load captured body data into the array of bodies
 using (BodyFrame bodyFrame = e.FrameReference.AcquireFrame())
 {
  if (bodyFrame != null)
  {
   bodyFrame.GetAndRefreshBodyData(bodies);
   dataReceived = true;
  }
 }

 if (dataReceived)
 {
  // Iterate through each body
  for (int i = 0; i < bodies.Length; i++)
  {
   Body body = bodies[i];
   if (body.IsTracked)
   {                        
    // See if hands need to be instantiated
    if (leftHands[i] == null)
     leftHands[i] = new Hand { Type = JointType.HandLeft };
    if (rightHands[i] == null)
     rightHands[i] = new Hand { Type = JointType.HandRight };

    // Get hand sensor data
    UpdateHandInfo(leftHands[i], body);
    UpdateHandInfo(rightHands[i], body);        
   }
  }
 }
}

private void UpdateHandInfo(Hand hand, Body body)
{
 if (hand.Type == JointType.HandLeft)
  hand.HandState = body.HandLeftState;
 else
  hand.HandState = body.HandRightState;

 // Map joint position to depth space
 CameraSpacePoint position = body.Joints[hand.Type].Position;
 DepthSpacePoint depthSpacePoint = kinectSensor.CoordinateMapper.MapCameraPointToDepthSpace(position);
 hand.ScreenPosition = new Vector2(depthSpacePoint.X, depthSpacePoint.Y);
}

Displaying the Hands

Find four different images that you would like to display for your left and right hands when they are open or closed. I used images that look a lot like hands, but you can be more creative. Make sure that you use only letters, numbers, and underscores in your filenames because these will be converted into variable names!

Add the images to the KinectMotionDemoContent project in the Solution Explorer by right-clicking on the KinectMotionDemoContent project and selecting Add → Existing Item.... An open dialog box will appear. Select the four PNG images you want to use and press OK. You should now see all four images in your KinectMotionDemoContent project as pictured below.


Now load the PNG images in the LoadContent method.
protected override void LoadContent()
{
 // Create a new SpriteBatch, which can be used to draw textures.
 spriteBatch = new SpriteBatch(GraphicsDevice);

 leftHandOpenSprite = 
    Content.Load<Texture2D>("openhand_left");
 leftHandClosedSprite = 
    Content.Load<Texture2D>("closedhand_left");
 rightHandOpenSprite = 
    Content.Load<Texture2D>("openhand_right");
 rightHandClosedSprite = 
    Content.Load<Texture2D>("closedhand_right");
}

The hands will be displayed in the Draw method.
protected override void Draw(GameTime gameTime)
{
 GraphicsDevice.Clear(Color.CornflowerBlue);

 spriteBatch.Begin();            
 
 // Draw all left hands
 foreach (Hand hand in leftHands)
 {
  if (hand != null)
  {
   if (hand.HandState == HandState.Closed)
    spriteBatch.Draw(leftHandClosedSprite, 
     hand.ScreenPosition, Color.White);
   else
    spriteBatch.Draw(leftHandOpenSprite, 
     hand.ScreenPosition, Color.White);                   
  }
 }

 // Draw all right hands
 foreach (Hand hand in rightHands)
 {
  if (hand != null)
  {
   if (hand.HandState == HandState.Closed)
    spriteBatch.Draw(rightHandClosedSprite, 
     hand.ScreenPosition, Color.White);
   else
    spriteBatch.Draw(rightHandOpenSprite, 
     hand.ScreenPosition, Color.White);
  }
 }

 spriteBatch.End();

 base.Draw(gameTime);
}


Press Ctrl-F5 to build and run the program. Stand in front of your Kinect, and you should see the PNG images move as you move your hands. Try opening and closing your hands to see the open/close images being displayed. If you have a friend nearby, ask them to join you so you can see four hands moving about the screen.


Problems?

When I first tried to build and run my program, I got the following error message:

The primary reference "Microsoft.Kinect, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL" could not be resolved because it was built against the ".NETFramework,Version=v4.5" framework. This is a higher version than the currently targeted framework ".NETFramework,Version=b4.0".
To fix this problem, I closed the project in Visual Studio and opened the project's .csproj file in a text editor and changed the following line:
<TargetFrameworkVersion>v4.0</TargetFrameworkVersion>
to
<TargetFrameworkVersion>v4.5</TargetFrameworkVersion>
Then I re-opened the project in Visual Studio and re-built the application with no problems.

Thursday, December 11, 2014

Hour of Code at Kensett Elementary

Eight students from our Computer Science department met with about 50 fifth graders at Kensett Elementary on Friday, Dec 5, to participate in the Hour of Code. As you can see from the pictures, the kids were having a blast! Most of them ran through the Angry Birds tutorial, and a few tried the Frozen and Flappy Bird tutorials. When the hour was over, many of them got really excited when I told them they could keep working on the tutorials when they got home... just open a web browser and go to code.org!


Wednesday, November 26, 2014

Kinect for Windows 2 and XNA demo

Over the Thanksgiving break I managed to get my new Kinect for Windows 2 working with XNA. I couldn't find any code samples online using XNA with the new Kinect SDK, so hopefully this will help out others who are trying to do something similar.

This demo will show how to display the Kinect's video feed which is provided by its 1080p color camera. This is similar to the Color Basics-WPF C# Sample provided in the Kinect SDK 2 except that it is tailored for XNA.


Prerequisites

You must have the Kinect for Windows 2 correctly installed along with the SDK. There are plenty of online tutorials showing you how to program with the older Kinect; this is for the latest version.

Microsoft is no longer maintaining XNA, but but you can still use it in Visual Studio 2013 as long as you install the following:

  1. Install Visual Studio 2010. Any edition will work.
  2. Install Games for Windows Marketplace Client
  3. Install XNA Game Studio 4.0
  4. Install XNA 4.0 Refresh for VS 2013. After you have downloaded the zip file, extract it and run XNA Game Studio 4.0.vsix

If you do not have access to Visual Studio 2010, you might try MonoGame. Just be aware that it does not have a content pipeline converter (software that converts content like sound files into xnb files). I have not tried MonoGame with Kinect, but I don't see any reason why it wouldn't work.


Create an XNA Project

First create an XNA project by selecting FileProject... from the menu. Then select XNA Game Studio 4.0 template under Visual C# and select Windows Game (2.0). Name the project KinectVideoXna.


Add Kinect Reference

Right-click the project in the Solution Explorer and from the context menu select AddReference.... Type kinect in the dialog box's search box, and check the Microsoft.Kinect reference. Then press OK. You should now see Microsoft.Kinect among the project's References in the Solution Explorer.


Adding Code

Add the Kinect namespace:

using Microsoft.Kinect;

Add some class-level variables:

// Texture to draw
Texture2D videoTexture;

// Active Kinect sensor
private KinectSensor kinectSensor;

// Reader for color frames
private ColorFrameReader colorFrameReader;
        
// Intermediate storage for receiving frame data from the sensor
private byte[] colorPixels;

Initialize the sensor and the data structures used for capturing data from the sensors:

protected override void Initialize()
{
 kinectSensor = KinectSensor.GetDefault();

 // Open the reader for the color frames
 colorFrameReader = 
  kinectSensor.ColorFrameSource.OpenReader();

 // Specify a handler for frame arrival
 colorFrameReader.FrameArrived += Reader_ColorFrameArrived;

 // Create the ColorFrameDescription using rgba format
 FrameDescription desc = kinectSensor.ColorFrameSource.
  CreateFrameDescription(ColorImageFormat.Rgba);
 
 // Allocate space to put the pixels to be rendered
 colorPixels = new byte[desc.Width * desc.Height * 
  desc.BytesPerPixel];

 // Open the sensor
 kinectSensor.Open();

 // Create texture large enough to hold the color frame
 videoTexture = new Texture2D(graphics.GraphicsDevice, 
  desc.Width, desc.Height);

 base.Initialize();
}

Also override the OnExiting method to free up the ColorFrameReder and Kinect sensor when the game exists:

protected override void OnExiting(object sender, EventArgs args)
{
 if (colorFrameReader != null)
 {
  colorFrameReader.Dispose();
  colorFrameReader = null;
 }

 if (kinectSensor != null)
 {
  kinectSensor.Close();
  kinectSensor = null;
 }

 base.OnExiting(sender, args);
}

Create the handler for the color photo sensor where we'll store the captured photo into videoTexture:

private void Reader_ColorFrameArrived(object sender, ColorFrameArrivedEventArgs e)
{          
 // ColorFrame is IDisposable
 using (ColorFrame colorFrame = e.FrameReference.AcquireFrame())
 {
  if (colorFrame != null)
  {
   // Copy color frame into the array
   colorFrame.CopyConvertedFrameDataToArray(
    colorPixels, 
    ColorImageFormat.Rgba);                   

   // Avoid exception when SetData method is used
   GraphicsDevice.Textures[0] = null;

   // Put pixel data into a texture
   videoTexture.SetData(colorPixels);
  }
 }
}

Finally, draw the videoTexture containing the color photo to the screen:

protected override void Draw(GameTime gameTime)
{
 GraphicsDevice.Clear(Color.CornflowerBlue);

 if (videoTexture != null)
 {
  // Draw color video
  spriteBatch.Begin();
  spriteBatch.Draw(videoTexture, new Rectangle(0, 0, 
   graphics.GraphicsDevice.Viewport.Width,
   graphics.GraphicsDevice.Viewport.Height), 
   Color.White);
  spriteBatch.End();
 }

 base.Draw(gameTime);
}

Press Ctrl-F5 to build and run the program. You should see color video of whatever your Kinect is pointed at.


Problems?

When I first tried to build and run my program, I got the following error message:

The primary reference "Microsoft.Kinect, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL" could not be resolved because it was built against the ".NETFramework,Version=v4.5" framework. This is a higher version than the currently targeted framework ".NETFramework,Version=b4.0".
To fix this problem, I closed the project in Visual Studio and opened the project's .csproj file in a text editor and changed the following line:
<TargetFrameworkVersion>v4.0</TargetFrameworkVersion>
to
<TargetFrameworkVersion>v4.5</TargetFrameworkVersion>
Then I re-opened the project in Visual Studio and re-built the application with no problems.

Friday, October 10, 2014

Teaching an upper-level Web Development course

We're half way through the fall semester, and the spring semester will be here before we know it. Every spring I teach an upper-level Web Development 2 elective which builds on the Web Dev 1 course that I teach every semester. The Web Dev 1 course covers the fundamentals: HTTP, HTML, CSS, JavaScript, Ajax, and PHP.

The goal of Web Dev 2 is to give students more breadth and depth in web development, focusing on both client and server-side technologies. I try to choose technologies that are widely used so my students will be more marketable after graduation, and in the past I've taught a variety of technologies: advanced JavaScript, jQuery, Java servlets and JSPs, ASP.NET Web Forms, ASP.NET MVC, and creating web services with ASP.NET and JAX-RS. Of course you can't cover everything in 16 weeks, so I have to put a lot of thought into what can be adequately covered in a limited amount of time.

This summer I did some development work at Flatirons Solutions using AngularJS and web services with Spring. It got me wondering if my Web Dev 2 course should focus more on JavaScript frameworks. One of my developer friends this summer tried to convince me that JavaScript was also taking over the server and that I should be teaching Node.js instead of Java.

It can be a struggle for computing professors to ensure our courses remain relevant when technologies are always changing. Fortunately I've got a lot of Facebook friends who are developers and Harding CS alumni, so I decided to ask their opinion:

If you were in college and could take an advanced Web Development course, what topics would you like it to cover?
This elicited quite a few responses which I'll summarize here:

  1. JavaScript frameworks like AngularJS, Ember, Knockout
  2. ASP.NET MVC and Web API
  3. Node.js
  4. Python and Django
  5. Ruby on Rails
  6. LESS
  7. Web services
  8. Web security
  9. Single Page Applications (SPAs)
  10. Automated web testing, A/B testing, UI testing
  11. Understanding HTTP
  12. Picking the right data store (relational, NoSQL, Map/Reduce, etc.)
  13. Caching and latency issues
  14. Teams that implement the same project on different platforms
  15. Git
  16. Web architecture - focusing less on development and more on architecture

There may have been a few things I left off, but these received the most mentions. (Thank you for your input, guys!) I'm still making up my mind, but this is what I'm leaning toward teaching this spring:

  1. JavaScript - The more JavaScript you are exposed to, the better. I'd like my students to be familiar with a number of advanced JavaScript features and some popular libraries like Underscore.js.
  2. jQuery- Everyone needs to know the most popular JavaScript library in use.
  3. Node.js and Express - Although I have little experience with Node, I think it would be helpful for my students to apply their JavaScript skills on the server as well as the client. I will probably tie in MongoDB which will give them experience with a NoSQL database. We will likely create a web service and consume it with an app written with...
  4. AngularJS - Very popular JavaScript framework with strong job demand. Plus I got some experience with it this summer. We will write unit tests using Jasmine and use Karma for integration testing.

My students will work in pairs on their projects and use GitHub to house their code like they do in my GUI course. We'll use the WebStorm IDE which I came to love this summer. I'll inject other topics like security where they are applicable.

A weakness of this plan is that my students will not get to use Java or ASP.NET which are very popular server-side technologies. However, they will leave this class fluent in JavaScript.

Now... need to find time in my schedule to learn Node.js, prepare all new quizzes, assignments, and projects, and also prepare for my Game Programming course.

Friday, August 01, 2014

Last day at Flatirons

Today is my last day at Flatirons Solutions. I've really enjoyed these eleven weeks and learned a ton. My web development courses, in particular, will definitely benefit from the exposure I received to some new technologies that I'll be integrating into those courses. I'm very thankful to George and Paula for giving me the opportunity to spend my summer here, and I hope that I made some major contributions.

Tuesday, June 24, 2014

Half way through our Colorado summer

We're now entering our sixth week living in Boulder. My family is getting more accustomed to apartment living. It's ironic though how we thought we were escaping the Arkansas heat only to live in an apartment lacking air conditioning! At least the temperature drops each night.

We had a little scare a few weeks ago. Becky had developed a large lump on her thyroid, and after meeting with a surgeon, she was strongly encouraged to have it removed in case it was cancerous. My sister (who is now eligible for sainthood) flew out here from Chicago so she could watch the kids while Becky recovered from surgery. The surgery went well, and praise God the lump was not cancerous! Becky is still healing from the surgery, but she is doing really well. The kids were pretty oblivious to the whole thing since Aunt Sass kept them busy with swimming and museums and making pizza. Have I mentioned how awesome my sister is?

Since the surgery, Becky and the kids have been occupying themselves with all kinds of activities like tennis lessons, swimming lessons, and now karate classes while I toil away at Flatirons each day. wink When I arrive home in the evenings we often eat and go out to a park or discover some new part of Boulder. On the weekends we have visited Red Rocks, Estes Park, and a few other places. In the photo below we hiked around Bear Lake at the Rocky Mountain National Park. As you can tell from our Chacos, we were not expecting to see snow!

We've gotten to see a number of old Colorado friends which has made our time out here very meaningful. Some of our best friends from Searcy also stopped by to visit for a couple of days, and we were able to dine at Casa Bonita and see a Rockies game with them.

Next week my brother and his wife will be visiting from Texas, and some more good friends from Searcy will be coming up July 4th weekend. Becky has some college friends coming up to visit the last week of July. Lots to look forward to!

I'll conclude with a note of thankfulness for the Boulder Valley Church of Christ. They have really taken us in, and we are so thankful for the many ways they supported us during Becky's surgery. One of the ladies watched the kids for us before Aunt Sara arrived, and one of the elders came to the hospital to pray with us immediately before the surgery. These are people who truly love the Lord.

I'm also thankful for the incredible VBS they put on. You would not believe how much effort they expended to entertain and teach about 50 children about God's love for a full week. Ethan and Braden absolutely loved going to VBS each day. I think it's really great that the body of Christ can be found nearly everywhere you go.

Sunday, June 01, 2014

Week 2: Finally making some contributions

Last week at Flatirons was admittedly difficult. I really like to learn new stuff, but if I'm not contributing much and all I'm doing is trying to take in lots of information, time can pass by very slowly.

This week I finally was able to write some code and fix some lingering bugs in our web application. We are using AngularJS which is new to me, and our code base is quite large, so I was a little on the slow side. By the end of the week I felt like I finally had a good idea how the application was designed and where to go to modify the app's functionality. My speed should start improving.

It really felt good to write code again. I really enjoy teaching, but I seldom have time to contribute to software that others are using. It's amazing to me how quickly time goes when I am programming and getting things to work right. Each bug fix makes me say "YES!" inside (and sometimes outside!).