This demo will show you how to write a simple XNA application that reads hand motion from the Kinect v2. The Kinect sensor can detect motion for your entire body, but here I'll focus on just detecting hand motion and whether the hand is open (all fingers out) or closed (in a fist) as shown in the screenshot below.
Prerequisites
You must have the Kinect for Windows 2 correctly installed along with the SDK. There are plenty of online tutorials showing you how to program with the older Kinect; this is for the latest version.
See the Prerequisites section from my previous post on installing the necessary software to code this demo using Visual Studio 2013.
Create an XNA Project
First create an XNA project by selecting File → Project... from the menu. Then select XNA Game Studio 4.0 template under Visual C# and select Windows Game (2.0). Name the project KinectMotionDemo.
Add Kinect Reference
Right-click the project in the Solution Explorer and from the context menu select Add → Reference.... Type kinect in the dialog box's search box, and check the Microsoft.Kinect reference. Then press OK. You should now see Microsoft.Kinect among the project's References in the Solution Explorer.
Initialization
Add the Kinect namespace to your Game1.cs file:
using Microsoft.Kinect;
Inside the KinectMotionDemo namespace and immediately after the Game1 class, create a class that represents a Hand. We will keep track of whether the hand is left or right, open or closed, and its location to be displayed on the screen.
class Hand
{
// HandLeft or HandRight
public JointType Type { get; set; }
// Open or closed
public HandState HandState { get; set; }
// Screen location of hand
public Vector2 ScreenPosition { get; set; }
}
Add some class-level variables which will be needed elsewhere:
// Active Kinect sensor private KinectSensor kinectSensor; // Body frame reader private BodyFrameReader bodyFrameReader; // Array for the bodies and hands private Body[] bodies; private Hand[] leftHands; private Hand[] rightHands; // Sprites used to display hands private Texture2D leftHandOpenSprite; private Texture2D leftHandClosedSprite; private Texture2D rightHandOpenSprite; private Texture2D rightHandClosedSprite;Note that a
Microsoft.Kinect.Body represents a person's body, and Kinect can track up to six people at the same time.
Add some code in the Initialize method to initialize the sensor, create arrays large enough to track up to six people, and add an event listener so we'll know when body sensor data is available.
protected override void Initialize()
{
// Allow mouse to be visible when on top of the window
IsMouseVisible = true;
// One sensor is currently supported
kinectSensor = KinectSensor.GetDefault();
// Determine how many bodies and hands can be tracked
int totalBodies = kinectSensor.BodyFrameSource.BodyCount;
bodies = new Body[totalBodies];
leftHands = new Hand[totalBodies];
rightHands = new Hand[totalBodies];
// Open the reader for the body frames
bodyFrameReader = kinectSensor.BodyFrameSource.OpenReader();
// Specify handler for frame arrival
bodyFrameReader.FrameArrived += this.Reader_BodyFrameArrived;
// Open the sensor
kinectSensor.Open();
base.Initialize();
}
Override the Game class's OnExiting method to free up Kinect sensor when the game window is being closed.
protected override void OnExiting(object sender, EventArgs args)
{
if (bodyFrameReader != null)
{
bodyFrameReader.Dispose();
bodyFrameReader = null;
}
if (kinectSensor != null)
{
kinectSensor.Close();
kinectSensor = null;
}
base.OnExiting(sender, args);
}
Where are the hands?
Now write the event listener for the body frame reader that will obtain the sensor data for the left and right hands of all the bodies that are being tracked. This method will call UpdateHandInfo, a method that will determine if the hand is open or closed and determine where it is located in depth space so it can be accurately mapped to an (x,y) location on the screen.
private void Reader_BodyFrameArrived(object sender,
BodyFrameArrivedEventArgs e)
{
bool dataReceived = false;
// Load captured body data into the array of bodies
using (BodyFrame bodyFrame = e.FrameReference.AcquireFrame())
{
if (bodyFrame != null)
{
bodyFrame.GetAndRefreshBodyData(bodies);
dataReceived = true;
}
}
if (dataReceived)
{
// Iterate through each body
for (int i = 0; i < bodies.Length; i++)
{
Body body = bodies[i];
if (body.IsTracked)
{
// See if hands need to be instantiated
if (leftHands[i] == null)
leftHands[i] = new Hand { Type = JointType.HandLeft };
if (rightHands[i] == null)
rightHands[i] = new Hand { Type = JointType.HandRight };
// Get hand sensor data
UpdateHandInfo(leftHands[i], body);
UpdateHandInfo(rightHands[i], body);
}
}
}
}
private void UpdateHandInfo(Hand hand, Body body)
{
if (hand.Type == JointType.HandLeft)
hand.HandState = body.HandLeftState;
else
hand.HandState = body.HandRightState;
// Map joint position to depth space
CameraSpacePoint position = body.Joints[hand.Type].Position;
DepthSpacePoint depthSpacePoint = kinectSensor.CoordinateMapper.MapCameraPointToDepthSpace(position);
hand.ScreenPosition = new Vector2(depthSpacePoint.X, depthSpacePoint.Y);
}
Displaying the Hands
Find four different images that you would like to display for your left and right hands when they are open or closed. I used images that look a lot like hands, but you can be more creative. Make sure that you use only letters, numbers, and underscores in your filenames because these will be converted into variable names!
Add the images to the KinectMotionDemoContent project in the Solution Explorer by right-clicking on the KinectMotionDemoContent project and selecting Add → Existing Item.... An open dialog box will appear. Select the four PNG images you want to use and press OK. You should now see all four images in your KinectMotionDemoContent project as pictured below.
Now load the PNG images in the
LoadContent method.
protected override void LoadContent()
{
// Create a new SpriteBatch, which can be used to draw textures.
spriteBatch = new SpriteBatch(GraphicsDevice);
leftHandOpenSprite =
Content.Load<Texture2D>("openhand_left");
leftHandClosedSprite =
Content.Load<Texture2D>("closedhand_left");
rightHandOpenSprite =
Content.Load<Texture2D>("openhand_right");
rightHandClosedSprite =
Content.Load<Texture2D>("closedhand_right");
}
The hands will be displayed in the
Draw method.
protected override void Draw(GameTime gameTime)
{
GraphicsDevice.Clear(Color.CornflowerBlue);
spriteBatch.Begin();
// Draw all left hands
foreach (Hand hand in leftHands)
{
if (hand != null)
{
if (hand.HandState == HandState.Closed)
spriteBatch.Draw(leftHandClosedSprite,
hand.ScreenPosition, Color.White);
else
spriteBatch.Draw(leftHandOpenSprite,
hand.ScreenPosition, Color.White);
}
}
// Draw all right hands
foreach (Hand hand in rightHands)
{
if (hand != null)
{
if (hand.HandState == HandState.Closed)
spriteBatch.Draw(rightHandClosedSprite,
hand.ScreenPosition, Color.White);
else
spriteBatch.Draw(rightHandOpenSprite,
hand.ScreenPosition, Color.White);
}
}
spriteBatch.End();
base.Draw(gameTime);
}
Press Ctrl-F5 to build and run the program. Stand in front of your Kinect, and you should see the PNG images move as you move your hands. Try opening and closing your hands to see the open/close images being displayed. If you have a friend nearby, ask them to join you so you can see four hands moving about the screen.
Problems?
When I first tried to build and run my program, I got the following error message:
The primary reference "Microsoft.Kinect, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35, processorArchitecture=MSIL" could not be resolved because it was built against the ".NETFramework,Version=v4.5" framework. This is a higher version than the currently targeted framework ".NETFramework,Version=b4.0".To fix this problem, I closed the project in Visual Studio and opened the project's .csproj file in a text editor and changed the following line:
<TargetFrameworkVersion>v4.0</TargetFrameworkVersion>to
<TargetFrameworkVersion>v4.5</TargetFrameworkVersion>Then I re-opened the project in Visual Studio and re-built the application with no problems.
