Kinect SDK and XNA

It seems many people are looking for the way to draw skeletons and store the RGB camera data into Texture2Ds. The KinectEngine class which is included in Neat game engine does all these stuff, but here’s the direct way to do these in XNA:

1. Initialization

First we have to initialize the Nui Runtime. Assuming we are going to use the first available Kinect device connected to our system, we can write:

public Runtime Nui = Runtime.Kinects[0];
Nui.Initialize(RuntimeOptions.UseSkeletalTracking | RuntimeOptions.UseColor);

The Runtime class is inside the Microsoft.Research.Kinect.Nui namespace. The second line tells the SDK that we are going to use both skeletal tracking and RGB (color) camera.

Now that we initialized the SDK, we have to tell it to do what we want when a video frame or a skeleton frame is ready:

Nui.VideoFrameReady += 
   new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady);
Nui.SkeletonFrameReady += 
   new EventHandler<SkeletonFrameReadyEventArgs>(nui_SkeletonFrameReady);

2. RGB Stream

One of the most common things that XNA developers want to do with Kinect SDK, is to show the video captured by the RGB camera in their game. To do this, first you have to store the stream into a Texture2D object. Therefore, before we begin, we have to create our target texture:

public Texture2D KinectRGB =  new Texture2D(GraphicsDevice, xMax, yMax);

Now, we can perform the conversion when video frames become ready:

void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
    PlanarImage image = e.ImageFrame.Image;

    int offset = 0;
    Color[] bitmap = new Color[xMax * yMax];
    for (int y = 0; y < yMax; y++)
        for (int x = 0; x < xMax; x++)
        {
            bitmap[y * xMax + x] = new Color(image.Bits[offset + 2], 
                image.Bits[offset + 1], image.Bits[offset], 255);
            offset += 4;
        }
    KinectRGB.SetData(bitmap);
}

The raw image data is stored in e.ImageFrame.Image. The usual method of filling XNA textures is by creating a Color array, fill it with pixel data and feed it to a texture using its SetData method. Color data is stored in BGR32 format, meaning that the value for blue channel is the first value stored, and the red value is the last. For more information about reading the camera data, watch this video.

To begin reading from the RGB camera, we have to open the video stream:

Nui.VideoStream.Open(ImageStreamType.Video, 2,
    ImageResolution.Resolution640x480, ImageType.Color);

In this example, the value of xMax is 640 and yMax is 480.

3. Skeletal Tracking

Getting skeleton data from Kinect SDK is easy. This example stores the data into an array for further use in the game:

void nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    var trackedSkeletons = from s in e.SkeletonFrame.Skeletons
           where s.TrackingState == SkeletonTrackingState.Tracked
           select s;
    trackedSkeletonsCount = trackedSkeletons.Count();
    for (int i = 0; i < trackedSkeletonsCount; i++)
        Skeletons[i] = trackedSkeletons.ElementAt(i);
}

To draw a skeleton, you just have to get each joint’s position and draw a line between adjacent joints.

shot0006

This helper function draws a skeleton using Neat engine’s LineBrush class.

public void DrawSkeleton(SpriteBatch spriteBatch, LineBrush lb, Vector2 position, 
Vector2 size, Color color, int skeletonId = 0) { if (Skeletons.Length <= skeletonId || Skeletons[skeletonId] == null) { //Skeleton not found. Draw an X lb.Draw(spriteBatch, position, position + size, color); lb.Draw(spriteBatch, new LineSegment(position.X+size.X, position.Y,
position.X, position.Y + size.Y), color); return; } //Right Hand lb.Draw(spriteBatch, ToVector2(JointID.HandRight, size, skeletonId), ToVector2(JointID.WristRight, size, skeletonId), color, position); lb.Draw(spriteBatch, ToVector2(JointID.WristRight, size, skeletonId), ToVector2(JointID.ElbowRight, size, skeletonId), color, position); lb.Draw(spriteBatch, ToVector2(JointID.ElbowRight, size, skeletonId), ToVector2(JointID.ShoulderRight, size, skeletonId), color, position); //Head & Shoulders lb.Draw(spriteBatch, ToVector2(JointID.ShoulderRight, size, skeletonId), ToVector2(JointID.ShoulderCenter, size, skeletonId), color, position); lb.Draw(spriteBatch, ToVector2(JointID.Head, size, skeletonId), ToVector2(JointID.ShoulderCenter, size, skeletonId), color, position); lb.Draw(spriteBatch, ToVector2(JointID.ShoulderCenter, size, skeletonId), ToVector2(JointID.ShoulderLeft, size, skeletonId), color, position); //Left Hand lb.Draw(spriteBatch, ToVector2(JointID.HandLeft, size, skeletonId), ToVector2(JointID.WristLeft, size, skeletonId), color, position); lb.Draw(spriteBatch, ToVector2(JointID.WristLeft, size, skeletonId), ToVector2(JointID.ElbowLeft, size, skeletonId), color, position); lb.Draw(spriteBatch, ToVector2(JointID.ElbowLeft, size, skeletonId), ToVector2(JointID.ShoulderLeft, size, skeletonId), color, position); //Hips & Spine lb.Draw(spriteBatch, ToVector2(JointID.HipLeft, size, skeletonId), ToVector2(JointID.HipCenter, size, skeletonId), color, position); lb.Draw(spriteBatch, ToVector2(JointID.HipRight, size, skeletonId), ToVector2(JointID.HipCenter, size, skeletonId), color, position); lb.Draw(spriteBatch, ToVector2(JointID.Spine, size, skeletonId), ToVector2(JointID.HipCenter, size, skeletonId), color, position); lb.Draw(spriteBatch, ToVector2(JointID.Spine, size, skeletonId), ToVector2(JointID.ShoulderCenter, size, skeletonId), color, position); //Left foot lb.Draw(spriteBatch, ToVector2(JointID.HipLeft, size, skeletonId), ToVector2(JointID.KneeLeft, size, skeletonId), color, position); lb.Draw(spriteBatch, ToVector2(JointID.KneeLeft, size, skeletonId), ToVector2(JointID.AnkleLeft, size, skeletonId), color, position); lb.Draw(spriteBatch, ToVector2(JointID.AnkleLeft, size, skeletonId), ToVector2(JointID.FootLeft, size, skeletonId), color, position); //Right foot lb.Draw(spriteBatch, ToVector2(JointID.HipRight, size, skeletonId), ToVector2(JointID.KneeRight, size, skeletonId), color, position); lb.Draw(spriteBatch, ToVector2(JointID.KneeRight, size, skeletonId), ToVector2(JointID.AnkleRight, size, skeletonId), color, position); lb.Draw(spriteBatch, ToVector2(JointID.AnkleRight, size, skeletonId), ToVector2(JointID.FootRight, size, skeletonId), color, position); }

Be sure to read the source code of Neat’s KinectEngine class for more information.

5 thoughts on “Kinect SDK and XNA

  1. Let me save you a bunch of CPU cycles …
    Instead of copying the RGB stream’s bytes into a Color array then use it to set the Texture’s Data.
    You can initialize the Texture2D using it’s first overload and not specify a SurfaceFormat, then get the bytes from the RGB stream and directly use them to set the Texture’s Data.
    The only problem is that you’ll have the color channels swapped, but this can be fixed easily using a very simple HLSL shader (custom effect).

    So what I did here is that instead of swapping the color channels on the CPU, I’m doing it on the GPU which is much much faster 🙂

    Check this link for details: http://social.msdn.microsoft.com/Forums/en-US/kinectsdk/thread/56f4293a-47b7-4ca0-bf5c-3c03ca2c4ee9

    1. Right! We can do a return float4(color.b,color.r,color.g,color.a); instead.

      I’ll need to update all my Kinect related posts anyway, because the new version of Kinect SDK has changed significantly 😉

      Thanks.

      1. yeah, and you can also just write:
        return color.bgra;
        which is the same thing but shorter and looks cooler 😀

        v1.0 of the SDK renamed A LOT of stuff, but now the naming is much more logical and makes your code more readable, other changes include changing the depth data from a byte array to a short array (no more bit manipulations!), they also force you now to copy the frame data into your own array.
        Just take care from one thing though, it says in the change notes that they renamed “JointID” to “Joint”, but that’s wrong, they renamed it to “JointType”.

        It took me about 3 hours to port my app from Kinect SDK Beta 2 to Kinect SDK v1.0, but I think it was worth it 🙂

Leave a Reply