Lib4U

‎"Behind every stack of books there is a flood of knowledge."

Kinect Fundamentals #3: Getting distance-data from the Depth Sensor

image_thumb11

Now that you know how to use the RGB Camera data, it’s time to take a look at how you can use the depth data from the Kinect sensor.

It’s quite similar to getting data from the RGB image, but instead of RGB values, you have distance data. We will convert the distance into an image representing the depth map.

Initializing the Kinect Sensor

Not much new here since tutorial #2. In InitializeKinect() we enable the DepthStream and and listen to the DepthFrameReady event instead of ColorStream and ColorFrameReady as we did when getting the RGB image in tutorial #2.

01 private bool InitializeKinect()
02 {
03     kinectSensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30);
04     kinectSensor.DepthFrameReady += new EventHandler<DepthImageFrameReadyEventArgs>(kinectSensor_DepthFrameReady);
05
06     try
07     {
08         kinectSensor.Start();
09     }
10     catch
11     {
12         connectedStatus = "Unable to start the Kinect Sensor";
13         return false;
14     }
15     return true;
16 }

Here, we open the DepthStream and tell the sensor that we want to have depth data with the resolution of 640×480. We also create an event handler that kicks in everytime the Kinect got some data ready for us.

Converting the depth data

Now is the time for the meat of this tutorial. Here we get the depth data from the device in millimeter, and convert it into a distance we can use for displaying a black and white map of the depth. The Kinect device got a range from 0.85m to 4m (Xbox, the PC-version can see closer and further). We can use this knowledge to create a black and white image where each pixel is the distance from the camera. We might also get some unknown depth pixels if the rays are hitting a window, shadow, mirror and so on (these will have the distance of 0).

First of all, we grab the captured DepthImageFrame from the device. Then we copy this data and convert the depth frame into a 32bit format that we can use as the source for our pixels. The ConverDepthFrame function convert’s the 16-bit grayscale depth frame that the Kinect captured into a 32-bit image frame. This function was copied from the Kinect for Windows Sample that came with the SDK.

Let’s take a look at the code.

 

01 void kinectSensor_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
02 {
03     using (DepthImageFrame depthImageFrame = e.OpenDepthImageFrame())
04     {
05         if (depthImageFrame != null)
06         {
07             short[] pixelsFromFrame = new short[depthImageFrame.PixelDataLength];
08
09             depthImageFrame.CopyPixelDataTo(pixelsFromFrame);
10             byte[] convertedPixels = ConvertDepthFrame(pixelsFromFrame, ((KinectSensor)sender).DepthStream, 640 * 480 * 4);
11
12             Color[] color = new Color[depthImageFrame.Height * depthImageFrame.Width];
13             kinectRGBVideo = new Texture2D(graphics.GraphicsDevice, depthImageFrame.Width, depthImageFrame.Height);
14
15             // Set convertedPixels from the DepthImageFrame to a the datasource for our Texture2D
16             kinectRGBVideo.SetData<byte>(convertedPixels);
17         }
18     }
19 }

Notice that we didn’t manually create a Color-array as we did in the previous tutorial. You could have used this method instead of the Color-array method in tutorial #2 as well. Just wanted to show a few ways to do this just in case you need better control. ;)

And the ConvertDepthFrame function:

 

01 // Converts a 16-bit grayscale depth frame which includes player indexes into a 32-bit frame
02 // that displays different players in different colors
03 private byte[] ConvertDepthFrame(short[] depthFrame, DepthImageStream depthStream, intdepthFrame32Length)
04 {
05     int tooNearDepth = depthStream.TooNearDepth;
06     int tooFarDepth = depthStream.TooFarDepth;
07     int unknownDepth = depthStream.UnknownDepth;
08     byte[] depthFrame32 = new byte[depthFrame32Length];
09
10     for (int i16 = 0, i32 = 0; i16 < depthFrame.Length && i32 < depthFrame32.Length; i16++, i32 += 4)
11     {
12         int player = depthFrame[i16] & DepthImageFrame.PlayerIndexBitmask;
13         int realDepth = depthFrame[i16] >> DepthImageFrame.PlayerIndexBitmaskWidth;
14
15         // transform 13-bit depth information into an 8-bit intensity appropriate
16         // for display (we disregard information in most significant bit)
17         byte intensity = (byte)(~(realDepth >> 4));
18
19         if (player == 0 && realDepth == 0)
20         {
21             // white
22             depthFrame32[i32 + RedIndex] = 255;
23             depthFrame32[i32 + GreenIndex] = 255;
24             depthFrame32[i32 + BlueIndex] = 255;
25         }
26         else if (player == 0 && realDepth == tooFarDepth)
27         {
28             // dark purple
29             depthFrame32[i32 + RedIndex] = 66;
30             depthFrame32[i32 + GreenIndex] = 0;
31             depthFrame32[i32 + BlueIndex] = 66;
32         }
33         else if (player == 0 && realDepth == unknownDepth)
34         {
35             // dark brown
36             depthFrame32[i32 + RedIndex] = 66;
37             depthFrame32[i32 + GreenIndex] = 66;
38             depthFrame32[i32 + BlueIndex] = 33;
39         }
40         else
41         {
42             // tint the intensity by dividing by per-player values
43             depthFrame32[i32 + RedIndex] = (byte)(intensity >> IntensityShiftByPlayerR[player]);
44             depthFrame32[i32 + GreenIndex] = (byte)(intensity >> IntensityShiftByPlayerG[player]);
45             depthFrame32[i32 + BlueIndex] = (byte)(intensity >> IntensityShiftByPlayerB[player]);
46         }
47     }
48
49     return depthFrame32;
50 }

What this function does is to convert the 16-bit format to a usable 32-bit format. It takes the near and far depth, and also the unknown depth (mirrors, shiny surfaces and so on) and calculates the correct color based on the distance.

This function requires a few variables. You can change the function so these are defined within if you want.

 

1 // color divisors for tinting depth pixels
2 private static readonly int[] IntensityShiftByPlayerR = { 1, 2, 0, 2, 0, 0, 2, 0 };
3 private static readonly int[] IntensityShiftByPlayerG = { 1, 2, 2, 0, 2, 0, 0, 1 };
4 private static readonly int[] IntensityShiftByPlayerB = { 1, 0, 2, 2, 0, 2, 0, 2 };
5
6 private const int RedIndex = 2;
7 private const int GreenIndex = 1;
8 private const int BlueIndex = 0;

 

Rendering

Last we render the texture using a sprite batch.

01 protected override void Draw(GameTime gameTime)
02 {
03     GraphicsDevice.Clear(Color.CornflowerBlue);
04
05     spriteBatch.Begin();
06     spriteBatch.Draw(kinectRGBVideo, new Rectangle(0, 0, 640, 480), Color.White);
07     spriteBatch.Draw(overlay, new Rectangle(0, 0, 640, 480), Color.White);
08     spriteBatch.DrawString(font, connectedStatus, new Vector2(20, 80), Color.White);
09     spriteBatch.End();
10
11     base.Draw(gameTime);
12 }

The result is something similar to this:
image

Download: Source (XNA 4.0 + Kinect for Windows SDK 1.0)

The entire source can be seen below:

001 using System;
002 using System.Collections.Generic;
003 using System.Linq;
004 using Microsoft.Xna.Framework;
005 using Microsoft.Xna.Framework.Audio;
006 using Microsoft.Xna.Framework.Content;
007 using Microsoft.Xna.Framework.GamerServices;
008 using Microsoft.Xna.Framework.Graphics;
009 using Microsoft.Xna.Framework.Input;
010 using Microsoft.Xna.Framework.Media;
011 using Microsoft.Kinect;
012
013 namespace KinectFundamentals
014 {
015     /// <summary>
016     /// This is the main type for your game
017     /// </summary>
018     public class Game1 : Microsoft.Xna.Framework.Game
019     {
020         GraphicsDeviceManager graphics;
021         SpriteBatch spriteBatch;
022
023         Texture2D kinectRGBVideo;
024         Texture2D overlay;
025
026         // color divisors for tinting depth pixels
027         private static readonly int[] IntensityShiftByPlayerR = { 1, 2, 0, 2, 0, 0, 2, 0 };
028         private static readonly int[] IntensityShiftByPlayerG = { 1, 2, 2, 0, 2, 0, 0, 1 };
029         private static readonly int[] IntensityShiftByPlayerB = { 1, 0, 2, 2, 0, 2, 0, 2 };
030
031         private const int RedIndex = 2;
032         private const int GreenIndex = 1;
033         private const int BlueIndex = 0;
034         //private byte[] depthFrame32 = new byte[640 * 480 * 4];
035
036         KinectSensor kinectSensor;
037
038         SpriteFont font;
039
040         string connectedStatus = "Not connected";
041
042         public Game1()
043         {
044             graphics = new GraphicsDeviceManager(this);
045             Content.RootDirectory = "Content";
046
047             graphics.PreferredBackBufferWidth = 640;
048             graphics.PreferredBackBufferHeight = 480;
049
050         }
051
052         void KinectSensors_StatusChanged(object sender, StatusChangedEventArgs e)
053         {
054             if (this.kinectSensor == e.Sensor)
055             {
056                 if (e.Status == KinectStatus.Disconnected ||
057                     e.Status == KinectStatus.NotPowered)
058                 {
059                     this.kinectSensor = null;
060                     this.DiscoverKinectSensor();
061                 }
062             }
063         }
064
065         private bool InitializeKinect()
066         {
067             kinectSensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30);
068             kinectSensor.DepthFrameReady += new EventHandler<DepthImageFrameReadyEventArgs>(kinectSensor_DepthFrameReady);
069
070             try
071             {
072                 kinectSensor.Start();
073             }
074             catch
075             {
076                 connectedStatus = "Unable to start the Kinect Sensor";
077                 return false;
078             }
079             return true;
080         }
081
082         void kinectSensor_DepthFrameReady(object sender, DepthImageFrameReadyEventArgs e)
083         {
084             using (DepthImageFrame depthImageFrame = e.OpenDepthImageFrame())
085             {
086                 if (depthImageFrame != null)
087                 {
088                     short[] pixelsFromFrame = new short[depthImageFrame.PixelDataLength];
089
090                     depthImageFrame.CopyPixelDataTo(pixelsFromFrame);
091                     byte[] convertedPixels = ConvertDepthFrame(pixelsFromFrame, ((KinectSensor)sender).DepthStream, 640 * 480 * 4);
092
093                     Color[] color = new Color[depthImageFrame.Height * depthImageFrame.Width];
094                     kinectRGBVideo = new Texture2D(graphics.GraphicsDevice, depthImageFrame.Width, depthImageFrame.Height);
095
096                     // Set convertedPixels from the DepthImageFrame to a the datasource for our Texture2D
097                     kinectRGBVideo.SetData<byte>(convertedPixels);
098                 }
099             }
100         }
101
102         // Converts a 16-bit grayscale depth frame which includes player indexes into a 32-bit frame
103         // that displays different players in different colors
104         private byte[] ConvertDepthFrame(short[] depthFrame, DepthImageStream depthStream,int depthFrame32Length)
105         {
106             int tooNearDepth = depthStream.TooNearDepth;
107             int tooFarDepth = depthStream.TooFarDepth;
108             int unknownDepth = depthStream.UnknownDepth;
109             byte[] depthFrame32 = new byte[depthFrame32Length];
110
111             for (int i16 = 0, i32 = 0; i16 < depthFrame.Length && i32 < depthFrame32.Length; i16++, i32 += 4)
112             {
113                 int player = depthFrame[i16] & DepthImageFrame.PlayerIndexBitmask;
114                 int realDepth = depthFrame[i16] >> DepthImageFrame.PlayerIndexBitmaskWidth;
115
116                 // transform 13-bit depth information into an 8-bit intensity appropriate
117                 // for display (we disregard information in most significant bit)
118                 byte intensity = (byte)(~(realDepth >> 4));
119
120                 if (player == 0 && realDepth == 0)
121                 {
122                     // white
123                     depthFrame32[i32 + RedIndex] = 255;
124                     depthFrame32[i32 + GreenIndex] = 255;
125                     depthFrame32[i32 + BlueIndex] = 255;
126                 }
127                 else if (player == 0 && realDepth == tooFarDepth)
128                 {
129                     // dark purple
130                     depthFrame32[i32 + RedIndex] = 66;
131                     depthFrame32[i32 + GreenIndex] = 0;
132                     depthFrame32[i32 + BlueIndex] = 66;
133                 }
134                 else if (player == 0 && realDepth == unknownDepth)
135                 {
136                     // dark brown
137                     depthFrame32[i32 + RedIndex] = 66;
138                     depthFrame32[i32 + GreenIndex] = 66;
139                     depthFrame32[i32 + BlueIndex] = 33;
140                 }
141                 else
142                 {
143                     // tint the intensity by dividing by per-player values
144                     depthFrame32[i32 + RedIndex] = (byte)(intensity >> IntensityShiftByPlayerR[player]);
145                     depthFrame32[i32 + GreenIndex] = (byte)(intensity >> IntensityShiftByPlayerG[player]);
146                     depthFrame32[i32 + BlueIndex] = (byte)(intensity >> IntensityShiftByPlayerB[player]);
147                 }
148             }
149
150             return depthFrame32;
151         }
152
153         private void DiscoverKinectSensor()
154         {
155             foreach (KinectSensor sensor in KinectSensor.KinectSensors)
156             {
157                 if (sensor.Status == KinectStatus.Connected)
158                 {
159                     // Found one, set our sensor to this
160                     kinectSensor = sensor;
161                     break;
162                 }
163             }
164
165             if (this.kinectSensor == null)
166             {
167                 connectedStatus = "Found none Kinect Sensors connected to USB";
168                 return;
169             }
170
171             // You can use the kinectSensor.Status to check for status
172             // and give the user some kind of feedback
173             switch (kinectSensor.Status)
174             {
175                 case KinectStatus.Connected:
176                     {
177                         connectedStatus = "Status: Connected";
178                         break;
179                     }
180                 case KinectStatus.Disconnected:
181                     {
182                         connectedStatus = "Status: Disconnected";
183                         break;
184                     }
185                 case KinectStatus.NotPowered:
186                     {
187                         connectedStatus = "Status: Connect the power";
188                         break;
189                     }
190                 default:
191                     {
192                         connectedStatus = "Status: Error";
193                         break;
194                     }
195             }
196
197             // Init the found and connected device
198             if (kinectSensor.Status == KinectStatus.Connected)
199             {
200                 InitializeKinect();
201             }
202         }
203
204         protected override void Initialize()
205         {
206             KinectSensor.KinectSensors.StatusChanged += newEventHandler<StatusChangedEventArgs>(KinectSensors_StatusChanged);
207             DiscoverKinectSensor();
208
209             base.Initialize();
210         }
211
212         protected override void LoadContent()
213         {
214             spriteBatch = new SpriteBatch(GraphicsDevice);
215
216             kinectRGBVideo = new Texture2D(GraphicsDevice, 1337, 1337);
217
218             overlay = Content.Load<Texture2D>("overlay");
219             font = Content.Load<SpriteFont>("SpriteFont1");
220         }
221
222         protected override void UnloadContent()
223         {
224             kinectSensor.Stop();
225             kinectSensor.Dispose();
226         }
227
228         protected override void Update(GameTime gameTime)
229         {
230             if (GamePad.GetState(PlayerIndex.One).Buttons.Back == ButtonState.Pressed)
231                 this.Exit();
232
233             base.Update(gameTime);
234         }
235
236         protected override void Draw(GameTime gameTime)
237         {
238             GraphicsDevice.Clear(Color.CornflowerBlue);
239
240             spriteBatch.Begin();
241             spriteBatch.Draw(kinectRGBVideo, new Rectangle(0, 0, 640, 480), Color.White);
242             spriteBatch.Draw(overlay, new Rectangle(0, 0, 640, 480), Color.White);
243             spriteBatch.DrawString(font, connectedStatus, new Vector2(20, 80), Color.White);
244             spriteBatch.End();
245
246             base.Draw(gameTime);
247         }
248     }
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Virtual Fashion Technology

Virtual Fashion Education

toitocuaanhem

"chúng tôi chỉ là tôi tớ của anh em, vì Đức Kitô" (2Cr 4,5b)

VentureBeat

News About Tech, Money and Innovation

digitalerr0r

Modern art using the GPU

Theme Showcase

Find the perfect theme for your blog.

lsuvietnam

Learn to Learn

Gocomay's Blog

Con tằm đến thác vẫn còn vương tơ

Toán cho Vật lý

Khoa Vật lý, Đại học Sư phạm Tp.HCM - ĐT :(08)-38352020 - 109

Maths 4 Physics & more...

Blog Toán Cao Cấp (M4Ps)

Bucket List Publications

Indulge- Travel, Adventure, & New Experiences

Lib4U

‎"Behind every stack of books there is a flood of knowledge."

The WordPress.com Blog

The latest news on WordPress.com and the WordPress community.

%d bloggers like this: