MOTION DETECTION USING DEPTH IMAGES

• US 20120177254A1
• Filed: 03/02/2012
• Published: 07/12/2012
• Est. Priority Date: 12/18/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lee, Johnny, Leyvand, Tommer, Peeper, Craig
• Granted Patent: US 8,374,423 B2
• Time in Patent Office: Days
• US Class Current: 382/107
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A sensor system creates a sequence of depth images that are used to detect and track motion of objects within range of the sensor system. A reference image is created and updated based on a moving average (or other function) of a set of depth images. A new depth images is compared to the reference image to create a motion image, which is an image file (or other data structure) with data representing motion. The new depth image is also used to update the reference image. The data in the motion image is grouped and associated with one or more objects being tracked. The tracking of the objects is updated by the grouped data in the motion image. The new positions of the objects are used to update an application.

Claims

1. A method for using depth images to sense motion, comprising:

creating a reference image that includes foreground data and background data based on multiple previous depth images;

receiving a new depth image;

creating a motion image based on the new depth image and the reference image;

identifying one or more objects in the motion image;

using position information for the identified one or more objects to update an application; and

updating the reference image based on the new depth image.

2. The method of claim 1, wherein:

the creating the motion image includes subtracting the new depth image from the reference image to create a set of difference data, identifying difference data greater than a threshold as motion data that includes forward motion data and backward motion data, and discarding the backward motion data.

3. The method of claim 1, wherein:

the identifying one or more objects in the motion image includes grouping pixels of the motion image to form one or more groups of pixels, associating each of the one or more groups of pixels with one or more objects identified in object history data, and updating the object history data; and

the using position information for the identified one or more objects to update the application includes reporting the one or more objects and positions for the one or more objects to the application.

4. The method of claim 3, wherein:

the grouping of pixels is based on identifying connecting pixels.

5. The method of claim 3, wherein:

the grouping of pixels is based on proximity to previously identified objects.

6. The method of claim 1, wherein:

the identifying one or more objects in the motion image includes grouping pixels of the motion image to form two or more groups of pixels, associating each of the two or more groups of pixels with two or more objects identified in object history data, and updating the object history data; and

the using position information for the identified one or more objects to update the application includes reporting the two or more objects and positions for the two or more objects to the application.

7. The method of claim 1, wherein:

the identifying one or more objects in the motion image includes grouping pixels of the motion image to an initial group of pixels, determining that the initial group of pixels represents two objects, splitting the initial group of pixels into a first group of pixels and a second group of pixels, associating the first group of pixels with a first object based on object history data, and associating the second group of pixels with a second object based on object history data.

8. The method of claim 1, wherein:

the identifying one or more objects in the motion image includes grouping pixels of the motion image to form two or more groups of pixels, predicting a trajectory of an object and identifying the group of pixels closest to the predicted trajectory.

9. The method of claim 1, wherein:

the identifying one or more objects in the motion image includes using object history data and structure information about items in real space in order to identify the one or more objects.

10. The method of claim 1, wherein:

the updating the reference image includes performing a weighted averaging function on the reference image and the new depth image; and

the creating the reference image include performing the weighted averaging function on previous depth images.

11. The method of claim 1, wherein:

the using position information for the identified one or more objects to update the application includes changing the position of an image in a video display in response to the position information.

12. The method of claim 1, wherein:

the creating the motion image includes subtracting the new depth image from the reference image to create a set of difference data, identifying difference data greater than a threshold as motion data that includes forward motion data and backward motion data, and discarding the backward motion data;

the identifying one or more objects in the motion image includes grouping pixels of the motion image to form one or more groups of pixels, associating each of the one or more groups of pixels with one or more objects identified in object history data, and updating the object history data;

the using position information for the identified one or more objects to update the application includes reporting the one or more objects and positions for the one or more objects to the application; and

the using position information for the identified one or more objects to update the application includes changing the position of an image in a video display in response to the position information.

13. An apparatus that uses depth images to sense motion, comprising:

a communication interface that receives depth images;

one or more storage devices that store depth images;

a display interface; and

one or more processors in communication with the one or more storage devices and the display interface, the one or more processors access a new depth image received from the communication interface and identify motion based comparing the new depth image to a reference image stored in the one or more storage devices, the one or more processors create a motion image representing identified motion, the one or more processors group pixels of the motion image and associate one or more groups of pixels with one or more objects identified in object history data stored in the one or more storage devices, the one or more processors use position information for the identified one or more objects to update an application running on the apparatus and provide signals on the display interface that indicate the update to the application.

14. The apparatus of claim 13, wherein:

the display interface connects to a video monitor;

the one or more processors use the position information for the identified one or more objects to update a position of an image on the video monitor; and

the image is displayed by the application.

15. The apparatus of claim 13, wherein:

the reference image includes foreground and background data based on multiple previous depth images.

16. The apparatus of claim 13, wherein:

the one or more processors determine that a particular group of pixels represents two objects, split the particular group of pixels into a first group of pixels and a second group of pixels, associate the first group of pixels with a first object based on object history data, and associate the second group of pixels with a second object based on object history data.

17. The apparatus of claim 13, wherein:

the one or more processors also use structure information about items in real space in order to associate the one or more groups of pixels with the one or more objects.

18. A method for using depth images to sense motion, comprising:

receiving a new depth image;

identifying motion based on comparing the new depth image to a reference image;

creating a motion image representing identified forward motion, and discarding identified backward motion when creating the motion image;

identifying one or more objects in the motion image; and

reporting the identified one or more objects in the motion image.

19. The method of claim 18, further comprising:

creating the reference image that includes foreground data and background data based on multiple previous depth images.

20. The method of claim 18, wherein:

the reporting the identified one or more objects includes reporting objects and positions of the objects to an application that uses the position to create video.