Motion detection using depth images

• US 8,374,423 B2
• Filed: 03/02/2012
• Issued: 02/12/2013
• Est. Priority Date: 12/18/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lee, Johnny, Leyvand, Tommer, Peeper, Craig
• Primary Examiner(s): Wang, Claire X
• Application Number:
  US13/410,546
• Publication Number:
  US 20120177254A1
• Time in Patent Office: 347 Days
• Field of Search: None
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A sensor system creates a sequence of depth images that are used to detect and track motion of objects within range of the sensor system. A reference image is created and updated based on a moving average (or other function) of a set of depth images. A new depth images is compared to the reference image to create a motion image, which is an image file (or other data structure) with data representing motion. The new depth image is also used to update the reference image. The data in the motion image is grouped and associated with one or more objects being tracked. The tracking of the objects is updated by the grouped data in the motion image. The new positions of the objects are used to update an application.

Claims

1. A method for using depth images to sense motion, comprising:

creating a reference image;

receiving a new depth image;

creating a motion image based on the new depth image and the reference image, wherein the motion image represents identified forward motion;

identifying one or more objects in the motion image;

using position information for the identified one or more objects to update an application; and

updating the reference image based on the new depth image.

2. The method of claim 1, wherein:

the creating the motion image includes subtracting the new depth image from the reference image to create a set of difference data, identifying difference data greater than a threshold as motion data that includes forward motion data and backward motion data, and discarding the backward motion data.

3. The method of claim 1, wherein:

the identifying one or more objects in the motion image includes grouping pixels of the motion image to form one or more groups of pixels, associating each of the one or more groups of pixels with one or more objects identified in object history data, and updating the object history data; and

the using position information for the identified one or more objects to update the application includes reporting the one or more objects and positions for the one or more objects to the application.

4. The method of claim 3, wherein:

the grouping of pixels is based on identifying connecting pixels.

5. The method of claim 3, wherein:

the grouping of pixels is based on proximity to previously identified objects.

6. The method of claim 1, wherein:

the reference image is created based on multiple previous depth images and is in a same format as the multiple previous depth images; and

the reference image is a moving average of the most recent N frames of depth images.

7. The method of claim 1, wherein:

the identifying one or more objects in the motion image includes grouping pixels of the motion image to an initial group of pixels, determining that the initial group of pixels represents two objects, splitting the initial group of pixels into a first group of pixels and a second group of pixels, associating the first group of pixels with a first object based on object history data, and associating the second group of pixels with a second object based on object history data.

8. The method of claim 1, wherein:

the identifying one or more objects in the motion image includes grouping pixels of the motion image to form two or more groups of pixels, predicting a trajectory of an object and identifying the group of pixels closest to the predicted trajectory.

9. The method of claim 1, wherein:

the new depth image and the reference image are two dimensional arrangements of pixels of a common captured scene where the pixels represent depth of the one or more objects.

10. The method of claim 1, wherein:

the new depth image includes a two dimensional arrangements of pixels, the pixels represent depth of the one or more objects.

11. The method of claim 1, wherein:

the using position information for the identified one or more objects to update the application includes changing the position of an image in a video display in response to the position information.

12. The method of claim 1, wherein:

the reference image is created based on a mathematical function of multiple previous depth images of a first object moving in a scene;

the received new depth image also portrays the first object moving in the scene;

the identifying one or more objects in the motion image includes identifying the first object in the scene; and

the position information includes position information for the first object in the scene.

13. An apparatus that uses depth images to sense motion, comprising:

a communication interface that receives depth images;

one or more storage devices that store depth images;

a display interface; and

one or more processors in communication with the one or more storage devices and the display interface, the one or more processors access a new depth image received from the communication interface and identify motion based comparing the new depth image to a reference depth image stored in the one or more storage devices, the one or more processors create a motion image representing identified motion, the one or more processors group pixels of the motion image and associate one or more groups of pixels with one or more objects identified in object history data stored in the one or more storage devices, the one or more processors use position information for the identified one or more objects to update an application running on the apparatus and provide signals on the display interface that indicate the update to the application, the new depth image and the reference depth image are two dimensional arrangements of pixels of a common captured scene where the pixels represent depth of the one or more objects.

14. The apparatus of claim 13, wherein:

the display interface connects to a video monitor;

the one or more processors use the position information for the identified one or more objects to update a position of an image on the video monitor; and

the image is displayed by the application.

15. The apparatus of claim 13, wherein:

the reference depth image includes foreground and background data based on multiple previous depth images.

16. The apparatus of claim 13, wherein:

the one or more processors determine that a particular group of pixels represents two objects, split the particular group of pixels into a first group of pixels and a second group of pixels, associate the first group of pixels with a first object based on object history data, and associate the second group of pixels with a second object based on object history data.

17. The apparatus of claim 13, wherein:

the one or more processors also use structure information about items in real space in order to associate the one or more groups of pixels with the one or more objects.

18. A method for using depth images to sense motion, comprising:

receiving a new depth image;

identifying motion based on comparing the new depth image to a reference image;

creating a motion image representing identified forward motion, and discarding identified backward motion when creating the motion image;

identifying one or more objects in the motion image; and

reporting the identified one or more objects in the motion image.

19. The method of claim 18, further comprising:

creating the reference image that includes foreground data and background data based on multiple previous depth images.

20. The method of claim 18, wherein:

the reporting the identified one or more objects includes reporting objects and positions of the objects to an application that uses the position to create video.