Motion detection using depth images

• US 8,588,517 B2
• Filed: 01/15/2013
• Issued: 11/19/2013
• Est. Priority Date: 12/18/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lee, Johnny, Leyvand, Tommer, Peeper, Craig
• Primary Examiner(s): Wang, Claire X
• Application Number:
  US13/742,094
• Publication Number:
  US 20130129155A1
• Time in Patent Office: 308 Days
• Field of Search: None
• US Class Current: 382/154
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A sensor system creates a sequence of depth images that are used to detect and track motion of objects within range of the sensor system. A reference image is created and updated based on a moving average (or other function) of a set of depth images. A new depth images is compared to the reference image to create a motion image, which is an image file (or other data structure) with data representing motion. The new depth image is also used to update the reference image. The data in the motion image is grouped and associated with one or more objects being tracked. The tracking of the objects is updated by the grouped data in the motion image. The new positions of the objects are used to update an application.

Claims

1. A method for using depth images to sense motion, comprising:

creating a reference image;

receiving a new depth image;

creating a motion image based on the new depth image and the reference image, wherein the motion image represents identified forward motion;

identifying one or more objects in the motion image; and

updating the reference image based on the new depth image.

2. The method of claim 1, wherein:

at least one of the one or more objects comprises a body part; and

the identifying one or more objects in the motion image comprises identifying one or more body parts in the motion image.

3. The method of claim 2, further comprising:

accessing structure data that includes structural information that enables the one or more body parts to be recognized;

wherein the identifying one or more body parts in the motion image is performed based on the structure data; and

wherein the structure information includes a skeletal model of a human.

4. The method of claim 2, further comprising:

tracking motion of at least one of the one or more body parts in the motion image;

wherein the identifying one or more body parts in the motion image includes grouping pixels of the motion image to form one or more groups of pixels, and associating at least one of the one or more groups of pixels with at least one body part for which motion is being tracked.

5. The method of claim 1, further comprising:

accessing a gestures library that includes a plurality of gesture filters, wherein each of the gesture filters comprises information concerning a gesture that can be performed; and

comparing the motion image to the gesture filters to thereby identify when one or more of the gestures are performed.

6. The method of claim 5, further comprising:

controlling an application based the one or more gestures that are identified as being performed.

7. The method of claim 1, further comprising:

accessing a gestures library; and

identifying when one or more gestures are performed based on the motion image and the gestures library.

8. The method of claim 1, wherein:

the reference image comprises a moving average of a plurality of previous depth images and is in a same format as the previous depth images.

9. The method of claim 1, wherein:

the new depth image includes a two dimensional arrangement of pixels, where the pixels represent depth of the one or more objects.

10. The method of claim 1, wherein:

the reference image is created based on a mathematical function of multiple previous depth images of a first object moving in a scene;

the received new depth image also portrays the first object moving in the scene; and

the identifying one or more objects in the motion image includes identifying the first object in the scene.

11. An apparatus that uses depth images to sense motion, comprising:

a communication interface that receives depth images;

one or more storage devices that store depth images;

a display interface; and

one or more processors in communication with the one or more storage devices and the display interface, wherein the one or more processors

create a reference image that includes foreground data and background data based on a mathematical function of multiple previous depth images of an object moving in a scene;

access a new depth image of the object in the scene received from the communication interface;

create a motion image based on the new depth image and the reference image;

identify the object in the motion image; and

update the reference image based on the new depth image and the mathematical function.

12. The apparatus of claim 11, wherein:

the one or more processors use position information for the identified object to update an application running on the apparatus and provide signals on the display interface that indicate an update to the application.

13. The apparatus of claim 11, wherein:

each reference image and depth image include a same number of pixels; and

the mathematical function is used to operate on each pixel of the reference image.

14. The apparatus of claim 11, wherein:

the mathematic function, which is used to create and update the reference image, calculates a moving average of a most recent N previous depth images of the object moving in the scene.

15. The apparatus of claim 11, wherein:

the new depth image and the reference image are two dimensional arrangements of pixels of a common captured scene where the pixels represent depth of the object.

16. One or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors to perform a method for using depth images to sense motion, the method comprising:

creating a reference image including a number of pixels;

receiving a new depth image including a same number of pixels as in the reference image;

creating a motion image based on the new depth image and the reference image, wherein the motion image includes a same number of pixels and is in a same format as the reference image and the new depth image;

identifying one or more objects in the motion image; and

updating the reference image based on the new depth image.

17. The one or more processor readable storage devices of claim 16, wherein:

the new depth image and the reference image are two dimensional arrangements of pixels of a common captured scene where the pixels represent depth of the one or more objects.

18. The one or more processor readable storage devices of claim 16, wherein:

the creating the motion image includes subtracting the new depth image from the reference image to create a set of difference data, identifying difference data greater than a threshold as motion data that includes one of either forward motion data and backward motion data, and discarding the backward motion data.

19. The one or more processor readable storage devices of claim 16, wherein:

the identifying one or more objects in the motion image includes grouping pixels of the motion image to form one or more groups of pixels, associating each of the one or more groups of pixels with one or more objects identified in object history data, and updating the object history data.

20. The one or more processor readable storage devices of claim 19, wherein:

the identifying one or more objects in the motion image includes identifying a center of each of the one or more groups of pixels; and

the associating each of the one or more groups of pixels with one or more objects identified in object history data includes associating the center of each of the one or more groups of pixels with an object having a closest proximity in a most recent motion image.