Environment and/or target segmentation

• US 8,379,101 B2
• Filed: 05/29/2009
• Issued: 02/19/2013
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mathe, Zsolt, Marais, Charles Claudius
• Primary Examiner(s): NGUYEN, LUONG TRUNG
• Application Number:
  US12/475,094
• Publication Number:
  US 20100302395A1
• Time in Patent Office: 1,362 Days
• Field of Search: None
• US Class Current: 348/222.1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 20/80 Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A depth image of a scene may be observed or captured by a capture device. The depth image may include a human target and an environment. One or more pixels of the depth image may be analyzed to determine whether the pixels in the depth image are associated with the environment of the depth image. The one or more pixels associated with the environment may then be discarded to isolate the human target and the depth image with the isolated human target may be processed.

Claims

1. A device for processing depth information of a scene, the device comprising:

a processor; and

a memory communicatively coupled to the processor when the device is operational, the memory bearing processor-executable instructions that, when executed upon the processor, cause the device to at least;

receive a depth image of the scene;

determine that one or more pixels in the depth image are associated with an environment of the depth image comprising determining a boundary around a target in the depth image and identifying at least a first pixel as within the boundary, the boundary being determined from depth history data;

discard at least a second pixel outside the boundary to isolate the target in the depth image; and

process the depth image with the target.

2. The device of claim 1, wherein the instructions for determining whether one or more pixels in the depth image are associated with the environment of the depth image comprise instructions for:

defining a bounding box based on a centroid of the target and a measurement of the target;

identifying one or more pixels outside of the bounding box as associated with the environment.

3. The device of claim 1, wherein the instructions for determining whether one or more pixels in the depth image are associated with the environment of the depth image comprise instructions for:

flood filling one or more pixels associated with the target.

4. The device of claim 3, wherein the instructions for determining whether one or more pixels in the depth image are associated with the environment of the depth image comprise instructions for:

detecting an edge of the target using a predetermined edge tolerance value.

5. The device of claim 4, wherein the instructions for determining whether one or more pixels in the depth image are associated with the environment of the depth image comprise instructions for:

adjusting the predetermined edge tolerance value based at least one of the following;

a measurement of the target, a position of the one or more pixels, and a measurement of the target.

6. The device of claim 1, wherein the instructions for determining whether one or more pixels in the depth image are associated with the environment of the depth image comprise instructions for:

generating a predefined number of sample points; and

flood filling one or more pixels associated with the target using the predefined number of sample points.

7. The device of claim 1, further comprising instructions for:

identifying a pixel associated with an infrared shadow in the depth image; and

replacing a depth value of the pixel associated with the infrared shadow with a calculated depth value.

8. The device of claim 1, wherein the depth image comprises a depth value for each pixel therein, and wherein the instructions for determining whether one or more pixels in the depth image are associated with the environment of the depth image comprise instructions for:

examining a pixel in the depth image;

retrieving a maximum depth value corresponding to the pixel from depth history data; and

comparing the depth value of the pixel with a maximum depth value of the pixel in the depth history data.

9. A computer-readable storage device having stored thereon executable instructions for processing depth information of a scene, the executable instructions comprising instructions for:

receiving a depth image of the scene, wherein the depth image comprises one or more targets and an environment;

separating the one or more targets from the environment using depth history data and associating one or more pixels with the one or more targets;

discarding environment pixels to isolate the pixels of the one or more targets, wherein the environment pixels include one or more pixels not associated with the one or more targets; and

processing the depth image with the pixels associated with the one or more targets.

10. The computer-readable storage device of claim 9, wherein the computer executable instructions for associating one or more pixels with the one or more targets further comprise instructions for:

generating a predefined number of sample points based on at least one of a location of the one or more targets and a measurement of the one or more targets; and

flood filling pixels associated with the one or more targets using the predefined number of sample points.

11. The computer-readable storage device of claim 9, wherein the computer executable instructions for associating one or more pixels with the one or more targets further comprise instructions for:

identifying one or more pixels associated with an object in the first depth image;

determining whether the object is associated with the one or more targets; and

flood filling the one or more pixels if, based on the determination, the one or more pixels are associated with the one or more targets.

12. The computer-readable storage device of claim 9, wherein the depth image comprises a plurality of pixels, and wherein each pixel has a depth value associated therewith.

13. The computer-readable storage device of claim 12, wherein the computer executable instructions further comprise instructions for:

tracking a maximum depth value for each of the plurality of pixels of the depth image; and

storing the maximum depth value in historic depth value data.

14. The computer-readable storage device of claim 13, wherein the computer executable instructions further comprise instructions for:

identifying a plurality of pixels associated with a floor in the scene; and

storing the depth values of the plurality of floor pixels in the historic depth value data.

15. The computer-readable storage device of claim 13, the computer executable instructions further comprise instructions for:

calculating the maximum depth value for a pixel associated with one of the plurality of pixels in the depth image using one or more surrounding pixels.

16. The computer-readable storage device of claim 13, wherein the computer executable instructions further comprising instructions for:

examining a pixel from the plurality of pixels of the depth image;

retrieving the maximum depth value corresponding to the pixel from depth history data;

comparing a depth value of the pixel with the corresponding maximum depth value; and

flood filling the pixel if, based on the determination, the depth value of the pixel and the corresponding maximum depth value is less than the maximum depth value.

17. A system for processing depth information of a scene, the system comprising:

a capture device, the capture device comprising a camera component that receives a depth image of the scene, wherein the depth image comprises a target and an environment;

a computing device in operative communication with the capture device, wherein the computing device comprises a processor that receives the depth image of the scene from the capture device, examines a pixel of the depth image, determines whether the pixel is associated with the target using depth history data, fills the pixel if the pixel is associated with the target and removes the pixel if the pixel is not associated with the target.

18. The system of claim 17, wherein the system further comprises a storage component that stores a maximum depth value for each pixel of the depth image, and wherein the memory further bears processor-executable instructions that, when executed on the processor, cause the system to at least:

retrieve the maximum depth value corresponding to the pixel of the depth image from the depth history data;

compare a depth value of the pixel with the maximum depth value of the pixel in the depth history data; and

remove the pixel in response to determining that the depth value of the pixel is less than the maximum depth value of the pixel.

19. The system of claim 17, wherein processor determines whether the pixel is associated with the target by defining a bounding box based on a centroid of the target and a measurement of the target, and determining whether the pixel is outside the bounding box.

20. The system of claim 19, wherein the processor removes the pixel if, based on the determination, the pixel is outside the bounding box.

21. A method for processing depth information of a scene, comprising:

receiving a depth image of the scene, the depth image comprising one or more targets and an environment;

separating the one or more targets from the environment using depth history data and associating one or more pixels with the one or more targets;

discarding environment pixels to isolate the one or more pixels of the one or more targets, the environment pixels including one or more pixels not associated with the one or more targets; and

processing the depth image with the one or more pixels associated with the one or more targets.