Environment And/Or Target Segmentation

• US 20100302395A1
• Filed: 05/29/2009
• Published: 12/02/2010
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Marais, Charles Claudius, Mathe, Zsolt
• Granted Patent: US 8,379,101 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 20/80 Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A depth image of a scene may be observed or captured by a capture device. The depth image may include a human target and an environment. One or more pixels of the depth image may be analyzed to determine whether the pixels in the depth image are associated with the environment of the depth image. The one or more pixels associated with the environment may then be discarded to isolate the human target and the depth image with the isolated human target may be processed.

Claims

1. A device for processing a depth information of a scene, the device comprising:

a camera component, wherein the camera component receives a depth image of the scene; and

a processor, wherein the processor executes computer executable instructions, and wherein the computer executable instructions comprise instructions for;

receiving the depth image of the scene from the camera component;

determining whether one or more pixels in the depth image are associated with an environment of the depth image; and

discarding the one or more pixels associated with the environment to isolate a human target in the depth image; and

processing the depth image with the isolated human target.

2. The device of claim 1, wherein the instructions for determining one or more pixels associated with the environement in the depth image comprising instructions for:

defining a bounding box based on a centroid of the human target and a measurement of the target;

identifying one or more pixels outside of the bounding box as associated with the environment.

3. The device of claim 1, wherein the instructions for determining one or more pixels associated with the environment in the depth image comprising instructions for:

flood filling one or more pixels associated with the human target.

4. The device of claim 3, wherein the instructions for determining one or more pixels associated with the environment in the depth image further comprising instructions for:

detecting an edge of the target using a predetermined edge tolerance value.

5. The device of claim 4, wherein the instructions for determining one or more pixels associated with the environment in the depth image further comprising instructions for:

adjusting the predetermined edge tolerance value based at least one of the following;

a measurement of the target, the position of the one or more pixels, and a measurement of the target.

6. The device of claim 1, wherein the instructions for determining one or more pixels associated with the environment in the depth image comprising instructions for:

generating a predefined number of sample points; and

flood filling one or more pixels associated with the human target using the predefined number of sample points.

7. The device of claim 1, further comprising instructions for:

identifying a pixel associated with a infrared shadow in the depth image; and

replacing the depth value of the pixel associated with the infrared shadow with a calculated depth value.

8. The device of claim 1, wherein the depth image comprises a depth value for each pixel therein, and wherein the instructions for determining one or more pixels associated with the environment in the depth image comprising instructions for:

examining a pixel in the depth image;

retrieving a maximum depth value corresponding to the pixel from depth history data; and

comparing the depth value of the pixel with a maximum depth value of the pixel in the depth history data.

9. A computer-readable storage medium having stored thereon executable instructions for processing depth information of a scene, the computer executable instructions comprising instructions for:

receiving a depth image of the scene, wherein the depth image comprises one or more human targets and an environment;

flood filling pixels associated with the one or more human targets;

discarding environment pixels to isolate the flood filled pixels of the one or more human targets, wherein the environment pixels include one or more pixels not associated with the flood filled pixels of the one or more human targets; and

processing the depth image with the flood filled pixels associated with the one or more human targets.

10. The computer-readable medium of claim 9, wherein the computer executable instructions for flood filling pixels associated with the one or more human targets further comprising instructions for:

generating a predefined number of sample points based on at least one of a location of the one or more human targets and a measurement of the one or more human targets; and

flood filling pixels associated with the one or more human targets using the predefined number of sample points.

11. The computer-readable medium of claim 9, wherein the computer executable instructions for flood filling the pixels associated with the one or more human targets further comprising instructions for:

identifying one or more pixels associated with an object in the first depth image;

determining whether the object is associated with the one or more human targets; and

flood filling the one or more pixels if, based on the determination, the one or more pixels are associated with the one or more human targets.

12. The computer-readable medium of claim 9, wherein the depth image comprises a plurality of pixels, and wherein each pixel has a depth value associated therewith.

13. The computer-readable medium of claim 12, wherein the computer executable instructions further comprising instructions for:

tracking a maximum depth value for each of the plurality of pixels of the depth image; and

storing the maximum depth value in historic depth value data.

14. The computer-readable medium of claim 13, wherein the computer executable instructions further comprising instructions for:

identifying a plurality of pixels associated with a floor in the scene; and

storing the depth values of the plurality of floor pixels in the historic depth value data.

15. The computer-readable medium of claim 13, the computer executable instructions further comprising instructions for:

calculating the maximum depth value for a pixel associated with one of the plurality of pixels in the depth image using one or more surrounding pixels.

16. The computer-readable medium of claim 13, wherein the computer executable instructions further comprising instructions for:

examining a pixel from the plurality of pixels of the depth image;

retrieving the maximum depth value corresponding to the pixel from depth history data;

comparing a depth value of the pixel with the corresponding maximum depth value; and

flood filling the pixel if, based on the determination, the depth value of the pixel and the corresponding maximum depth value is less than the maximum depth value.

17. A system for processing depth information of a scene, the system comprising:

a capture device, the capture device comprising a camera component that receives a depth image of a scene, wherein the depth image comprises a human target;

a computing device in operative communication with the capture device, wherein the computing device comprises a processor that receives the depth image of the scene from the capture device, examines a pixel of the depth image, determines whether the pixel is associated with the human target; and

removes the pixels if, based on the determination, the pixel is not associated with the human target.

18. The system of claim 17, wherein the system further comprising a storage component that stores a maximum depth value for each pixel of the first depth image, and wherein the processor further retrieves the maximum depth value corresponding to the pixel of the depth image from the depth history data;

, compares a depth value of the pixel with the maximum depth value of the pixel in the depth history data, and removes the pixel if the depth value of the pixel is less than the maximum depth value of the pixel.

19. The system of claim 17, wherein processor determines whether the pixel is associated with the human target by defining a bounding box based on a centroid of the human target and a measurement of the target, and determining whether the pixel is outside the bounding box.

20. The system of claim 19, wherein the processor removes the pixels if, based on the determination, the pixel is outside the bounding box.

21. A computer-implemented method for processing depth information of a scene, the computer-implemented method comprising:

receiving a depth image of the scene, wherein the depth image includes a desired object and an environment; and

segmenting the desired object from the environment.