ENVIRONMENTAL MODIFICATIONS TO MITIGATE ENVIRONMENTAL FACTORS

• US 20120201424A1
• Filed: 02/03/2011
• Published: 08/09/2012
• Est. Priority Date: 02/03/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Masalkar, Prafulla, Leach, Alan
• Granted Patent: US 8,724,887 B2
• Time in Patent Office: Days
• US Class Current: 382/106
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/217 using environment-related information, i.e. information generated otherwise than by the player, e.g. ambient temperature or humidity
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A method of depth imaging includes acquiring a depth image from a depth camera, identifying an environmental factor invalidating depth information in one or more portions of the depth image, and outputting an environmental modification to mitigate the environmental factor.

Claims

1. A data-holding subsystem holding instructions executable by a logic subsystem to:

acquire a depth image from a depth camera;

identify an environmental factor invalidating depth information in one or more portions of the depth image; and

output an environmental modification to mitigate the environmental factor.

2. The data-holding subsystem of claim 1, further holding instructions executable by the logic subsystem to model a human target imaged by the depth image with a virtual skeleton.

3. The data-holding subsystem of claim 1, further holding instructions executable by the logic subsystem to assign one or more areas of the depth image including invalid depth information to a problem category from a plurality of predetermined problem categories.

4. The data-holding subsystem of claim 3:

wherein an area is assigned to the problem category because the area contains invalid depth pixels in a shape of a clothing item of a human target and a test image of the area while exposed to infrared test light does not change by a threshold amount compared to a control image of the area not exposed to the infrared test light; and

wherein the environmental modification output to mitigate the environmental factor includes a recommendation to the human target to move closer to the depth camera.

5. The data-holding subsystem of claim 3:

wherein an area is assigned to the problem category because the area contains invalid depth pixels in a round shape surrounded by near-depth pixels and a test image of the area while exposed to infrared test light is high in intensity and the intensity reduces by a threshold amount in a control image of the area not exposed to the infrared test light; and

wherein the environmental modification output to mitigate the environmental factor includes a recommendation to a human target to move away from the depth camera and/or reduce ambient light in a field of view of the depth camera.

6. The data-holding subsystem of claim 3:

wherein an area is assigned to the problem category because the depth image has a high noise level and a test image of a field of view around the area while exposed to infrared test light has a high intensity and does not change by a threshold amount compared to a control image of the field of view not exposed to the infrared test light; and

wherein the environmental modification output to mitigate the environmental factor includes identification of a light source in the field of view and a recommendation to the human target to reduce light from the light source.

7. The data-holding subsystem of claim 3:

wherein an area is assigned to the problem category because the area contains invalid depth pixels in a specific and localized stripe across the depth image and a test image of the area while exposed to infrared test light contains saturated pixels and does not change by a threshold amount compared to a control image of the area not exposed to the infrared test light; and

wherein the environmental modification output to mitigate the environmental factor includes a recommendation to the human target to reduce light in a field of view of the depth camera.

8. The data-holding subsystem of claim 3:

wherein an area is assigned to the problem category because the area contains invalid depth pixels in a round shape and a test image of the area while exposed to infrared test light contains saturated pixels in a round shape and does not change by a threshold amount compared to a control image of the area not exposed to the infrared test light; and

wherein the environmental modification output to mitigate the environmental factor includes identification of a light source shining in a field of view of the depth camera and a recommendation to the human target to reduce light from the light source.

9. The data-holding subsystem of claim 3:

wherein an area is assigned to the problem category because the area contains invalid depth pixels in small groups and a control image of a sample surrounding the area not exposed to infrared test light is reduced by a threshold amount compared to a test image of the sample exposed to the infrared test light; and

wherein the environmental modification output to mitigate the environmental factor includes a recommendation to a human target to move closer to the depth camera and/or reduce light in the field of view.

10. The data-holding subsystem of claim 3, wherein the environmental modification is selected based on the problem category.

11. The data-holding subsystem of claim 1, wherein the instructions are executable to output the environment modification by outputting a communication message for presentation to a human target, the communication message providing information on mitigating the environmental factor.

12. The data-holding subsystem of claim 1, wherein the instructions are executable to output the environment modification by outputting a control message for execution by an environmental modification system so that the environmental modification system alters an environment to mitigate the environmental factor without human intervention.

13. A method of depth imaging, comprising:

imaging a scene with a depth camera configured to measure a distance between the depth camera and surfaces in the scene;

in a test mode, acquiring from the depth camera a test image in which the scene is exposed with an infrared test light;

in a control mode, acquiring from the depth camera a control image in which the scene is not exposed with the infrared test light; and

outputting an environmental modification to mitigate an environmental factor identified by comparing the test image to the control image.

14. A method of depth imaging, comprising:

acquiring a depth image from a depth camera, the depth camera configured to measure a distance between the depth camera and surfaces in an imaged scene and configured to output the depth image as an indication of such measured distances;

modeling a human target imaged by the depth image with a virtual skeleton;

responsive to a failure to model the human target with the virtual skeleton, identifying a cluster of invalid depth pixels in the depth image;

acquiring a test image from the depth camera while the imaged scene is exposed with an infrared test light;

acquiring a control image from the depth camera while the imaged scene is not exposed with the infrared test light;

identifying an environmental factor responsible for the cluster of invalid depth pixels based on the cluster of invalid depth pixels, the test image, and the control image; and

outputting a communication message for presentation to the human target, the communication message providing information useable by the human target to mitigate the environmental factor.

15. The method of claim 14:

wherein the environmental factor responsible for the cluster of invalid depth pixels is identified by the cluster of invalid depth pixels having a shape of a clothing item of the human target and the test image having differences below a threshold amount compared to the control image; and

wherein the communication message to mitigate the environmental factor includes a recommendation to the human target to move closer to the depth camera.

16. The method of claim 14:

wherein the environmental factor responsible for the cluster of invalid depth pixels is identified by the cluster of invalid depth pixels having a round shape and being surrounded by near-depth pixels and the test image having high intensity pixels in a round shape that are reduced by a threshold amount in the control image; and

wherein the communication message to mitigate the environmental factor includes a recommendation to the human target to move away from the depth camera and/or reduce ambient light present in the imaged scene.

17. The method of claim 14:

wherein the environmental factor responsible for the cluster of invalid depth pixels is identified by the depth image having a high noise level and the test image having high intensity pixels in the imaged scene that are not reduced by a threshold amount in the control image; and

wherein the communication message to mitigate the environmental factor includes identification of a light source in the imaged scene and a recommendation to the human target to reduce light from the light source.

18. The method of claim 14:

wherein the environmental factor responsible for the cluster of invalid depth pixels is identified by the depth image having a specific and localized stripe of invalid depth pixels and the test image having high intensity pixels at the stripe that are not reduced by a threshold amount in the control image; and

wherein the communication message to mitigate the environmental factor includes a recommendation to the human target to reduce light in the imaged scene.

19. The method of claim 14:

wherein the environmental factor responsible for the cluster of invalid depth pixels is identified by the cluster of invalid depth pixels having a round shape and the test image having high intensity pixels in a round shape that are not reduced by a threshold amount in the control image; and

wherein the communication message to mitigate the environmental factor includes identification of a light source in the imaged scene and a recommendation to the human target to reduce light from the light source.

20. The method of claim 14:

wherein the environmental factor responsible for the cluster of invalid depth pixels is identified by groups of invalid depth pixels in the control image having a pixel intensity that is less than a pixel intensity of corresponding groups in the test image by a threshold amount; and

wherein the communication message to mitigate the environmental factor includes a recommendation to the human target to move closer to the depth camera and/or reduce ambient light present in the imaged scene.