Depth Image Noise Reduction

• US 20100302365A1
• Filed: 05/29/2009
• Published: 12/02/2010
• Est. Priority Date: 05/29/2009
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Geiss, Ryan Michael, Finocchio, Mark J.
• Application Number:
  US12/475,074
• Publication Number:
  US 20100302365A1
• Time in Patent Office: Days
• US Class Current: 348/142
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/833 Hand-to-hand fighting, e.g. martial arts competition A63F13/837 takes precedence
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/5553 user representation in the game field, e.g. avatar
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 2300/8029 Fighting without shooting
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06V 10/30 Noise filtering
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format

Abstract

A depth image of a scene may be received, observed, or captured by a device. The depth image may then be analyzed to determine whether the depth image includes noise. For example, the depth image may include one or more holes having one or more empty pixels or pixels without a depth value. Depth values for the one or more empty pixels may be estimated and a depth image that includes the estimated depth values for the one or empty more pixels may be rendered.

Claims

1. A device for processing depth information of a scene, the device comprising:

a camera component, wherein the camera component receives a first depth image of the scene; and

a processor, wherein the processor executes computer executable instructions, and wherein the computer executable instructions comprise instructions for;

receiving the first depth image of the scene from the camera component;

estimating a depth value for one or more pixels associated with noise in the depth image; and

rendering a second depth image with the estimated depth value for the one or more pixels associated with the noise in the scene.

2. The device of claim 1, wherein the one or more pixels associated with noise in the depth image includes one or more pixels having a depth value of zero.

3. The device of claim 1, wherein the instructions for estimating the depth value for the one or more pixels associated with the noise in the first depth image comprise instructions for:

identifying a first pixel from the one or more of the pixels associated with the noise in the first depth image;

determining whether pixels adjacent to the first pixel have valid depth values; and

generating a depth value for the pixel based the valid depth values.

4. The device of claim 3, wherein the instructions for generating the depth value for the pixel based the valid depth values comprise instructions for:

identifying a second pixel adjacent to first pixel having the smallest, valid depth value and a third pixel adjacent to the first pixel having the largest, valid depth value;

determining a difference between the smallest, valid depth value and the largest, valid depth value; and

assigning the smallest, valid depth value to the depth value for the first pixel if, based on the determination, the difference is greater than a threshold value.

5. The device of claim 4, wherein the instructions for generating the depth value for the pixel based the valid depth values further comprise instructions for:

calculating an average depth value based on the valid depth values of the adjacent pixels if, based on the determination, the difference is less than a threshold value; and

assigning the average depth value to the depth value for the first pixel.

6. The device of claim 1, wherein the instructions for estimating the depth value for the one or more pixels associated with the noise in the first depth image comprise instructions for:

generating a noise severity value;

adjusting the noise severity value based on a growth value; and

determining whether to estimate the depth value for the one or more pixels associated the noise based on the adjusted growth value.

7. The device of claim 1, further comprising instructions for processing the second depth image.

8. The device of claim 7, wherein the instructions for processing the second depth image comprise instructions for:

determining whether the second depth image includes a human target; and

generating a model of the human target if, based on the determination, the second depth image includes the human target.

9. A computer-readable storage medium having stored thereon computer executable instructions for processing depth information of a captured scene, the computer executable instructions comprising instructions for:

receiving a first depth image of the scene, wherein the first depth image includes an empty pixel;

determining whether a first pixel adjacent to the empty pixel has a first valid depth value;

generating a depth value for the empty pixel based the first valid depth value if, based on the determination, the first pixel has a first valid depth value; and

rendering a second depth image with the generated depth value for the empty pixel.

10. The computer-readable storage medium of claim 9, further comprising instructions for determining whether a second pixel adjacent to the empty pixel has a second valid depth value, wherein the depth value for the empty pixel is generated based on the first and second valid depth values if, based on the determination, the second pixel has a second valid depth value.

11. The computer-readable storage medium of claim 10, wherein the instructions for generating the depth value for the pixel based the first and second valid depth values comprise instructions for:

determining whether the first and second valid depth values include a smallest depth value adjacent to the empty pixel and a largest depth value adjacent to the empty pixel;

calculating a difference between the first and second valid depth values if, based on the determination, the first and second valid depth values are the smallest depth value adjacent to the empty pixel and the largest depth value adjacent to the empty pixel;

assigning the smallest depth value to the depth value for the empty pixel if the difference is greater than a threshold value.

12. The computer-readable storage medium of claim 11, wherein the instructions for generating the depth value for the pixel based the first and second valid depth values further comprise instructions for:

calculating an average depth value based on at least the first and second valid depth values if the difference is less than the threshold value; and

assigning the average depth value to the depth value for the empty pixel.

13. The computer-readable storage medium of claim 9, further comprising instructions for:

generating a noise severity value;

adjusting the noise severity value based on a growth value; and

determining whether to estimate the depth value for the empty pixel based on the adjusted growth value.

14. The computer-readable storage medium of claim 9, further comprising instructions for processing the second depth image.

15. The computer-readable storage medium of claim 15, wherein the instructions for processing the second depth image comprise instructions for:

determining whether the second depth image includes a human target; and

generating a model of the human target if, based on the determination, the second depth image includes the human target.

16. A system for processing depth information of a scene;

the system comprising;

a capture device, wherein the capture device comprises a camera component that receives a first depth image of a scene; and

a computing device in operative communication with the capture device, wherein the computing device comprises a processor that receives the first depth image of the scene from the capture device;

determines whether the first depth image includes a hole having pixels without depth values; and

estimates the depth values for the pixels if, based on the determination, the first depth image includes the hole.

17. The system of claim 16, wherein the processor estimates the depth value for the pixels by identifying a first pixel from the pixels associated with the hole in the first depth image;

determining whether pixels adjacent to the first pixel have valid depth values; and

generating a depth value for the first pixel based the valid depth values of the adjacent pixels.

18. The system of claim 16, wherein the processor further generates a noise severity value;

adjusts the noise severity value based on a growth value; and

determines whether to estimate the depth value for one of the pixels in the hole based on the adjusted growth value.

19. The system of claim 16, wherein the processor further renders a second depth image that includes the estimated depth values, determines whether the second depth image includes a human target, and generates a model of the human target if the second depth image includes the human target.

20. The system of claim 19, wherein the processor further tracks one or more movements of the model.

21. The system of claim 20, wherein the computing device further comprises a gestures library stored thereon, and wherein the processor compares the one or more tracked movements with the gestures library to determine whether to perform a control to perform based on the one or more movements.