DEPTH MEASUREMENT QUALITY ENHANCEMENT

• US 20140253679A1
• Filed: 06/22/2012
• Published: 09/11/2014
• Est. Priority Date: 06/24/2011
• Status: Active Grant

• Current Assignee: SoftKinetic Software
• Original Assignee: SoftKinetic Software
• Inventors: Guigues, Laurent, Samuel, Michel, Thollot, Julien
• Granted Patent: US 9,426,444 B2
• Time in Patent Office: Days
• US Class Current: 348/42
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Described herein is a method for correcting defective depth values in depth map images. Defective values correspond to “noisy pixels” located on continuous flat surfaces and interpolated “flying pixels” located along an edge between a foreground object and a background object. The method comprising the steps of accessing a depth map of a scene which includes the foreground and background objects, detecting and identifying noisy and interpolated depth measurements within the depth map using a method, defining and applying a correction to each of the detected defective noisy and interpolated depth measurements using a specifically defined weighted correction factor. By providing the corrected defective depth values in depth map images, edges are sharpened in depth and continuous surfaces are flattened, enabling higher efficiency and robustness of further image processing.

Claims

1. A method for depth map quality enhancement of defective pixel depth data values in a three-dimensional image, the method comprising the steps of:

a) determining depth measurement data relating to a scene;

b) detecting defective pixels within the depth measurement data by determining and using, for each pixel, depth related directional derivatives in at least one direction;

c) defining a depth correction for each detected defective pixel; and

d) applying the depth correction to the depth measurement data of each detected defective pixel;

characterised in that the defective pixels comprise interpolated pixel data values located at edges between a foreground object and a background object in the three-dimensional image, and wherein step b) further comprises using the depth related directional derivatives to identify defective depth measurements of pixels at edges when at least one depth directional derivative of a pixel is greater than a predetermined threshold and if at least two consecutive directional derivatives have same sign; and

in that step c) comprises, for each identified defective pixel, the steps of;

c1) determining a vector in relation to at least one of the depth directional derivatives;

c2) determining the normal to the determined vector;

c3) determining a weighting factor parameter using at least one of the determined vector and the normal to the determined vector; and

c4) determining a correction factor using at least one of the weighting factor parameter and the information relating to neighbouring pixels.

4. A method according to claim 1, wherein step c4) uses only data values of valid neighbouring pixels.

5. A method according to claim 1, wherein step c4) further comprises using at least one of depth values, weighting factors, and correction factors of the neighbouring pixels.

6. A method according to claim 1, wherein step c4) comprises using a statistical mode of the information relating to neighbouring pixels.

7. A method according to claim 1, wherein step c4) further comprises using the depth information extracted from a regressive plane determined over the neighbouring pixels.

10. A method according to claim 1, wherein the defective pixels comprise noisy pixel data values located in continuous surfaces of the three-dimensional image, and wherein step b) further comprises using the depth related directional derivatives to identify defective depth measurements of pixels on a continuous surface when at least one depth directional derivative of a pixel is greater than another predetermined threshold and when another depth directional derivative of that pixel is also greater than another predetermined threshold, the two directional derivatives having opposite signs.

12. A method according to claim 10, wherein step c) further comprises, for each identified defective pixel, the steps of:

c5) determining a vector in relation to the depth directional derivatives data values using two orthogonal axes;

c6) determining a weighting factor parameter using at least one of a radius value of the determined vector, normal information to the determined vector, and real width in scene represented by the pixel; and

c7) applying a correction factor using the determined weighting factor parameter in combination with information relating to neighbouring pixels.

13. A method according to claim 1, wherein the depth related directional derivatives are determined using at least two orthogonal axes.

14. A method according to claim 5, wherein the depth related directional derivatives are determined using a normal map.

15. A method according to claim 5, wherein the depth related directional derivatives are used for determining a normal map.

16. A method according to claim 1, wherein step c) further comprises, for each identified defective pixel, the steps of:

c5) determining a vector in relation to the depth related directional derivatives data values using two orthogonal axes;

c6) determining a weighting factor using at least one of a radius value of the determined vector, normal information to the determined vector, and real width in scene represented by the pixel; and

c7) applying a correction factor using the determined weighting factor in combination with information relating to neighbouring pixels.

17. A method according to claim 6, wherein the depth related directional derivatives are determined using a normal map.

18. A method according to claim 6, wherein the depth related directional derivatives are used for determining a normal map.

19. A method according to claim 7, wherein the depth related directional derivatives are determined using a normal map.

20. A method according to claim 7, wherein the depth related directional derivatives are used for determining a normal map.

21. A method according to claim 13, wherein the depth related directional derivatives are determined using a normal map.

22. A method according to claim 13, wherein the depth related directional derivatives are used for determining a normal map.

23. A method according to claim 1, wherein the depth related directional derivatives are determined using a normal map.

24. A method according to claim 1, wherein the depth related directional derivatives are used for determining a normal map.

2-3. -3. (canceled)

8-9. -9. (canceled)

11. (canceled)