REFINEMENT OF STRUCTURED LIGHT DEPTH MAPS USING RGB COLOR DATA

• US 20180232859A1
• Filed: 02/14/2017
• Published: 08/16/2018
• Est. Priority Date: 02/14/2017
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Siddiqui, Hasib, Atanassov, Kalin, Nash, James
• Granted Patent: US 10,445,861 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/90 Determination of colour characteristics

Abstract

Systems and method for refining a depth map of a scene based upon a captured image of the scene. A captured depth map of the scene may contain outage areas or other areas of low confidence. The depth map may be aligned with a color image of the scene, and the depth values of the depth map may be adjusted based upon corresponding color values of the color image. An amount of refinement for each depth value of the aligned depth map is based upon the confidence value of the depth value and a smoothing function based upon a corresponding location of the depth value on the color image.

Claims

1. A computer-implemented method for refining a depth map, comprising:

receiving a color image of the scene;

receiving a depth map of a scene, the depth map comprising depth values corresponding to different locations of the scene, wherein each location on the depth map corresponds to a location on the color image; and

refining the depth values of the depth map based upon color values of corresponding locations on the color image.

2. The computer-implemented method of claim 1, wherein each depth value of the depth map is associated with a confidence value, and wherein an amount of refinement for each depth value of the depth map is based upon the corresponding confidence value of the depth value.

3. The computer-implemented method of claim 1, wherein an amount of refinement for a depth value of the depth map is based upon a smoothing function applied to a corresponding location of the depth value on the color image.

4. The computer-implemented method of claim 3, wherein refining the depth values of the depth map comprises, over a plurality of iterations, adjusting a depth value of the depth map based upon the smoothing function and a value indicating a current adjustment value.

5. The computer-implemented method of claim 3, wherein adjusting the depth value of the depth map is based upon a weighted average of a corresponding unadjusted depth value and a calculated adjustment value based upon the smoothing function.

6. The computer-implemented method of claim 3, wherein the smoothing function comprises a denoising function.

7. The computer-implemented method of claim 1, wherein receiving the depth map comprises receiving an unaligned depth map of the scene, and aligning the unaligned depth map with the color image such that each location on the aligned depth map corresponds to a corresponding location on the color image

8. The computer-implemented method of claim 7, wherein aligning the unaligned depth map with the color image comprises warping the unaligned depth map.

9. The computer-implemented method of claim 1, wherein refining the depth values of the depth map comprises adjusting a depth value of the depth map using a denoising function.

10. The computer-implemented method of claim 1, wherein the depth map of the scene is generated using a transmitter configured to project a pattern of codewords over the scene, and a receiver configured to detect the projected pattern of codewords on the scene.

23. The non-transitory computer readable medium of claim 2, wherein an amount of refinement for each depth value of the depth map is based upon a smoothing function applied to a corresponding location of the depth value on the color image.

24. The non-transitory computer readable medium of claim 23, wherein refining the depth values of the depth map comprises, over a plurality of iterations, adjusting a depth value of the depth map based upon the smoothing function and a value indicating a current adjustment value.

25. The non-transitory computer readable medium of claim 23, wherein adjusting the depth value of the depth map is based upon a weighted average of a corresponding unadjusted depth value and a calculated adjustment value based upon the smoothing function.

26. The non-transitory computer readable medium of claim 23, wherein the smoothing function comprises a denoising function.

11. An apparatus for refining a depth map, comprising:

a depth map generation system configured to generate a depth map of a scene, the depth map comprising depth values corresponding to different locations of the scene;

at least one processor configured to;

receive a color image of the scene;

receive the depth map of the scene from the depth map generation system, wherein each location on the depth map corresponds to a location on the color image; and

refine the depth values of the depth map based upon color values of corresponding locations on the color image.

12. The apparatus of claim 11, wherein each depth value of the depth map is associated with a confidence value, and wherein an amount of refinement for each depth value of the depth map is based upon the corresponding confidence value of the depth value.

13. The apparatus of claim 11, wherein an amount of refinement for each depth value of the depth map is based upon a smoothing function applied to a corresponding location of the depth value on the color image.

14. The apparatus of claim 13, wherein the at least one processor is further configured to refine the depth values of the depth map by adjusting, over a plurality of iterations, a depth value of the depth map based upon the smoothing function and a value indicating a current adjustment value.

15. The apparatus of claim 13, wherein the at least one processor is further configured to adjust the depth value of the depth map based upon a weighted average of a corresponding unadjusted depth value and a calculated adjustment value based upon the smoothing function.

16. The apparatus of claim 13, wherein the smoothing function comprises a denoising function.

17. The apparatus of claim 11, wherein the processor is further configured to align the depth map with the color image such that each location on the depth map corresponds to a corresponding location on the color image

18. The apparatus of claim 17, wherein the at least one processor is further configured to align the depth map with the color image by warping the depth map.

19. The apparatus of claim 11, wherein the at least one processor is further configured to refine the depth values of the depth map by adjusting a depth value of the depth map using a denoising function.

20. The apparatus of claim 11, wherein the depth map generation system comprises a transmitter configured to project a pattern of codewords over the scene, and a receiver configured to detect the projected pattern of codewords on the scene.

21. A non-transitory computer readable medium comprising instructions that when executed cause a processor to perform a method for refining a depth map, the method comprising:

receiving a color image of the scene;

receiving a depth map of a scene, the depth map comprising depth values corresponding to different locations of the scene, wherein each location on the depth map corresponds to a location on the color image; and

refining the depth values of the depth map based upon color values of corresponding locations on the color image.

22. The non-transitory computer readable medium of claim 21, wherein each depth value of the depth map is associated with a confidence value, and wherein an amount of refinement for each depth value of the depth map is based upon the corresponding confidence value of the depth value.

27. The non-transitory computer readable medium of claim 21, wherein receiving the depth map comprises receiving an unaligned depth map of the scene, and aligning the unaligned depth map with the color image such that each location on the aligned depth map corresponds to a corresponding location on the color image

28. The non-transitory computer readable medium of claim 27, wherein aligning the unaligned depth map with the color image comprises warping the unaligned depth map.

29. The non-transitory computer readable medium of claim 21, wherein refining the depth values of the depth map comprises adjusting a depth value of the depth map using a denoising function.

30. The non-transitory computer readable medium of claim 21, wherein the depth map of the scene is generated using a transmitter configured to project a pattern of codewords over the scene, and a receiver configured to detect the projected pattern of codewords on the scene.