Refinement of structured light depth maps using RGB color data

• US 10,445,861 B2
• Filed: 02/14/2017
• Issued: 10/15/2019
• Est. Priority Date: 02/14/2017
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Siddiqui, Hasib, Atanassov, Kalin, Nash, James
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US15/432,783
• Publication Number:
  US 20180232859A1
• Time in Patent Office: 973 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/90 Determination of colour characteristics

Abstract

Systems and method for refining a depth map of a scene based upon a captured image of the scene. A captured depth map of the scene may contain outage areas or other areas of low confidence. The depth map may be aligned with a color image of the scene, and the depth values of the depth map may be adjusted based upon corresponding color values of the color image. An amount of refinement for each depth value of the aligned depth map is based upon the confidence value of the depth value and a smoothing function based upon a corresponding location of the depth value on the color image.

Claims

1. A computer-implemented method comprising:

receiving a color image of a scene;

receiving a depth map of the scene, the depth map comprising one or more depth values, wherein each depth value of the one or more depth values of the depth map indicates a depth of an object in the scene and a location of each depth value of the one or more depth values of the depth map corresponds to a location in the color image; and

iteratively adjusting the one or more depth values of the depth map based on one or more color values of the color image, wherein iteratively adjusting the one or more depth values includes;

adjusting a first depth value of the one or more depth values to a first adjusted depth value during a first iteration of adjusting the one or more depth values, wherein the first depth value is associated with a first confidence value and adjusting the first depth value includes;

determining whether the first confidence value is greater than a threshold;

determining that the first adjusted depth value is based on the first depth value in response to determining that the first confidence value is greater than the threshold; and

determining that the first adjusted depth value is based on an adjusted depth value determined by applying a smoothing filter to the depth map at the location of the first depth value in response to determining that the first confidence value is less than the threshold; and

adjusting the first adjusted depth value to a second adjusted depth value during a second iteration of adjusting the one or more depth values, wherein the first adjusted depth value is associated with a second confidence value and adjusting the first adjusted depth value is further based on the second confidence value;

wherein the first adjusted depth value and the second adjusted depth value are based on a first color value at a first location in the color image corresponding to a location of the first depth value in the depth map.

2. The computer-implemented method of claim 1, wherein adjusting the first adjusted depth value comprises applying the smoothing filter to the depth map at the location of the first adjusted depth value.

3. The computer-implemented method of claim 1, wherein adjusting the first depth value further comprises combining the first depth value and the adjusted depth value.

4. The computer-implemented method of claim 1, wherein the depth map is unaligned with the color image, the method further comprising:

aligning the depth map and the color image; and

determining corresponding locations between the depth map and the color image for each depth value of the one or more depth values.

5. The computer-implemented method of claim 1, further comprising:

projecting, by a transmitter, codewords on to the scene;

receiving, by a receiver, an image of the scene, wherein the image comprises reflections of the projected codewords from the scene;

identifying codewords from the reflections of the projected codewords in the received image;

determining the one or more depth values for the depth map based on the identified codewords; and

capturing, by an image sensor, the color image of the scene.

16. The computer-implemented method of claim 3, wherein combining the first depth value and the adjusted depth value comprises a weighted average of the first depth value and the adjusted depth value, wherein weights of the weighted average are based on the first confidence value.

17. The computer-implemented method of claim 2, wherein adjusting the first depth value further comprises:

determining if the first location in the color image includes an edge; and

preventing the smoothing filter from being applied across the edge.

22. The computer-implemented method of claim 4, wherein aligning the depth map and the color image includes warping the depth map to align with the color image.

23. The computer-implemented method of claim 22, wherein the first confidence value is associated with the warping of the depth map.

24. The computer-implemented method of claim 1, wherein the first confidence value is associated with a tilted surface of the object.

25. The computer-implemented method of claim 1, wherein the first confidence value is associated with a textured surface of the object.

26. The computer-implemented method of claim 1, wherein the first confidence value is associated with one or more shadows cast on the object.

6. An apparatus, comprising:

at least one processor configured to;

receive a color image of a scene;

receive a depth map of the scene, the depth map comprising one or more depth values, wherein each depth value of the one or more depth values of the depth map indicates a depth of an object in the scene and a location of each depth value of the one or more depth values of the depth map corresponds to a location in the color image; and

iteratively adjust the one or more depth values of the depth map based on one or more color values of the color image, wherein iteratively adjusting the one or more depth values includes;

adjusting a first depth value of the one or more depth values to a first adjusted depth value during a first iteration of adjusting the one or more depth values, wherein the first depth value is associated with a first confidence value and adjusting the first depth value includes;

determining whether the first confidence value is greater than a threshold;

determining that the first adjusted depth value is based on the first depth value in response to determining that the first confidence value is greater than the threshold; and

determining that the first adjusted depth value is based on an adjusted depth value determined by applying a smoothing filter to the depth map at the location of the first depth value in response to determining that the first confidence value is less than the threshold; and

adjusting the first adjusted depth value to a second adjusted depth value during a second iteration of adjusting the one or more depth values, wherein the first adjusted depth value is associated with a second confidence value and adjusting the first adjusted depth value is further based on the second confidence value;

wherein the first adjusted depth value and the second adjusted depth value are based on a first color value at a first location in the color image corresponding to a location of the first depth value in the depth map.

7. The apparatus of claim 6, wherein the at least one processor, in adjusting the first adjusted depth value, is configured to apply the smoothing filter to the depth map at the location of the first adjusted depth value.

8. The apparatus of claim 6, wherein the at least one processor, in adjusting the first depth value, is further configured to combine the first depth value and the adjusted depth value.

9. The apparatus of claim 6, wherein:

the depth map is unaligned with the color image; and

the at least one processor is further configured to;

align the depth map and the color image; and

determine corresponding locations between the depth map and the color image for each depth value of the one or more depth values.

10. The apparatus of claim 6, further comprising:

a transmitter configured to project codewords on to the scene;

a receiver configured to receive an image of the scene, wherein the image comprises reflections of the projected codewords from the scene; and

an image sensor configured to capture the color image of the scene;

wherein the at least one processor is further configured to;

identify codewords from the reflections of the projected codewords in the received image; and

determine the one or more depth values for the depth map based on the identified codewords.

18. The apparatus of claim 8, wherein the at least one processor, in combining the first depth value and the adjusted depth value, is further configured to perform a weighted average of the first depth value and the adjusted depth value, wherein weights of the weighted average are based on the first confidence value.

19. The apparatus of claim 7, wherein the at least one processor, in adjusting the first depth value, is further configured to:

determine if the first location in the color image includes an edge; and

prevent the smoothing filter from being applied across the edge.

27. The apparatus of claim 9, wherein the at least one processor, in aligning the depth map and the color image, is configured to warp the depth map to align with the color image.

28. The apparatus of claim 27, wherein the first confidence value is associated with the warping of the depth map.

29. The apparatus of claim 6, wherein the first confidence value is associated with a tilted surface of the object.

30. The apparatus of claim 6, wherein the first confidence value is associated with a textured surface of the object.

31. The apparatus of claim 6, wherein the first confidence value is associated with one or more shadows cast on the object.

11. A non-transitory computer readable medium comprising instructions that when executed cause at least one processor to perform a method, the method comprising:

receiving a color image of a scene;

receiving a depth map of the scene, the depth map comprising one or more depth values, wherein each depth value of the one or more depth values of the depth map indicates a depth of an object in the scene and a location of each depth value of the one or more depth values of the depth map corresponds to a location in the color image; and

iteratively adjusting the one or more depth values of the depth map based on one or more color values of the color image, wherein iteratively adjusting the one or more depth values includes;

adjusting a first depth value of the one or more depth values to a first adjusted depth value during a first iteration of adjusting the one or more depth values, wherein the first depth value is associated with a first confidence value and adjusting the first depth value includes;

determining whether the first confidence value is greater than a threshold;

determining that the first adjusted depth value is based on the first depth value in response to determining that the first confidence value is greater than the threshold; and

determining that the first adjusted depth value is based on an adjusted depth value determined by applying a smoothing filter to the depth map at the location of the first depth value in response to determining that the first confidence value is less than the threshold; and

adjusting the first adjusted depth value to a second adjusted depth value during a second iteration of adjusting the one or more depth values, wherein the first adjusted depth value is associated with a second confidence value and adjusting the first adjusted depth value is further based on the second confidence value;

wherein the first adjusted depth value and the second adjusted depth value are based on a first color value at a first location in the color image corresponding to a location of the first depth value in the depth map.

12. The non-transitory computer readable medium of claim 11, wherein adjusting the first adjusted depth value comprises applying the smoothing filter to the depth map at the location of the first adjusted depth value.

13. The non-transitory computer readable medium of claim 11, wherein adjusting the first depth value further comprises combining the first depth value and the adjusted depth value.

14. The non-transitory computer readable medium of claim 11, wherein:

the depth map is unaligned with the color image; and

execution of the instructions further cause the at least one processor to;

align the depth map and the color image; and

determine corresponding locations between the depth map and the color image for each depth value of the one or more depth values.

15. The non-transitory computer readable medium of claim 11, wherein execution of the instructions causes the apparatus to:

project, by a transmitter, codewords on to the scene;

receive, by a receiver, an image of the scene, wherein the image reflections of the projected codewords from the scene;

identify, by the at least one processor, codewords from the reflections of the projected codewords in the received image;

determine, by the at least one processor, the one or more depth values for the depth map based on the identified codewords; and

capture, by an image sensor, the color image of the scene.

20. The non-transitory computer readable medium of claim 13, wherein combining the first depth value and the adjusted depth value comprises a weighted average of the first depth value and the adjusted depth value, wherein weights of the weighted average are based on the first confidence value.

21. The non-transitory computer readable medium of claim 12, wherein adjusting the first depth value further comprises:

determining if the first location in the color image includes an edge; and

preventing the smoothing filter from being applied across the edge.