Color noise reduction in 3D depth map

• US 10,536,684 B2
• Filed: 12/07/2016
• Issued: 01/14/2020
• Est. Priority Date: 12/07/2016
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Shintani, Peter, Gonzalez Hernandez, Jose Omar, Clay, William, Espinosa, Pablo Antonio, Ansfield, Fred, Mohapatra, Bibhudendu, Resch, Keith, Usami, Morio
• Primary Examiner(s): Perez-Fuentes, Luis
• Application Number:
  US15/371,433
• Publication Number:
  US 20180160094A1
• Time in Patent Office: 1,133 Days
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/15 for colour aspects of image signals

Abstract

A camera based depth mapping system. Depth information is coded with colors to make a laser-generated 3D depth map easier to interpret. In the event that the laser illumination is not sufficient, the depth information can have a low signal to noise ratio, i.e., the depth map can be noisy. Color noise reduction techniques are used to alleviate this problem.

Claims

1. A device comprising:

at least one computer medium that is not a transitory signal and that comprises instructions executable by at least one processor to;

generate, using at least one laser, a three dimensional (3D) depth map;

apply color noise reduction to the depth map to increase a signal-to-noise ratio (SNR) of the depth map compared to a SNR of the depth map prior to application of noise reduction;

access a transmission schedule; and

transmit laser range-finding beams using the at least one laser at transmission times according to the transmission schedule to generate the depth map.

2. The device of claim 1, comprising the at least one processor.

3. The device of claim 2, comprising at least one camera providing signals to the at least one processor representative of reflections of the laser range-finding beams.

4. The device of claim 1, wherein the instructions are executable to synchronize at least one computer clock of the device with a heartbeat signal of a master device in an optical micromesh that includes the device and at least one other device also synchronized to the heartbeat signal, transmission of laser range-finding beams according to the transmission schedule being executed at least in part by accessing the computer clock.

5. The device of claim 1, wherein the instructions are executable to assign slots in the transmission schedule to other devices in an optical micromesh.

6. The device of claim 1, wherein the depth map is of at least one object, and the instructions are executable to:

access a color image of the object; and

based at least in part on the color image, augment the depth map with color.

7. The device of claim 1, wherein the depth map is of at least one object, and the instructions are executable to:

access default colors; and

based at least in part on the defeat colors, augment the depth map with color.

8. An assembly comprising:

plural computerized devices;

each computerized device comprising at least one laser emitter configured to output signals useful for generating at least one three dimensional (3D) depth map; and

at least one processor programmed with instructions to augment at least one of the 3D depth maps with color and to apply noise reduction to 3D depth maps augmented with color to reduce noise in the depth map augmented with color.

9. The assembly of claim 8, wherein the devices are programmed with instructions to emit laser beams to establish the at least one 3D depth map using time division emission rules.

10. The assembly of claim 8, comprising at least one camera providing signals to at least one processor representative of reflections of the laser range-finding beams.

11. The assembly of claim 8, wherein at least one of the depth maps is of at least one object, and the instructions are executable to:

access a color image of the object; and

based at least in part on the color image, augment the depth map with color.

12. The assembly of claim 8, wherein at least one of the depth maps is of at least one object, and the instructions are executable to:

access default colors; and

based at least in part on the defeat colors, augment the depth map with color.

13. A method comprising:

emitting laser emissions;

using reflections of the emissions, generating a three dimensional (3D) depth map; and

applying color to the 3D depth map to render an augmented depth map by associating a first color with a first region of the 3D depth map representing a “

closest”

surface and associating a second color with at least a second region of the 3D depth map on the basis that the first color is warmer than the second color.

14. The method of claim 13, comprising:

accessing a transmission schedule; and

transmitting the laser emissions at transmission times according to the transmission schedule.

15. The method of claim 13, comprising receiving signals from at least one camera representative of reflections of the laser emissions.

16. The method of claim 14, comprising synchronizing at least one computer clock with a heartbeat signal of a master device in an optical micromesh, transmission of laser range-finding beams according to the transmission schedule being executed at least in part by accessing the computer clock.

17. The method of claim 14, comprising assigning slots in the transmission schedule to other devices in an optical micromesh.

18. The method of claim 13, wherein the depth map is of at least one object, and the method comprises:

accessing a color image of the object; and

based at least in part on the color image, augmenting the depth map with color.

19. The method of claim 13, wherein the depth map is of at least one object, and the method comprises:

accessing default colors; and

based at least in part on the defeat colors, augmenting the depth map with color.