3D Depth Map

• US 20180252815A1
• Filed: 05/09/2017
• Published: 09/06/2018
• Est. Priority Date: 03/02/2017
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Shintani, Peter, Usami, Morio, Matsumoto, Kissei, Shikama, Kazuyuki, Mohapatra, Bibhudendu, Resch, Keith
• Granted Patent: US 10,795,022 B2
• Time in Patent Office: Days
• CPC Class Codes: G01S 17/10 using transmission of interrupted, pulse-modulated waves determination of distance by phase measurements G01S17/32
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

Machine learning is applied to both 2D images from an infrared imager imaging laser reflections from an object and to the 3D depth map of the object that is generated using the 2D images and time of flight (TOF) information. In this way, the 3D depth map accuracy can be improved without increasing laser power or using high resolution imagers.

Claims

1. A first device comprising:

at least one computer medium that is not a transitory signal and that comprises instructions executable by at least one processor to;

access at least one machine learning algorithm (MLA);

process at least one representation of at least one object using the MLA to render an output;

use the output to render a first three dimensional (3D) depth map; and

provide the first 3D depth map to an aggregator device that aggregates the first 3D depth map with a second 3D depth map generated by a second device, the first and second 3D depth maps being aggregated together for a reference location.

2. The first device of claim 1, wherein the representation is a two dimensional (2D) image of the object.

3. The first device of claim 1, wherein the representation is a three dimensional (3D) depth map of the object.

4. The first device of claim 1, wherein the instructions are executable to:

input to the MLA a training set of classified images; and

modify MLA processing by inputting to the MLA a test set of unclassified images.

5. The first device of claim 1, comprising the at least one processor.

6. The first device of claim 1, wherein the instructions are executable to:

classify the representation of the object using the MLA to render a classification; and

use the classification to modify the representation.

7. An assembly comprising:

plural motorized computerized devices;

each motorized computerized device comprising at least one laser emitter configured to output signals useful for generating at least one three dimensional (3D) depth map;

a first one of the motorized computerized devices being programmed with instructions to;

access at least one machine learning algorithm (MLA);

process at least one representation of at least one object using the MLA to render an output; and

use the output to render a three dimensional (3D) depth map.

8. The assembly of claim 7, wherein the representation is a two dimensional (2D) image of the object.

9. The assembly of claim 7, wherein the representation is a three dimensional (3D) depth map of the object.

10. The assembly of claim 7, wherein the instructions are executable to:

input to the MLA a training set of classified images; and

modify MLA processing by inputting to the MLA a test set of unclassified images.

11. The assembly of claim 7, wherein the instructions are executable to:

classify the representation of the object using the MLA to render a classification; and

use the classification to modify the representation.

12. A method comprising:

using a laser, illuminating an object;

using a camera, receiving reflections from the object;

using the reflections, establishing a representation of the object;

modifying the representation using a neural network to render a modified representation; and

using the modified representation, establish a three dimensional (3D) depth map of the object.

13. The method of claim 12, wherein the modifying and using are executed by at least one computer processor.

14. The method of claim 12, wherein the representation is a two dimensional (2D) image of the object.

15. The method of claim 12, wherein the representation is a three dimensional (3D) depth map of the object.

16. The method of claim 12, comprising:

inputting to the neutral network a training set of classified images; and

modifying the neural network processing by inputting to the neural network a test set of unclassified images.

17. The method of claim 12, comprising:

classifying the representation of the object using the neural network to render a classification; and

using the classification to modify the representation.