3D depth map

• US 10,795,022 B2
• Filed: 05/09/2017
• Issued: 10/06/2020
• Est. Priority Date: 03/02/2017
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Shintani, Peter, Usami, Morio, Matsumoto, Kissei, Shikama, Kazuyuki, Mohapatra, Bibhudendu, Resch, Keith
• Primary Examiner(s): Perlman, David
• Application Number:
  US15/590,497
• Publication Number:
  US 20180252815A1
• Time in Patent Office: 1,246 Days
• Field of Search: None
• CPC Class Codes: G01S 17/10 using transmission of interrupted, pulse-modulated waves determination of distance by phase measurements G01S17/32
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

Machine learning is applied to both 2D images from an infrared imager imaging laser reflections from an object and to the 3D depth map of the object that is generated using the 2D images and time of flight (TOF) information. In this way, the 3D depth map accuracy can be improved without increasing laser power or using high resolution imagers.

Claims

1. A first device comprising:

at least one processor configured with instructions to;

generate at least one image of a second device to render a first three dimensional (3D) depth map;

provide the first 3D depth map to an aggregator device configured to aggregate the first 3D depth map with a second 3D depth map generated by the second device using an image of the first device taken by the second device such that the first and second depth maps when aggregated include both the first and second devices;

classify a representation of at least a first object in at least one image used to render the first 3D depth map using a machine learning algorithm (MLA) to render a classification; and

use the classification to modify the representation.

2. The first device of claim 1, wherein the aggregator device comprises at least one of the first or second devices.

3. The first device of claim 1, wherein the aggregator device does not comprise either the first or second device.

4. The first device of claim 1, wherein the instructions are executable to generate the first 3D depth map at least in part by:

inputting to a machine learning algorithm (MLA) a training set of classified images; and

modifying MLA processing by inputting to the MLA a test set of unclassified images.

5. The first device of claim 1, comprising the aggregator device programmed with instructions to aggregate the first 3D depth map with the second 3D depth map such that the first and second depth maps when aggregated include both the first and second devices.

6. An assembly comprising:

plural motorized computerized devices in a system;

each motorized computerized device comprising at least one camera and at least one emitter configured to output signals useful for generating at least one three dimensional (3D) depth map by imaging an object to render a two dimensional (2D) image thereof, illuminating the object using the emitter, receiving time of flight (TOF) information using reflections from the object of light emitted by the emitter, and combining the 2D image with the TOF information to render the 3D depth map;

at least a first one of the motorized computerized devices being programmed with instructions to;

render an aggregated three dimensional (3D) depth map that is aggregated from the 3D depth maps from the motorized computing device, the aggregated depth map comprising images of plural of the motorized devices;

classify a representation of at least a first object in at least one image used to render the first 3D depth map using a machine learning algorithm (MLA) to render a classification; and

use the classification to modify the representation.

7. The assembly of claim 6, wherein the 2D image is combined with the TOF information on a pixel-by-pixel basis to render the 3D depth map.

8. The assembly of claim 6, wherein the instructions are executable to:

input to the MLA a training set of classified images; and

modify MLA processing by inputting to the MLA a test set of unclassified images.

9. A method comprising:

using an illuminator, illuminating an object;

using a camera, receiving reflections from the object;

using the reflections, establishing a representation of the object using a first device;

based at least in part on the representation, establish a first three dimensional (3D) depth map of the object; and

provide the first 3D depth map to an aggregator device that aggregates the first 3D depth map with a second 3D depth map generated by a second device, the 3D depth aggregated from the first and second 3D depth maps comprising images of the first and second devices, wherein the method includes;

classifying a representation of at least a first object in at least one image used to render the first 3D depth map using a machine learning algorithm (MLA) to render a classification; and

using the classification to modify the representation.

10. The method of claim 9, comprising modifying the representation using a neural network to render a modified representation using at least one computer processor.

11. The method of claim 9, wherein the representation is a two dimensional (2D) image of the object.

12. The method of claim 9, wherein the representation is a three dimensional (3D) depth map of the object.

13. The method of claim 10, comprising:

inputting to the neutral network a training set of classified images; and

modifying the neural network processing by inputting to the neural network a test set of unclassified images.