Method and system for generating multidimensional maps of a scene using a plurality of sensors of various types

• US 10,445,928 B2
• Filed: 02/08/2018
• Issued: 10/15/2019
• Est. Priority Date: 02/11/2017
• Status: Active Grant

• Current Assignee: Vayavision Sensing Ltd. (LeddarTech, Inc.), VayaVision, Ltd.
• Original Assignee: VayaVision, Ltd.
• Inventors: Nehmadi, Youval, Mangan, Shmuel, Ben-Ezra, Shahar, Cohen, Anna, Cohen, Ronny, Goldentouch, Lev, Ur, Shmuel
• Primary Examiner(s): Nguyen, Phu K
• Application Number:
  US15/892,039
• Publication Number:
  US 20180232947A1
• Time in Patent Office: 614 Days
• Field of Search: 345418
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/89 for mapping or imaging
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 2013/9323 Alternative operation using light waves
• CPC Class Codes: G01S 2013/93271 in the front of the vehicles
• CPC Class Codes: G01S 2013/93274 on the side of the vehicles
• CPC Class Codes: G01S 2013/93276 in the windshield area
• CPC Class Codes: G01S 7/295 Means for transforming co-ordinates or for evaluating data, e.g. using computers
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2210/61 Scene description
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source

Abstract

A system and method for generating a high-density three-dimensional (3D) map are disclosed. The system comprises acquiring at least one high density image of a scene using at least one passive sensor; acquiring at least one new set of distance measurements of the scene using at least one active sensor; acquiring a previously generated 3D map of the scene comprising a previous set of distance measurements; merging the at least one new set of distance measurements with the previous set of upsampled distance measurements, wherein merging the at least one new set of distance measurements further includes accounting for a motion transformation between a previous high-density image frame and the acquired high density image and the acquired distance measurements; and overlaying the new set of distance measurements on the high-density image via an upsampling interpolation, creating an output 3D map.

Claims

1. A method for generating a high-density three-dimensional (3D) map, comprising:

acquiring at least one high-density image of a scene using at least one passive sensor;

acquiring at least one new set of distance measurements of the scene using at least one active sensor;

acquiring a previously generated 3D map of the scene comprising a previous set of upsampled distance measurements;

merging the at least one new set of distance measurements with the previous set of upsampled distance measurements, wherein merging the at least one new set of distance measurements further includes accounting for a motion transformation between a previous high-density image frame and the acquired high-density image and the acquired new set of distance measurements; and

overlaying the merged new set of distance measurements on the acquired high-density image via an upsampling interpolation, creating an output high-density 3D map.

2. The method of claim 1, wherein the upsampling interpolation process comprises:

defining a visual similarity metric for a tested pixel within the high-density image, wherein the visual similarity metric is employed to weigh the significance of neighboring pixels based on visual similarity to the tested pixel;

defining an angular distance metric, wherein the angular distance metric is employed to weigh the signification of neighboring pixels based on their angular distance from the tested pixel;

generating an optimal model of the scene based on the visual similarity metric and the angular similarity metric;

estimating 3D coordinates based on the generated optimal model.

3. The method of claim 1, wherein the active sensor includes at least one of:

a LiDAR and a radar.

4. The method of claim 3, wherein two or more active sensors are used, each active sensor having different sensor characteristics, wherein acquiring at least one set of distance measurements comprises:

identifying a point of interest within the high-density 3D map;

determining a performance advantage of each active sensor with regard to each identified point of interest based on the different characteristics;

determine an optimal method of acquire the measurements by the sensors;

employing the active sensors in the optimal method to provide a performance advantage when acquiring distance measurements for the point of interest.

5. The method of claim 4, wherein characteristics of the at least one active sensor include at least one of:

field of view, beam size, angular position accuracy, target relative speed, dead time between measurements;

amount of saturation captured with a direct light source, and degradation of image quality in diminished visibility conditions.

6. The method of claim 4, wherein identifying a point of interest further comprises at least one of:

detecting isolated objects based on unsupervised 3D processing or supervised trained neural network;

detecting stationary objects within the high density 3D map;

detecting moving objects within the high density 3D map;

tracking movements of objects from a first high density 3D map to second high density 3D map;

identify the angular region enclosing the road behind a specific distance;

identify the angular region enclosing groups of pedestrians;

identify the angular region enclosing low-light area;

identify the angular region enclosing saturated light areas;

detecting edges in the high density 3D map;

detecting patches in the high density 3D map;

detecting visually distinguishable segments in the high density three dimensional map and detecting super-pixels in the high density three dimensional map.

7. The method of claim 1, wherein acquiring the previously generated 3D map further comprises:

retrieving the previously generated 3D map from a storage.

8. The method of claim 1, wherein merging the at least one new set of distance measurements further comprises:

aligning the new set of distance measurements with the previous set of distance measurements to compensate for motion between the two sets.

9. The method of claim 8, wherein aligning the new set of distance measurements further comprises:

adjusting the previously generated 3D map using a motion transformation based on at least one of;

the momentary 6-axis motion between captured high-density images, a speed vector, an acceleration vector, and the difference of the time stamps between the previous set of distance measurements and the new set of distance measurements.

10. A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to perform a process for generating a high-density three-dimensional (3D) map, the process comprising:

acquiring at least one high density image of a scene using at least one passive sensor;

acquiring at least one new set of distance measurements of the scene using at least one active sensor;

acquiring a previously generated 3D map of the scene comprising a previous set of distance measurements;

merging the at least one new set of distance measurements with the previous set of upsampled distance measurements, wherein merging the at least one new set of distance measurements further includes accounting for a motion transformation between a previous high-density image frame and the acquired high density image and the acquired distance measurements; and

overlaying the merged new set of distance measurements on the high-density image via an upsampling interpolation, creating an output 3D map.

11. A system for generating a high-density three-dimensional (3D) map, comprising:

a passive sensor configured to acquire at least one high density image of a scene using a passive sensor;

at least one active sensor configured to acquire at least one set of distance measurements of the scene using at least one active sensor;

a processing circuitry; and

a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to;

acquire at least one high density image of a scene using at least one passive sensor;

acquire at least one new set of distance measurements of the scene using at least one active sensor;

acquire a previously generated 3D map of the scene comprising a previous set of distance measurements;

merge the at least one new set of distance measurements with the previous set of upsampled distance measurements, wherein merging the at least one new set of distance measurements further includes accounting for a motion transformation between a previous high-density image frame and the acquired high density image and the acquired distance measurements; and

overlay the merged new set of distance measurements on the high-density image via an upsampling interpolation, creating an output high-density 3D map.

12. The system of claim 11, wherein the system is further configured:

define a visual similarity metric for a tested pixel within the high-density image, wherein the visual similarity metric is employed to weigh the significance of neighboring pixels based on visual similarity to the tested pixel;

define an angular distance metric, wherein the angular distance metric is employed to weigh the signification of neighboring pixels based on their angular distance from the tested pixel;

generate an optimal model of the scene based on the visual similarity metric and the angular similarity metric;

estimate 3D coordinates based on the generated optimal model.

13. The system of claim 11, wherein the active sensor includes at least one of:

a LiDAR and a radar.

14. The system of claim 13, wherein two or more active sensors are used, each active sensor having different sensor characteristics.

15. The system of claim 14, wherein the system is further configured to:

identify a point of interest within the high-density image;

determine a performance advantage of each active sensor with regard to each identified point of interest based on the different characteristics;

employ the active sensor determined to provide a performance advantage when acquiring distance measurements for the point of interest.

16. The system of claim 15, wherein characteristics of the at least one active sensor include:

field of view, beam size, angular position accuracy, target relative speed, dead time between measurements;

amount of saturation captured with a direct light source, and degradation of image quality in diminished visibility conditions.

17. The system of claim 15, wherein identifying a point of interest further comprises at least one of:

detecting isolated objects based on unsupervised 3D processing or supervised trained neural network;

detecting stationary objects within the high density 3D map;

detecting moving objects within the high density 3D map;

tracking movements of objects from a first high density 3D map to second high density 3D map;

identify the angular region enclosing the road behind a specific distance;

identify the angular region enclosing groups of pedestrians;

identify the angular region enclosing low-light area;

identify the angular region enclosing saturated light areas;

detecting edges in the high density 3D map;

detecting patches in the high density 3D map;

detecting visually distinguishable segments in the high density 3D map and detecting super-pixels in the high density 3D map.

18. The system of claim 11, wherein the system is further configured to:

retrieve the previously generated 3D map from a storage.

19. The system of claim 18, wherein the system is further configured to:

align the new set of distance measurements with the previous set of distance measurements to compensate for motion between the two sets.

20. The system of claim 19, wherein the system is further configured to:

adjust the previously generated 3D map using a motion transformation based on at least one of;

the momentary 6-axis motion between captured high-density images, a speed vector, an acceleration vector, and the difference of the time stamps between the previous set of distance measurements and the new set of distance measurements.