Generating 3-dimensional maps of a scene using passive and active measurements

• US 10,024,965 B2
• Filed: 03/31/2016
• Issued: 07/17/2018
• Est. Priority Date: 04/01/2015
• Status: Active Grant

• Current Assignee: Vayavision Sensing Ltd. (LeddarTech, Inc.), VayaVision, Ltd.
• Original Assignee: VayaVision, Ltd.
• Inventors: Nehmadi, Youval, Ur, Shmuel, Cohen, Ronny
• Primary Examiner(s): GORADIA, SHEFALI DINESH
• Application Number:
  US15/087,222
• Publication Number:
  US 20160292905A1
• Time in Patent Office: 838 Days
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: G01C 21/26 specially adapted for navigation in a road network
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 3/08 Use of electric radiation detectors
• CPC Class Codes: G01S 17/08 for measuring distance only indirect measurement G01S17/46; active triangulation systems G01S17/48
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 19/03 Cooperating elements; Interaction or communication between different cooperating elements or between cooperating elements and receivers
• CPC Class Codes: G01S 7/497 Means for monitoring or calibrating
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06K 7/10217 parameter settings controlling the transmission power of the interrogator
• CPC Class Codes: G06K 7/10584 Source control
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10052 : Images from lightfield camera
• CPC Class Codes: G06T 2207/30236 : Traffic on road, railway or crossing
• CPC Class Codes: G06T 2207/30261 : Obstacle
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/141 : Control of illumination
• CPC Class Codes: G06V 10/751 : Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/584 : of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: Y10S 901/01 : Mobile robot

Abstract

A method and apparatus for generating 3D-maps for acquiring three-dimensional (3D) maps are presented. The method includes analyzing at least one image acquired by a passive sensor to identify a plurality of objects in the at least one image; classifying the plurality of objects; determining, based on the classification, whether to passively measure a distance to each of the plurality of objects; passively measuring the distance to at least one of the plurality of objects based on the determination; actively measuring a distance to some of the plurality of objects, wherein the distance to one of the same of the plurality of objects is actively measured when the distance to the object cannot be passively measured; and generating a 3D map of a scene based on the distance measured to each of the plurality of objects.

Claims

1. A method for acquiring three-dimensional (3D) maps, comprising:

analyzing at least one image acquired by a passive sensor to identify a plurality of objects in the at least one image;

classifying the plurality of objects as any of;

a stationary object, a non-stationary object, a suspicious object, and a non-suspicious object;

determining, based on the classification, whether to passively measure a distance to each of the plurality of objects;

passively measuring the distance to at least one of the plurality of objects based on the determination;

wherein the distance to the at least one object is passively measured, for each object classified as a stationary object, when a previous distance to the object is not available;

actively measuring a distance to some of the plurality of objects, wherein the distance to one of the same of the plurality of objects is actively measured when the distance to the object cannot be passively measured; and

generating a 3D map of a scene based on the distance measured to each of the plurality of objects.

2. The method of claim 1, wherein analyzing the at least one image further comprises:

segmenting the at least one image to render a segmentation map; and

identifying the plurality of objects in the segmentation map.

3. The method of claim 1, further comprising:

identifying a type of each of the plurality of objects; and

classifying each of the plurality of objects based on the identified type of the object.

4. The method of claim 1, further comprising:

comparing the at least one image to a previously acquired image; and

identifying a change in a location of each of the plurality of objects based on the comparison, wherein each of the plurality of objects is classified based on the change in the location of the object.

5. The method of claim 1, further comprising:

determining, for each object of the plurality of objects, whether the object is identified in at least one geographic map; and

classifying each object identified in the at least one geographic map as a stationary object.

6. The method of claim 1, wherein the passively measuring the distance to the at least one object further comprising:

determining if the passive sensor is in motion;

determining the distance to the least one object using a motion sensor, when the passive sensor is in motion; and

determining the distance to the least one object based on the number of the pixels to the object and a known reference measurement, when the passive sensor is not in motion.

7. The method of claim 6, wherein the known reference measurement is at least one of:

a number of pixels from a valid reference object, and a dimension of the object.

8. The method of claim 1, wherein passively measuring the distance further comprising, for each object classified as a non-stationary object and as a non-suspicious object:

determining a boundary speed of the object;

determining a direction of movement of the object; and

estimating the distance to the object based on the determined direction and the determined boundary speed.

9. The method of claim 8, further comprising:

determining if the estimated distance is below a safe distance threshold; and

actively measuring the distance to the at least one object, when the estimated distance is below the safe distance threshold.

10. The method of claim 1, wherein the distance is actively measured to any object classified as any of:

a non-stationary object, and a suspicious object.

11. The method of claim 1, wherein the distance to one of the plurality of objects is actively measured when a reliable passive measurement for the object cannot be determined.

12. The method of claim 1, wherein actively measuring the distance to some of the plurality of objects further comprises:

performing a number of laser measurements for each actively measured object, wherein the number of laser measurements is determined based on a required resolution.

13. The method of claim 12, further comprising:

computing, for each actively measured object, a distance to each point in the object based on the laser measurements.

14. The method of claim 1, wherein generating the 3D-map further comprising:

rendering a 3D-surface of each object using the distance measurements.

15. The method of claim 1, wherein the scene is a field of view captured by an apparatus for causing generation of the 3D-map.

16. A non-transitory computer-readable medium having stored thereon instructions for executing the method according to claim 1.

17. An apparatus for acquiring three-dimensional (3D) maps, comprising:

a passive sensor for acquiring images of a scene;

an active sensor for measuring distances from objects in the scene; and

a processing system, wherein the processing system is configured to;

analyze at least one image acquired by the passive sensor to identify a plurality of objects in the at least one image;

classify the plurality of objects as any of;

a stationary object, a non-stationary object, a suspicious object, and a non-suspicious object;

determine, based on the classification, whether to passively measure a distance to each of the plurality of objects;

passively measure the distance to at least one of the plurality of objects based on the determination;

wherein the distance to the at least one object is passively measured, for each object classified as a stationary object, when a previous distance to the object is not available;

actively measure, by the active sensor, a distance to some of the plurality of objects, wherein the distance to one of the plurality of objects is actively measured when the distance to the object cannot be passively measured; and

generate a 3D map of a scene based on the distance measured to each of the plurality of objects.

18. A method for acquiring three-dimensional (3D) maps, comprising:

acquiring at least one image by a passive sensor;

segmenting the at least one image to render a segmentation map;

classifying a plurality of pixels in each segment identified in the segmentation map as any of;

a stationary pixel, a non-stationary pixel, a suspicious pixel, and a non-suspicious pixel;

determining, based on the classification, whether to passively measure a distance to each of the plurality of pixels;

passively measuring the distance to at least one of the plurality of pixels based on the determination;

wherein the distance to the at least one pixel is passively measured, for each pixel classified as a stationary pixel, when a previous distance to the pixel is not available;

actively measuring a distance value to each pixel of a first number of pixels in each segment identified in the segmentation map, wherein the first number of pixels is less than the number of pixels in each segment;

computing a distance value to each pixel in the segment using the distance of pixel in the first number of pixels; and

generating a 3D map of a scene based on the measured and computed distance values.