Systems and methods for refining landmark positions

• US 9,939,813 B2
• Filed: 09/22/2016
• Issued: 04/10/2018
• Est. Priority Date: 02/10/2015
• Status: Active Grant

• Current Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Original Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Inventors: Shashua, Amnon, Gdalyahu, Yoram, Braunstein, Daniel
• Primary Examiner(s): Tissot, Adam D
• Assistant Examiner(s): BERNS, MICHAEL ANDREW
• Application Number:
  US15/273,190
• Publication Number:
  US 20170010121A1
• Time in Patent Office: 565 Days
• CPC Class Codes: B60W 2420/403 Image sensing, e.g. optical camera
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2552/30 Road curve radius
• CPC Class Codes: B60W 2552/53 Road markings, e.g. lane marker or crosswalk
• CPC Class Codes: B60W 2554/20 Static objects
• CPC Class Codes: B60W 2554/40 Dynamic objects, e.g. animals, windblown objects
• CPC Class Codes: B60W 2554/60 Traversable objects, e.g. speed bumps or curbs
• CPC Class Codes: B60W 2555/60 Traffic rules, e.g. speed limits or right of way
• CPC Class Codes: B60W 2556/35 Data fusion
• CPC Class Codes: B60W 2556/40 High definition maps
• CPC Class Codes: B60W 2556/45 External transmission of data to or from the vehicle
• CPC Class Codes: B60W 2556/50 of positioning data, e.g. GPS [Global Positioning System] data
• CPC Class Codes: B60W 2710/18 Braking system
• CPC Class Codes: B60W 2710/20 Steering systems
• CPC Class Codes: B60W 2720/10 Longitudinal speed
• CPC Class Codes: B60W 30/10 Path keeping cruise control for automatically following a preceding vehicle B60W30/165
• CPC Class Codes: B60W 30/14 Adaptive cruise control
• CPC Class Codes: B60W 30/18 Propelling the vehicle
• CPC Class Codes: B60W 40/06 Road conditions
• CPC Class Codes: B60W 60/00 Drive control systems specially adapted for autonomous road vehicles
• CPC Class Codes: B60W 60/0015 : specially adapted for safety
• CPC Class Codes: B60W 60/00274 : considering possible movement changes
• CPC Class Codes: B62D 15/025 : Active steering aids, e.g. helping the driver by actively influencing the steering system after environment evaluation B62D1/28 takes precedence
• CPC Class Codes: G01C 21/14 : by recording the course traversed by the object G01C21/16 takes precedence
• CPC Class Codes: G01C 21/1652 : with ranging devices, e.g. LIDAR or RADAR
• CPC Class Codes: G01C 21/1656 : with passive imaging devices, e.g. cameras
• CPC Class Codes: G01C 21/20 : Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/30 : Map- or contour-matching
• CPC Class Codes: G01C 21/32 : Structuring or formatting of map data
• CPC Class Codes: G01C 21/34 : Route searching; Route guidance
• CPC Class Codes: G01C 21/3407 : specially adapted for specific applications
• CPC Class Codes: G01C 21/3476 : using point of interest [POI] information, e.g. a route passing visible POIs
• CPC Class Codes: G01C 21/36 : Input/output arrangements for on-board computers
• CPC Class Codes: G01C 21/3602 : Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01C 21/3623 : using a camera or code reader, e.g. for optical or magnetic codes
• CPC Class Codes: G01C 21/3644 : Landmark guidance, e.g. using POIs or conspicuous other objects
• CPC Class Codes: G01C 21/3658 : Lane guidance
• CPC Class Codes: G01C 21/3691 : Retrieval, searching and output of information related to real-time traffic, weather, or environmental conditions arrangements for giving variable traffic instructions G08G1/09
• CPC Class Codes: G01C 21/3811 : Point data, e.g. Point of Interest [POI]
• CPC Class Codes: G01C 21/3819 : Road shape data, e.g. outline of a route
• CPC Class Codes: G01C 21/3822 : Road feature data, e.g. slope data
• CPC Class Codes: G01C 21/3841 : Data obtained from two or more sources, e.g. probe vehicles
• CPC Class Codes: G01C 21/3896 : Transmission of map data from central databases
• CPC Class Codes: G01S 13/867 : Combination of radar systems with cameras
• CPC Class Codes: G01S 19/10 : providing dedicated supplementary positioning signals
• CPC Class Codes: G01S 2013/93271 : in the front of the vehicles
• CPC Class Codes: G01S 5/0027 : of actual mobile position, i.e. position determined on mobile
• CPC Class Codes: G01S 5/16 : using electromagnetic waves other than radio waves
• CPC Class Codes: G05D 1/0088 : characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0212 : with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0219 : ensuring the processing of the whole working surface
• CPC Class Codes: G05D 1/0221 : involving a learning process
• CPC Class Codes: G05D 1/0246 : using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 : extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0253 : extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G05D 1/0278 : using satellite positioning signals, e.g. GPS
• CPC Class Codes: G05D 1/0287 : involving a plurality of land vehicles, e.g. fleet or convoy travelling traffic control systems for road vehicles G08G1/00, particularly anticollision systems G08G1/16
• CPC Class Codes: G06F 16/2379 : Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/29 : Geographical information databases
• CPC Class Codes: G06T 2207/20081 : Training; Learning
• CPC Class Codes: G06T 2207/30256 : Lane; Road marking
• CPC Class Codes: G06T 2207/30261 : Obstacle
• CPC Class Codes: G06T 7/00 : Image analysis
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/58 : Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/582 : of traffic signs
• CPC Class Codes: G06V 20/584 : of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/588 : Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 20/63 : Scene text, e.g. street names
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G08G 1/0112 : from the vehicle, e.g. floating car data [FCD]
• CPC Class Codes: G08G 1/0129 : for creating historical data or processing based on historical data
• CPC Class Codes: G08G 1/0141 : for traffic information dissemination
• CPC Class Codes: G08G 1/0145 : for active traffic flow control
• CPC Class Codes: G08G 1/09623 : Systems involving the acquisition of information from passive traffic signs by means mounted on the vehicle G08G1/0967 takes precedence
• CPC Class Codes: G08G 1/096725 : where the received information generates an automatic action on the vehicle control
• CPC Class Codes: G08G 1/096805 : where the transmitted instructions are used to compute a route
• CPC Class Codes: G08G 1/167 : Driving aids for lane monitoring, lane changing, e.g. blind spot detection
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04W 4/44 : for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]

Abstract

A system is provided for determining a location of a landmark for use in navigation of an autonomous vehicle. The system includes a processor programmed to receive a measured position of the landmark. The processor is also programmed to determine a refined position of the landmark based on the measured position of the landmark and at least one previously acquired position for the landmark. The measured position and the previously acquired position are determined based on acquisition of an environmental image associated with the host vehicle, analysis of the environmental image to identify the landmark, reception of global positioning system (GPS) data representing a location of the host vehicle, analysis of the environmental image to determine a relative position of the identified landmark with respect to the host vehicle, and determination of a globally localized position of the landmark based on the GPS data and the relative position.

Claims

1. A system for determining a location of a landmark for use in navigation of an autonomous vehicle, the system comprising:

at least one processor programmed to;

receive a measured position of the landmark;

determine a refined position of the landmark based on the measured position of the landmark and at least one previously acquired position for the landmark, wherein the measured position and the at least one previously acquired position are determined based on acquisition, from a camera associated with a host vehicle, of at least one environmental image associated with the host vehicle, analysis of the at least one environmental image to identify the landmark in the environment of the host vehicle, reception of global positioning system (GPS) data representing a location of the host vehicle, analysis of the at least one environmental image to determine a relative position of the identified landmark with respect to the host vehicle, and determination of a globally localized position of the landmark based on at least the GPS data and the determined relative position;

selectively preclude transmission of the refined position of the landmark or the measured position of the landmark to a server, the server storing a sparse data model including landmark position information, based on at least one value indicative of a determined confidence level; and

selectively preclude transmission of a distance between two measured landmark positions to the server based on the at least one value indicative of the determined confidence level.

2. The system of claim 1, wherein the landmark includes at least one of a traffic sign, an arrow, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a landmark beacon, or a lamppost.

3. The system of claim 1, wherein analysis of the at least one image to determine the relative position of the identified landmark with respect to the vehicle includes calculating a distance based on a scale associated with the at least one image.

4. The system of claim 1, wherein analyzing the at least one image to determine the relative position of the identified landmark with respect to the vehicle includes calculating a distance based on an optical flow associated with at least two images.

5. The system of claim 1, wherein the GPS data is received from a GPS device included in the host vehicle.

6. The system of claim 1, wherein the camera is included in the host vehicle.

7. The system of claim 1, wherein determining the refined position of the landmark includes averaging the measured position of the landmark with the at least one previously acquired position.

8. The system of claim 1, wherein the landmark is associated with a road segment, and the at least one value indicative of the determined confidence level is associated with the road segment.

9. The system of claim 1, wherein the landmark is associated with a geographical region, and the at least one value indicative of the determined confidence level is associated with the geographical region.

10. The system of claim 1, wherein the landmark is associated with a local map area, and the at least one value indicative of the determined confidence level is associated with the local map area.

11. The system of claim 1, wherein the at least one processor is further programmed to receive the at least one value indicative of the determined confidence level from the server.

12. The system of claim 1, wherein the server is configured to selectively control data flow from the autonomous vehicle based on the at least one value indicative of the determined confidence level.

13. The system of claim 1, wherein the sparse data model has a landmark density of no more than twenty landmarks per kilometer.

14. The system of claim 1, wherein the server is configured to receive additional information from the autonomous vehicle other than the refined position of the landmark and the measured position of the landmark.

15. The system of claim 14, wherein the additional information includes an image captured by an image capturing device.

16. The system of claim 1, wherein the at least one processor is included in the autonomous vehicle.

17. The system of claim 1, wherein the at least one processor is included in the server.

18. A method for determining a location of a landmark for use in navigation of an autonomous vehicle, the method comprising:

receiving a measured position of the landmark;

determining a refined position of the landmark based on the measured position of the landmark and at least one previously acquired position for the landmark, wherein the measured position and the at least one previously acquired position are determined based on;

acquisition, from a camera associated with a host vehicle, of at least one environmental image associated with the host vehicle,

analysis of the at least one environmental image to identify the landmark in the environment of the host vehicle,

reception of global positioning system (GPS) data representing a location of the host vehicle,

analysis of the at least one environmental image to determine a relative position of the identified landmark with respect to the host vehicle, and

determination of a globally localized position of the landmark based on at least the GPS data and the determined relative position;

selectively precluding transmission of the refined position of the landmark or the measured position of the landmark to a server, the server storing a sparse data model including landmark position information, based on at least one value indicative of a determined confidence level; and

selectively preclude transmission of a distance between two measured landmark positions to the server based on the at least one value indicative of the determined confidence level.

19. The method of claim 18, wherein the landmark includes at least one of a traffic sign, an arrow, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a landmark beacon, or a lamppost.

20. The method of claim 18, wherein analysis of the at least one image to determine the relative position of the identified landmark with respect to the vehicle includes calculating a distance based on a scale associated with the at least one image.

21. The method of claim 18, wherein analysis of the at least one image to determine the relative position of the identified landmark with respect to the vehicle includes calculating a distance based on an optical flow associated with at least two images.

22. The method of claim 18, wherein the GPS data is received from a GPS device included in the host vehicle.

23. The method of claim 18, wherein the camera is included in the host vehicle.

24. The method of claim 18, wherein determining the refined position of the landmark includes averaging the measured position of the landmark with the at least one previously acquired position.

25. The method of claim 18, wherein the method is performed by at least one processor included in the autonomous vehicle.

26. The method of claim 18, wherein the method is performed by at least one processor included in the server.

27. An autonomous vehicle, comprising:

a body; and

at least one processor programmed to;

receive a measured position of a landmark;

determine a refined position of the landmark based on the measured position of the landmark and at least one previously acquired position for the landmark, wherein the at least one processor is further programmed to determine the measured position and the at least one previously acquired position based on acquisition, from a camera associated with the vehicle, of at least one environmental image associated with the vehicle, analysis of the at least one environmental image to identify the landmark in the environment of the vehicle, reception of global positioning system (GPS) data representing a location of the vehicle, analysis of the at least one environmental image to determine a relative position of the identified landmark with respect to the vehicle, and determination of a globally localized position of the landmark based on at least the GPS data and the determined relative position;

selectively preclude transmission of the refined position of the landmark to a server, the server storing a sparse data model including landmark position information, based on at least one value indicative of a determined confidence level; and

selectively preclude transmission of a distance between two measured landmark positions to the server based on the at least one value indicative of the determined confidence level.

28. The vehicle of claim 27, wherein the landmark includes at least one of a traffic sign, an arrow, a lane marking, a dashed lane marking, a traffic light, a stop line, a directional sign, a landmark beacon, or a lamppost.

29. The vehicle of claim 27, wherein analysis of the at least one image to determine the relative position of the identified landmark with respect to the vehicle includes calculating a distance based on a scale associated with the at least one image.

30. The vehicle of claim 27, wherein analyzing the at least one image to determine the relative position of the identified landmark with respect to the vehicle includes calculating a distance based on an optical flow associated with at least two images.

31. The vehicle of claim 27, wherein the GPS data is received from a GPS device included in the host vehicle.

32. The vehicle of claim 27, wherein determining the refined position of the landmark includes averaging the measured position of the landmark with the at least one previously acquired position.

33. The vehicle of claim 27, wherein the at least one processor is further programmed to receive the at least one value indicative of the determined confidence level from the server.

34. The vehicle of claim 27, wherein the sparse data model has a landmark density of no more than twenty landmarks per kilometer.