VISUAL PLACE RECOGNITION BASED SELF-LOCALIZATION FOR AUTONOMOUS VEHICLES

• US 20190129431A1
• Filed: 10/30/2017
• Published: 05/02/2019
• Est. Priority Date: 10/30/2017
• Status: Active Grant

• Current Assignee: NIO Technology Company Limited (NIO, Inc. (China))
• Original Assignee: Nio Usa Inc. (NIO, Inc. (China))
• Inventors: Yalla, Veera Ganesh, Kasturi Rangan, Sathya Narayanan, Bacchet, Davide
• Granted Patent: US 10,606,274 B2
• Time in Patent Office: Days
• CPC Class Codes: G01C 21/30 Map- or contour-matching
• CPC Class Codes: G01C 21/3815 Road data
• CPC Class Codes: G01C 21/3837 Data obtained from a single source
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01C 21/3896 Transmission of map data from central databases
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0255 using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: G05D 1/0257 using a radar radar systems designed for anti-collision purposes between land vehicles or between land vehicle and fixed obstacles G01S13/931
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

Methods and systems herein can let an autonomous vehicle localize itself precisely and in near real-time in a digital map using visual place recognition. Commercial GPS solutions used in the production of autonomous vehicles generally have very low accuracy. For autonomous driving, the vehicle may need to be able to localize in the map very precisely, for example, within a few centimeters. The method and systems herein incorporate visual place recognition into the digital map and localization process. The roadways or routes within the map can be characterized as a set of nodes, which can be augmented with feature vectors that represent the visual scenes captured using camera sensors. These feature vectors can be constantly updated on the map server and then provided to the vehicles driving the roadways. This process can help create and maintain a diverse set of features for visual place recognition.

Claims

1. An autonomous vehicle control system, comprising:

a processor in communication with a sensor, the sensor sensing an environment surrounding a vehicle associated with the vehicle control system, wherein a feature vector describes at least a portion of the environment at a current location of the vehicle on a route, the processor being programmed to;

receive information from the sensor regarding the feature vector;

retrieve feature vector information associated with one or more segments of a route of travel for the vehicle;

compare the received feature vector to the retrieved feature vector information;

determine if the received feature vector and the retrieved feature vector information match; and

if the received feature vector and the retrieved feature vector information match, determine that the current location of the vehicle is at a segment location of the one or more segments associated with the retrieved feature vector information.

2. The control system of claim 1, wherein the feature vector is a portion of a machine view of the environment obtained by the sensor.

3. The control system of claim 2, wherein the received feature vector is compared visually to the retrieved feature vector information.

4. The control system of claim 3, wherein the route is delineated into two or more segments.

5. The control system of claim 4, wherein each segment has a known segment location.

6. The control system of claim 5, further comprising the processor being programmed to determine a preliminary location to reduce a number of segments retrieved to compare to the received feature vector.

7. The control system of claim 6, wherein the preliminary location is determined by a GPS signal, by dead reckoning, by visual odometry, and/or by particle filtering.

8. The control system of claim 7, further comprising the processor being programmed to determine a next segment to compare to a next received feature vector based on a vector of travel for the vehicle.

9. The control system of claim 8, wherein the next received feature vector is obtained by the sensor after the vehicle has traveled to a next location.

10. The control system of claim 9, further comprising the processor being programmed to repeat one or more of:

receive next information from the sensor regarding the next feature vector;

retrieve next feature vector information associated with one or more next segments of the route of travel for the vehicle;

compare the next received feature vector to the next retrieved feature vector information;

determine if the next received feature vector and the next retrieved feature vector information match; and

if the next received feature vector and the next retrieved feature vector information match, determine that a next current location of the vehicle is at a next segment location of the next segment associated with the next retrieved feature vector information.

11. The control system of claim 1, further comprising the processor being programmed to, after determining the segment location, determine a new feature vector in view of the sensor.

12. The control system of claim 11, further comprising the processor being programmed to characterize the new feature vector.

13. The control system of claim 12, further comprising the processor being programmed to provide the new feature vector to another vehicle to aid the other vehicle in determining the location at the segment.

14. A method comprising:

determining, by a processor, a preliminary location of a vehicle on a route of travel, wherein the route of travel is delineated into two or more segments;

receiving, by the processor, information, from a sensor, wherein the sensor senses an environment surrounding the vehicle, wherein the information comprises a feature vector, wherein the feature vector describes at least a portion of the environment at a current location of the vehicle;

retrieving feature vector information associated with the one or more segments of the route of travel for the vehicle;

comparing the received feature vector and the retrieved feature vector information;

determining if the received feature vector and the retrieved feature vector information match; and

if the received feature vector and the retrieved feature vector information match, determining that the current location of the vehicle is at a segment location of the segment associated with the retrieved feature vector information.

15. The method of claim 14, wherein the feature vector is a portion of a machine view of the environment obtained by the sensor, and wherein the received feature vector is compared visually to the retrieved feature vector information.

16. The method of claim 15, wherein the route is delineated into two or more segments, and wherein each segment has a known segment location.

17. The method of claim 16, wherein determining the preliminary location of the vehicle reduces a number of segments retrieved to compare to the received feature vector, and wherein the preliminary location is determined by a GPS signal, by dead reckoning, by visual odometry, and/or by particle filtering.

18. The method of claim 17, further comprising:

determining a next segment to compare to a next received feature vector based on a vector of travel for the vehicle, wherein the next received feature vector is obtained by the sensor after the vehicle has traveled to a next location;

repeating one or more of;

receiving next information from the sensor regarding the next feature vector;

retrieving next feature vector information associated with one or more next segments of the route of travel for the vehicle;

comparing the next received feature vector to the next retrieved feature vector information;

determining if the next received feature vector to the next retrieved feature vector information match; and

if the next received feature vector to the next retrieved feature vector information match, determining that a next current location of the vehicle is at a next segment location of the next segment associated with the next retrieved feature vector information.

19. The method of claim 18, further comprising:

after determining the segment location, determining a new feature vector in view of the sensor;

characterizing the new feature vector; and

providing the new feature vector to another vehicle to aid the other vehicle in determining the location at the segment.

20. A vehicle, comprising:

a sensor to sense an environment surrounding the vehicle, wherein a feature vector describes at least a portion of the environment at a current location of the vehicle on a route;

a processor in communication with the sensor, the processor being programmed to;

receive information from the sensor regarding the feature vector;

retrieve feature vector information associated with one or more segments of a route of travel for the vehicle;

compare the received feature vector to the retrieved feature vector information;

determine if the received feature vector and the retrieved feature vector information match; and

if the received feature vector and the retrieved feature vector information match, determine that the current location of the vehicle is at a segment location of the segment associated with the retrieved feature vector information.