Vehicle localization system

• US 10,642,269 B2
• Filed: 01/03/2018
• Issued: 05/05/2020
• Est. Priority Date: 01/03/2018
• Status: Active Grant

• Current Assignee: Denso International America Incorporated (DENSO Corporation)
• Original Assignee: Denso International America Incorporated (DENSO Corporation)
• Inventors: Lull, Joseph, Hunt, Shawn
• Primary Examiner(s): Dunn, Alex C
• Application Number:
  US15/860,973
• Publication Number:
  US 20190204831A1
• Time in Patent Office: 853 Days
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/0255 using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

A method for localizing and navigating a vehicle on underdeveloped or unmarked roads. The method includes: gathering image data with non-hyperspectral image sensors and audio data of a current scene; classifying the current scene based on the gathered image data and audio data to identify a stored scene model that most closely corresponds to the current scene; and setting spectral range of hyperspectral image sensors based on a spectral range used to capture a stored scene model that most closely corresponds to the current scene.

Claims

1. A method for localizing and navigating a vehicle on underdeveloped or unmarked roads, the method comprising:

gathering image data with non-hyperspectral image sensors and audio data of a current scene;

classifying, using a processer, the current scene based on the gathered image data and audio data to identify a stored scene model that most closely corresponds to the current scene;

setting spectral range of hyperspectral image sensors based on a spectral range used to capture a stored scene model that most closely corresponds to the current scene;

imaging the current scene with the hyperspectral image sensors set at the spectral range used to capture the stored scene model that most closely corresponds to the current scene; and

plotting a route for the vehicle based on images of the current scene captured by the hyperspectral image sensors set at the spectral range used to capture the stored scene model that most closely corresponds to the current scene.

2. The method of claim 1, wherein gathering audio data includes gathering audio data of interaction between wheels of the vehicle and terrain that the vehicle is traveling over.

3. The method of claim 2, further comprising calculating a spectrogram of the audio data.

4. The method of claim 1, further comprising extracting features from the image data gathered with the non-hyperspectral image sensors, including mean and standard deviation of RGB (red, green, blue), HSV (hue, saturation, value), and LAB colorspaces of the image data of the current scene.

5. The method of claim 1, further comprising building a feature set of the current scene including image features extracted from the image data, and calculating a spectrogram of the audio data.

6. The method of claim 5, wherein the classifying the current scene includes comparing the extracted image features and the spectrogram of the audio data with a plurality of different stored scene models, and identifying the stored scene model of the plurality of different stored scene models that most closely corresponds to the current scene.

7. The method of claim 1, further comprising classifying each pixel of the images of the current scene captured by the hyperspectral image sensors as “

go”

or “

no-go”

for the vehicle, and plotting the route for the vehicle based on the classification of each pixel.

8. A system for localizing and navigating a vehicle on underdeveloped or unmarked roads, the system comprising:

a non-hyperspectral image sensor for gathering image data of a current scene about the vehicle;

an audio sensor for gathering audio data of the current scene about the vehicle;

a control module in receipt of the image data and the audio data of the current scene, the control module configured to;

classify the current scene based on the gathered image data and audio data to identify a stored scene model that most closely corresponds to the current scene;

set spectral range of a hyperspectral image sensor based on a stored spectral range used to capture the stored scene model that most closely corresponds to the current scene;

image the current scene with the hyperspectral image sensor set at the stored spectral range used to capture the stored scene model that most closely corresponds to the current scene; and

plot a route for the vehicle based on images of the current scene captured by the hyperspectral image sensor set at the spectral range used to capture the stored scene model that most closely corresponds to the current scene.

9. The system of claim 8, wherein the hyperspectral image sensor is configured to capture at least one of the following spectral ranges:

visible range (400-705 nm);

near infrared (750-1,400 nm);

short wave infrared (1,400-3,000 nm);

medium wave infrared (3,000-8,000 nm);

long wave infrared 8,000-15,000 nm); and

far infrared (15,000 to 1*10^6 nm).

10. The system of claim 8, wherein the audio sensor includes at least one microphone.

11. The system of claim 10, wherein the at least one microphone includes a plurality of microphones mounted proximate to wheels of the vehicle to hear interaction between the wheels and underlying terrain.

12. The system of claim 8, wherein the control module is further configured to calculate a spectrogram based on the audio data.

13. The system of claim 8, wherein the control module is further configured to build a feature set of the current scene including image features extracted from the image data and a spectrogram of the audio data.

14. The system of claim 13, wherein the control module is further configured to classify the current scene by comparing the extracted image features and the spectrogram of the audio data with a plurality of different stored scene models, and identify the stored scene model of the plurality of different stored scene models that most closely correspond to the current scene.

15. The system of claim 8, further comprising classifying each pixel of the images captured by the hyperspectral image as “

go”

or “

no-go”

for the vehicle to travel, the navigation module plots the route based on the classification.