Systems and methods for identification of objects using audio and sensor data

• US 10,445,597 B2
• Filed: 10/17/2017
• Issued: 10/15/2019
• Est. Priority Date: 10/17/2017
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Research Institute, Inc. (Toyota Motor Corporation)
• Inventors: Jackson, Astrid Glende
• Primary Examiner(s): Bitar, Nancy
• Application Number:
  US15/786,106
• Publication Number:
  US 20190114489A1
• Time in Patent Office: 728 Days
• Field of Search: None
• CPC Class Codes: G06F 18/241 relating to the classification model, e.g. parametric or non-parametric approaches
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: G10L 15/065 Adaptation
• CPC Class Codes: G10L 15/083 Recognition networks G10L15/142, G10L15/16 take precedence
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: H04R 2499/13 Acoustic transducers and sound field adaptation in vehicles
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

System, methods, and other embodiments described herein relate to identifying objects using audio data in combination with sensor data. In one embodiment, a method includes, in response to acquiring audio data from at least one microphone integrated with the vehicle and feature data from at least one sensor of the vehicle, analyzing, using a deep learning algorithm, the feature data to generate a classification of an object embodied by the feature data. The method includes selectively refining the classification as a function of at least the audio data to further identify the object using the classification. The classification identifies at least characteristics about a shape and size of the object. The method includes controlling one or more vehicle systems of the vehicle according to the classification.

Claims

1. A fusion system for identifying aspects of a surrounding environment of a vehicle comprising:

one or more processors;

a memory communicably coupled to the one or more processors and storing;

a monitoring module including instructions that when executed by the one or more processors cause the one or more processors to, in response to acquiring audio data from at least one microphone integrated with the vehicle and feature data from at least one sensor of the vehicle, analyze, using a deep learning algorithm, the feature data to generate a classification of an object embodied by the feature data, the classification including a confidence interval; and

an identification module including instructions that when executed by the one or more processors cause the one or more processors to selectively refine the classification using the deep learning algorithm as a function of at least the audio data by regenerating the classification using the feature data and the audio data when at least one of;

the confidence interval satisfies a non-detection threshold, and the classification indicates the object is partially occluded, wherein the classification identifies at least characteristics about a shape and size of the object; and

wherein the identification module further includes instructions to control one or more vehicle systems of the vehicle according to the classification.

2. The fusion system of claim 1, wherein the identification module includes the instructions to selectively refine the classification including instructions to generate the classification as a function of the audio data, wherein the identification module includes the instructions to selectively refine the classification using the deep learning algorithm including instructions to use at least two neural networks that form the deep learning algorithm, and wherein the confidence interval defines a strength of the classification.

3. The fusion system of claim 1, wherein the feature data includes at least one or more visible light images, and wherein the at least one sensor is a camera, wherein the deep learning algorithm includes at least two neural networks that form the deep learning algorithm, the at least two neural networks including a recurrent neural network (RNN) to process the audio data, and a convolutional neural network (CNN) to process the feature data and an output of the RNN.

4. The fusion system of claim 1, wherein the monitoring module includes the instructions to analyze the feature data including instructions to use the feature data as an electronic input to the deep learning algorithm that is a convolutional neural network (CNN) to analyze the feature data.

5. The fusion system of claim 4, wherein the identification module includes the instructions to selectively refine the classification including instructions to provide the classification and the audio data to a recurrent neural network (RNN) and obtaining an electronic output as the classification that accounts for the audio data in generating the classification, and

wherein the identification module includes the instructions to selectively refine the classification including instructions to regenerate the classification using the audio data when the classification initially indicates that the object is partially occluded.

6. The fusion system of claim 1, wherein the monitoring module includes instructions to, prior to analyzing the feature data to generate the classification, analyze the audio data to identify a present context associated with the surrounding environment, wherein the present context indicates aspects relating to objects that are likely present in the surrounding environment,

wherein the monitoring module includes the instructions to analyze the feature data including instructions to analyze the present context in combination with the feature data to generate the classification,

wherein the monitoring module includes the instructions to analyze the audio data including instructions to localize the object in the surrounding environment in relation to the vehicle, and wherein the audio data includes information from at least two microphones.

7. The fusion system of claim 1, wherein the at least one microphone is integrated with the vehicle to acquire the audio data of the surrounding environment outside of the vehicle, wherein the at least one sensor includes a light detection and ranging (LIDAR) sensor that is integrated with the vehicle, and wherein the feature data includes point cloud data.

8. The fusion system of claim 1, wherein the identification module refines the classification by using the audio data to supplement how the deep learning algorithm determines the classification.

9. A non-transitory computer-readable medium storing for identifying aspects of a surrounding environment of a vehicle and including instructions that when executed by one or more processors cause the one or more processors to:

in response to acquiring audio data from at least one microphone integrated with the vehicle and feature data from at least one sensor of the vehicle, analyze, using a deep learning algorithm, the feature data to generate a classification of an object embodied by the feature data, the classification including a confidence interval; and

selectively refine the classification using the deep learning algorithm as a function of at least the audio data by regenerating the classification using the feature data and the audio data when at least one of;

the confidence interval satisfies a non-detection threshold, and the classification indicates the object is partially occluded, wherein the classification identifies at least characteristics about a shape and size of the object; and

control one or more vehicle systems of the vehicle according to the classification.

10. The non-transitory computer-readable medium of claim 9, wherein the instructions to selectively refine the classification include instructions to generate the classification as a function of the audio data, wherein the deep learning algorithm includes at least two neural networks that form the deep learning algorithm, and wherein the confidence interval defines a strength of the classification.

11. The non-transitory computer-readable medium of claim 9, wherein the instructions to analyze the feature data include instructions to use the feature data as an electronic input to the deep learning algorithm that is a convolutional neural network (CNN) to analyze the feature data,

wherein the instructions to selectively refine the classification include instructions to provide the classification and the audio data to a recurrent neural network (RNN) and obtain an electronic output as the classification that accounts for the audio data in generating the classification, and

wherein the instructions to selectively refine the classification including instructions to regenerate the classification using the audio data when the classification initially indicates that the object is partially occluded.

12. The non-transitory computer-readable medium of claim 9, further comprising instructions to:

prior to analyzing the feature data to generate the classification, analyze the audio data to identify a present context associated with the surrounding environment, wherein the present context indicates aspects relating to objects that are likely present in the surrounding environment,

wherein the instructions to analyze the feature data include instructions to analyze the present context in combination with the feature data to generate the classification,

wherein the instructions to analyze the audio data include instructions to localize the object in the surrounding environment in relation to the vehicle, and wherein the audio data includes information from at least two microphones.

13. The non-transitory computer-readable medium of claim 9, wherein the at least one microphone is integrated with the vehicle to acquire the audio data of the surrounding environment outside of the vehicle, wherein the at least one sensor includes a light detection and ranging (LIDAR) sensor that is integrated with the vehicle, and wherein the feature data includes point cloud data.

14. A method of identifying aspects of a surrounding environment of a vehicle, comprising:

in response to acquiring audio data from at least one microphone integrated with the vehicle and feature data from at least one sensor of the vehicle, analyzing, using a deep learning algorithm, the feature data to generate a classification of an object embodied by the feature data, the classification including a confidence interval;

selectively refining the classification using the deep learning algorithm as a function of at least the audio data by regenerating the classification using the feature data and the audio data when at least one of;

the confidence interval satisfies a non-detection threshold, and the classification indicates the object is partially occluded, wherein the classification identifies at least characteristics about a shape and size of the object; and

controlling one or more vehicle systems of the vehicle according to the classification.

15. The method of claim 14, wherein selectively refining the classification includes generating the classification as a function of the audio data, wherein selectively refining the classification using the deep learning algorithm includes using at least two neural networks that form the deep learning algorithm, and wherein the confidence interval defines a strength of the classification.

16. The method of claim 14, wherein the feature data includes at least one or more visible light images, and wherein the at least one sensor is a camera.

17. The method of claim 14, wherein analyzing the feature data includes using the feature data as an electronic input to the deep learning algorithm that is a convolutional neural network (CNN).

18. The method of claim 17, wherein selectively refining the classification includes providing the classification and the audio data to a recurrent neural network (RNN) and obtaining an electronic output as the classification that accounts for the audio data in generating the classification, and

wherein selectively refining the classification includes regenerating the classification using the audio data when the classification initially indicates that the object is partially occluded.

19. The method of claim 14, further comprising:

prior to analyzing the feature data to generate the classification, analyzing the audio data to identify a present context associated with the surrounding environment, wherein the present context indicates aspects relating to objects that are likely present in the surrounding environment,

wherein analyzing the feature data includes analyzing the present context in combination with the feature data to generate the classification, and

wherein analyzing the audio data includes localizing the object in the surrounding environment in relation to the vehicle, wherein the audio data includes information from at least two microphones.

20. The method of claim 14, wherein the at least one microphone is integrated with the vehicle to acquire the audio data of the surrounding environment outside of the vehicle, wherein the at least one sensor includes a light detection and ranging (LIDAR) sensor that is integrated with the vehicle, and wherein the feature data includes point cloud data.