Vehicular threat detection based on audio signals

• US 9,107,012 B2
• Filed: 01/31/2012
• Issued: 08/11/2015
• Est. Priority Date: 12/01/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Lord, Richard T., Lord, Robert W., Myhrvold, Nathan P., Tegreene, Clarence T., Hyde, Roderick A., Wood, Lowell L. Jr., Ishikawa, Muriel Y., Wood, Victoria Y. H., Whitmer, Charles, Bahl, Paramvir, Burger, Douglas C., Chandra, Ranveer, Gates, William H. III, Holman, Paul, Kare, Jordin T., Mundie, Craig J., Paek, Tim, Tan, Desney S., Zhong, Lin, Dyor, Matthew G.
• Primary Examiner(s): Chin, Vivian
• Assistant Examiner(s): Odunukwe, Ubachukwu
• Application Number:
  US13/362,823
• Publication Number:
  US 20130142347A1
• Time in Patent Office: 1,288 Days
• Field of Search: 381/58, 381/86, 381/56, 701/29, 701/431, 340/425.5, 340/459, 340/441, 340/517, 340/933, 340/331, 340/901, 367/99, 367/127, 382/104
• US Class Current: 1/1
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 2460/07 Use of position data from wide-area or local-area positioning systems in hearing devices, e.g. program or information selection
• CPC Class Codes: H04R 2499/13 Acoustic transducers and sound field adaptation in vehicles
• CPC Class Codes: H04R 29/005 Microphone arrays

Abstract

Techniques for ability enhancement are described. Some embodiments provide an ability enhancement facilitator system (“AEFS”) configured to enhance a user'"'"'s ability to operate or function in a transportation-related context as a pedestrian or a vehicle operator. In one embodiment, the AEFS is configured perform vehicular threat detection based at least in part on analyzing audio signals. An example AEFS receives data that represents an audio signal emitted by a vehicle. The AEFS analyzes the audio signal to determine vehicular threat information, such as that the vehicle may collide with the user. The AEFS then informs the user of the determined vehicular threat information, such as by transmitting a warning to a wearable device configured to present the warning to the user.

Claims

1. A method for enhancing ability in a transportation-related context, the method comprising:

receiving data representing an audio signal obtained in proximity to a user, the audio signal emitted by a first vehicle;

determining vehicular threat information based at least in part on the data representing the audio signal, wherein the determining vehicular threat information includes;

performing acoustic source localization to determine a position of the first vehicle based on the audio signal emitted by the first vehicle measured via multiple microphones. wherein the performing acoustic source localization includes triangulating the position of the first vehicle based on a first and second angle, wherein the first angle is measured between a first one of the multiple microphones and the first vehicle, wherein the first angle is based on the audio signal emitted by the first vehicle as measured by the first microphone, wherein the second angle is measured between a second one of the multiple microphones and the first vehicle. wherein the second angle is based on the audio signal emitted by the first vehicle as measured by the second microphone;

determining vehicular threat information related to factors other than ones related to the first vehicle, by;

determining that poor surface conditions exist on a road traveled by the user by considering weather conditions, temperature, road surface type, and foreign materials on the road; and

presenting the vehicular threat information via a visual display device and/or an audio output device of a wearable device of the user by presenting a visual and/or audio message directing the user to accelerate, decelerate, or turn.

2. The method of claim 1, wherein the receiving data representing an audio signal includes:

receiving data obtained at a microphone array that includes the multiple microphones.

3. The method of claim 2, wherein the receiving data obtained at a microphone array includes:

receiving data obtained at a microphone array, the microphone array coupled to a vehicle of the user.

4. The method of claim 2, wherein the receiving data obtained at a microphone array includes:

receiving data obtained at a microphone array, the microphone array coupled to the wearable device.

5. The method of claim 1, wherein the determining vehicular threat information includes:

determining a position of the first vehicle.

6. The method of claim 1, wherein the determining vehicular threat information includes:

determining a velocity of the first vehicle.

7. The method of claim 1, wherein the determining vehicular threat information includes:

determining a direction of travel of the first vehicle.

8. The method of claim 1, wherein the determining vehicular threat information includes:

determining whether the first vehicle is approaching the user.

9. The method of claim 1, wherein the performing acoustic source localization includes:

receiving an audio signal via a first one of the multiple microphones, the audio signal representing a sound created by the first vehicle;

receiving the audio signal via a second one of the multiple microphones; and

determining the position of the first vehicle by determining a difference between an arrival time of the audio signal at the first microphone and an arrival time of the audio signal at the second microphone.

10. The method of claim 1, wherein the determining vehicular threat information includes:

performing a Doppler analysis of the data representing the audio signal to determine whether the first vehicle is approaching the user.

11. The method of claim 10, wherein the performing a Doppler analysis includes:

determining whether frequency of the audio signal is increasing or decreasing.

12. The method of claim 1, wherein the determining vehicular threat information includes:

performing a volume analysis of the data representing the audio signal to determine whether the first vehicle is approaching the user.

13. The method of claim 12, wherein the performing a volume analysis includes:

determining whether volume of the audio signal is increasing or decreasing.

14. The method of claim 1, wherein the determining vehicular threat information includes:

determining the vehicular threat information based on gaze information associated with the user.

15. The method of claim 14, further comprising:

receiving an indication of a direction in which the user is looking;

determining that the user is not looking towards the first vehicle; and

in response to determining that the user is not looking towards the first vehicle, directing the user to look towards the first vehicle.

16. The method of claim 1, further comprising:

identifying multiple threats to the user;

identifying a first one of the multiple threats that is more significant than at least one other of the multiple threats; and

causing the user to avoid the first one of the multiple threats.

17. The method of claim 1, wherein the determining vehicular threat information related to factors other than ones related to the first vehicle includes determining that there is a pedestrian in proximity to the user based on a heat signature of the pedestrian detected by an infrared sensor of the wearable device.

18. The method of claim 1, wherein the determining vehicular threat information related to factors other than ones related to the first vehicle includes determining that there is an animal that is not a pedestrian and that is in proximity to the user.

19. The method of claim 1, wherein the determining vehicular threat information related to factors other than ones related to the first vehicle includes:

determining that there is an accident in proximity to the user based on information received from a vehicle-based system that transmits when a collision occurs.

20. The method of claim 1, wherein the determining vehicular threat information includes:

determining the vehicular threat information based on kinematic information.

21. The method of claim 20, wherein the determining the vehicular threat information based on kinematic information includes:

determining the vehicular threat information based on information about position, velocity, and/or acceleration of the user obtained from sensors in the wearable device.

22. The method of claim 20, wherein the determining the vehicular threat information based on kinematic information includes:

determining the vehicular threat information based on information about position, velocity, and/or acceleration of the user obtained from devices in a vehicle of the user.

23. The method of claim 20, wherein the determining the vehicular threat information based on kinematic information includes:

determining the vehicular threat information based on information about position, velocity, and/or acceleration of the first vehicle.

24. The method of claim 1, wherein the presenting the vehicular threat information includes:

presenting the vehicular threat information via an audio output device of the wearable device.

25. The method of claim 1, wherein the presenting the vehicular threat information via a visual display device includes:

displaying an indicator that instructs the user to look towards the first vehicle.

26. The method of claim 1, wherein the presenting the vehicular threat information includes at least one of:

directing the user to accelerate, directing the user to decelerate, and/or directing the user to turn.

27. The method of claim 1, further comprising:

when the user and the first vehicle are approaching head on and not turning away from one another, transmitting to the first vehicle a warning based on the vehicular threat information, wherein the warning is complementary to the message presented to the user, thereby reducing risk of a collision between the first vehicle and the user when the warning and the presented message are both followed.

28. The method of claim 1, further comprising:

presenting the vehicular threat information via an output device of a vehicle of the user, the output device including a visual display and/or an audio speaker.

29. The method of claim 1, wherein the wearable device is one of a helmet, goggles, eyeglasses, or a hat worn by the user.

30. The method of claim 1, wherein the presenting the vehicular threat information includes:

presenting the vehicular threat information via audio speakers that are part of at least one of earphones, a headset, earbuds, and/or a hearing aid.

31. The method of claim 1, further comprising:

performing the receiving data representing an audio signal, the determining vehicular threat information, and/or the presenting the vehicular threat information on a computing device in the wearable device of the user.

32. The method of claim 1, further comprising:

performing the receiving data representing an audio signal, the determining vehicular threat information, and/or the presenting the vehicular threat information on a road-side computing system; and

transmitting the vehicular threat information from the road-side computing system to the wearable device of the user.

33. The method of claim 1, further comprising:

performing the receiving data representing an audio signal, the determining vehicular threat information, and/or the presenting the vehicular threat information on a computing system in the first vehicle; and

transmitting the vehicular threat information from the computing system to the wearable device of the user.

34. The method of claim 1, further comprising:

performing the receiving data representing an audio signal, the determining vehicular threat information, and/or the presenting the vehicular threat information on a computing system in a second vehicle, wherein the user is not traveling in the second vehicle; and

transmitting the vehicular threat information from the computing system to the wearable device of the user.

35. The method of claim 1, further comprising:

receiving data representing a visual signal that represents the first vehicle, the receiving including receiving an image of the first vehicle obtained by a camera of the wearable device; and

determining the vehicular threat information based further on the data representing the visual signal, the determining including identifying the first vehicle in an image represented by the data representing a visual signal and determining whether the first vehicle is moving towards the user based on multiple images represented by the data representing the visual signal.

36. The method of claim 1, further comprising:

receiving data representing the first vehicle obtained at a road-based device; and

determining the vehicular threat information based further on the data representing the first vehicle.

37. The method of claim 36, wherein the receiving data representing the first vehicle obtained at a road-based device includes at least one of:

receiving the data from a sensor deployed at an intersection;

receiving an image of the rust vehicle from a camera deployed at an intersection;

receiving ranging data from a range sensor deployed at an intersection, the ranging data representing a distance between the first vehicle and the intersection; and

/or receiving data from an induction loop deployed in a road surface, the induction loop configured to detect the presence and/or velocity of the first vehicle.

38. The method of claim 36, wherein the determining the vehicular threat information based further on the data representing the first vehicle includes:

identifying the first vehicle in an image obtained from the road-based sensor.

39. The method of claim 36, wherein the determining the vehicular threat information based further on the data representing the first vehicle includes:

determining a trajectory of the first vehicle based on multiple images obtained from the road-based device.

40. The method of claim 1, further comprising:

receiving data representing vehicular threat information relevant to a second vehicle, the second vehicle not being used for travel by the user; and

determining the vehicular threat information based on the data representing vehicular threat information relevant to the second vehicle.

41. The method of claim 40, wherein the receiving data representing vehicular threat information relevant to a second vehicle includes:

receiving from the second vehicle at least one of;

an indication of stalled or slow traffic encountered by the second vehicle, an indication of poor driving conditions experienced by the second vehicle, an indication that the first vehicle is driving erratically, and/or an image of the first vehicle.

42. The method of claim 1, further comprising:

transmitting the vehicular threat information to a second vehicle by transmitting the vehicular threat information to an intermediary server system for distribution to other vehicles in proximity to the user.

43. The method of claim 1, wherein the determining vehicular threat information related to factors other than ones related to the first vehicle includes determining that there is a pedestrian in proximity to the user based on a location signal transmitted by a device worn by the pedestrian.

44. The method of claim 1, further comprising:

receiving data representing vehicular threat information relevant to a second vehicle by receiving an image of the first vehicle and an indication that the first vehicle is driving erratically, wherein neither the first nor the second vehicle are being used for travel by the user; and

determining the vehicular threat information based on the data representing vehicular threat information relevant to the second vehicle.

45. A non-transitory computer-readable medium including instructions that are configured, when executed by a processor of a computing system, to cause the computing system to perform a method for ability enhancement in a transportation-related context, the method comprising:

receiving data representing an audio signal obtained in proximity to a user, the audio signal emitted by a first vehicle;

determining vehicular threat information based at least in part on the data representing the audio signal, wherein the determining vehicular threat information includes;

performing acoustic source localization to determine a position of the first vehicle based on the audio signal emitted by the first vehicle measured via multiple microphones, wherein the performing acoustic source localization includes triangulating the position of the first vehicle based on a first and second angle, wherein the first angle is measured between a first one of the multiple microphones and the first vehicle, wherein the first angle is based on the audio signal emitted by the first vehicle as measured by the first microphone, wherein the second angle is measured between a second one of the multiple microphones and the first vehicle, wherein the second angle is based on the audio signal emitted by the first vehicle as measured by the second microphone;

determining vehicular threat information related to factors other than ones related to the first vehicle, by;

determining that poor surface conditions exist on a road traveled by the user by considering weather conditions, temperature, road surface type, and foreign materials on the road; and

presenting the vehicular threat information via a visual display device and/or an audio output device of a wearable device of the user by presenting a visual and/or audio message directing the user to accelerate, decelerate, or turn.

46. A computing system for ability enhancement in a transportation-related context, the computing system comprising:

a processor;

a memory;

multiple microphones; and

logic instructions that are stored in the memory and that are configured, when executed by the processor, to perform a method comprising;

receiving data representing an audio signal obtained in proximity to a user, the audio signal emitted by a first vehicle;

determining vehicular threat information based at least in part on the data representing the audio signal, wherein the determining vehicular threat information includes;

performing acoustic source localization to determine a position of the first vehicle based on multiple audio signals received via the multiple microphones, wherein the performing acoustic source localization includes triangulating the position of the first vehicle based on a first and second angle, wherein the first angle is measured between a first one of the multiple microphones and the first vehicle wherein the first angle is based on the audio signal emitted by the first vehicle as measured by the first microphone, wherein the second angle is measured between a second one of the multiple microphones and the first vehicle, wherein the second angle is based on the audio signal emitted by the first vehicle as measured by the second microphone;

determining vehicular threat information related to factors other than ones related to the first vehicle, by;

determining that poor surface conditions exist on a road traveled by the user by considering weather conditions, temperature, road surface type, and foreign materials on the road; and

presenting the vehicular threat information via a visual display device and/or an audio output device of a wearable device of the user by presenting a visual and/or audio message directing the user to accelerate, decelerate, or turn.