Wearable smart device for hazard detection and warning based on image and audio data

• US 9,576,460 B2
• Filed: 01/21/2015
• Issued: 02/21/2017
• Est. Priority Date: 01/21/2015
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Dayal, Rajiv
• Primary Examiner(s): HSU, AMY R
• Application Number:
  US14/601,506
• Publication Number:
  US 20160210834A1
• Time in Patent Office: 762 Days
• US Class Current: 1/1
• CPC Class Codes: A61H 2003/063 with tactile perception
• CPC Class Codes: A61H 2201/0184 by raising an alarm
• CPC Class Codes: A61H 2201/1609 Neck
• CPC Class Codes: A61H 2201/165 Wearable interfaces
• CPC Class Codes: A61H 2201/5012 using the internet
• CPC Class Codes: A61H 2201/5046 Touch screens
• CPC Class Codes: A61H 2201/5048 Audio interfaces, e.g. voice or music controlled
• CPC Class Codes: A61H 2201/5064 Position sensors
• CPC Class Codes: A61H 2201/5082 Temperature sensors
• CPC Class Codes: A61H 2201/5089 Gas sensors, e.g. for oxygen or CO2
• CPC Class Codes: A61H 2201/5092 Optical sensor
• CPC Class Codes: A61H 2201/5097 wireless
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: G01C 21/12 executed aboard the object being navigated; Dead reckoning
• CPC Class Codes: G01C 21/3691 Retrieval, searching and output of information related to real-time traffic, weather, or environmental conditions arrangements for giving variable traffic instructions G08G1/09
• CPC Class Codes: G08B 21/02 Alarms for ensuring the safety of persons

Abstract

A wearable smart device for providing hazard warning information to a user. The wearable smart device includes a microphone configured to detect audio data associated with a potential hazard. The wearable smart device also includes a camera configured to detect image data associated with the potential hazard. The wearable smart device also includes a processor coupled to the microphone and the camera and configured to determine whether the potential hazard presents a real hazard based on the detected audio data and the detected image data.

Claims

1. A wearable computing device for providing hazard warning information comprising:

a microphone configured to detect audio data associated with a potential hazard;

a camera configured to detect image data associated with the potential hazard; and

a processor coupled to the microphone and the camera and configured to determine which of the detected audio data or the detected image data has a higher quality, and to determine whether the potential hazard presents a real hazard to a user based on the detected audio data and the detected image data such that more weight is provided to whichever of the detected audio data or the detected image data has the higher quality.

2. The wearable computing device of claim 1 further comprising a memory coupled to the processor and configured to store memory audio data and memory image data associated with at least one hazard, and wherein the processor is configured to determine whether the potential hazard presents the real hazard by comparing at least one of the detected audio data or the detected image data to at least one of the memory audio data or the memory image data.

3. The wearable computing device of claim 1 further comprising an inertial measurement unit (IMU) coupled to the processor and configured to detect motion data and a GPS sensor configured to detect location data and wherein the processor is further configured to determine a current location of the wearable computing device based on at least two of the detected motion data, the detected location data, the detected audio data, and the detected image data.

4. The wearable computing device of claim 3 further comprising a memory coupled to the processor and configured to store a stored location and at least one hazard associated with the stored location, and wherein the processor is further configured to determine whether the at least one hazard presents a danger to the user based on a comparison of the current location to the stored location.

5. The wearable computing device of claim 1 wherein the microphone includes two microphones spaced apart and the processor is further configured to determine a direction of the potential hazard based on the detected audio data.

6. The wearable computing device of claim 5 wherein the processor is further configured to instruct the camera to focus towards the direction of the potential hazard in response to determining the direction of the potential hazard.

7. The wearable computing device of claim 1 wherein the processor is further configured to determine a distance to the potential hazard based on a volume of the detected audio data.

8. The wearable computing device of claim 1 wherein the processor is further configured to determine a severity level of the potential hazard based on at least one factor.

9. The wearable computing device of claim 8 wherein the at least one factor includes a distance to the potential hazard, whether a distance between the potential hazard and the user is decreasing, a size of the potential hazard, or an audio volume of the potential hazard.

10. The wearable computing device of claim 1 wherein the higher quality corresponds to at least one of a lack of obstruction of the detected audio data or the detected image data, an accuracy of the detected audio data or the detected image data, or a match of the detected audio data or the detected image data to data stored in a memory.

11. A wearable computing device for providing hazard warnings comprising:

an upper portion having a first end and a second end;

a first lower portion coupled to the first end of the upper portion;

a second lower portion coupled to the second end of the upper portion;

at least one microphone positioned on the first lower portion, the second lower portion or the upper portion and configured to detect audio data associated with a potential hazard;

a camera positioned on the first lower portion or the second lower portion and configured to detect image data associated with the potential hazard; and

a processor coupled to the camera and the at least one microphone and configured to determine which of the detected audio data or the detected image data has a higher quality, and to determine whether the potential hazard presents a real hazard to a user by analyzing whichever of the detected audio data or the detected image data has the higher quality.

12. The wearable computing device of claim 11 further comprising a memory coupled to the processor and configured to store memory audio data and memory image data associated with at least one hazard and wherein the processor is configured to determine whether the potential hazard presents the real hazard by comparing at least one of the detected audio data or the detected image data to at least one of the memory audio data or the memory image data.

13. The wearable computing device of claim 11 wherein the at least one microphone includes two microphones and the processor is further configured to determine a direction of the potential hazard based on the detected audio data.

14. The wearable computing device of claim 13 wherein the processor is further configured to cause the camera to focus towards the direction of the potential hazard in response to determining the direction of the potential hazard.

15. The wearable computing device of claim 11 wherein the processor is further configured to determine a distance to the potential hazard based on a volume of the detected audio data.

16. The wearable computing device of claim 11 wherein the processor is further configured to determine a severity level of the potential hazard based on at least one of a distance to the potential hazard, whether the potential hazard is approaching the user, a size of the potential hazard or an audio volume of the potential hazard.

17. The wearable computing device of claim 11 wherein the higher quality corresponds to at least one of a lack of obstruction of the detected audio data or the detected image data, an accuracy of the detected audio data or the detected image data, whether the detected audio data or the detected image data fails to indicate that the potential hazard is present, or a match of the detected audio data or the detected image data to data stored in a memory.

18. A method for providing hazard warnings to a user of a wearable computing device comprising:

detecting, by at least two microphones, audio data associated with a potential hazard and including volume information;

detecting, by a camera, image data associated with the potential hazard;

determining, by a processor, whether the detected audio data or the detected image data has a higher quality; and

determining, by the processor, whether the potential hazard presents a real hazard to the user based on the detected audio data and the detected image data by providing more weight to whichever of the detected audio data or the detected image data has the higher quality.

19. The method of claim 18 wherein the higher quality corresponds to at least one of a lack of obstruction of the detected audio data or the detected image data, an accuracy of the detected audio data or the detected image data, whether the detected audio data or the detected image data fails to indicate that the potential hazard is present, or a match of the detected audio data or the detected image data to data stored in a memory.

20. The method of claim 18 further comprising storing, in a memory, stored audio data and stored image data associated with at least one hazard, and wherein determining whether the potential hazard presents the real hazard is based on a comparison of at least one of the detected audio data or the detected image data to at least one of the stored audio data or the stored image data.

21. A method for providing hazard warnings to a user of a wearable smart device comprising:

storing, in a memory, hazard data;

detecting, by a microphone, audio data associated with a potential hazard;

detecting, by a camera, image data associated with the potential hazard;

determining, by a processor,

which of the detected audio data or the detected image data has a higher quality; and

determining, by the processor, whether the potential hazard presents a real hazard to the user based on which of the detected audio data or the detected image data has the higher quality.

22. The method of claim 21 further comprising generating, by a pair of vibration units, haptic feedback indicating a presence of the potential hazard in response to determining that the potential hazard presents the real hazard to the user.