SMART NECKLACE WITH STEREO VISION AND ONBOARD PROCESSING

• US 20150198454A1
• Filed: 09/08/2014
• Published: 07/16/2015
• Est. Priority Date: 01/14/2014
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Moore, Douglas A., Djugash, Joseph M.A., Ota, Yasuhiro
• Granted Patent: US 10,024,679 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A61H 2003/063 with tactile perception
• CPC Class Codes: A61H 2201/1609 Neck
• CPC Class Codes: A61H 2201/165 Wearable interfaces
• CPC Class Codes: A61H 2201/5048 Audio interfaces, e.g. voice or music controlled
• CPC Class Codes: A61H 2201/5058 Sensors or detectors
• CPC Class Codes: A61H 2201/5084 Acceleration sensors
• CPC Class Codes: A61H 2201/5089 Gas sensors, e.g. for oxygen or CO2
• CPC Class Codes: A61H 2201/5092 Optical sensor
• CPC Class Codes: A61H 2201/5097 wireless
• CPC Class Codes: A61H 3/06 Walking aids for blind persons replacing direct visual perception by another kind of perception A61F9/08
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G01C 21/3617 using user history, behaviour, conditions or preferences, e.g. predicted or inferred from previous use or current movement
• CPC Class Codes: G01C 21/3623 using a camera or code reader, e.g. for optical or magnetic codes
• CPC Class Codes: G01C 21/3629 Guidance using speech or audio output, e.g. text-to-speech text to speech systems per se G10L13/00
• CPC Class Codes: G01C 21/3652 Guidance using non-audiovisual output, e.g. tactile, haptic or electric stimuli
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 17/00 : Speaker identification or verification techniques
• CPC Class Codes: G10L 2015/223 : Execution procedure of a spoken command
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 : using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 : using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor

Abstract

A wearable neck device includes an IMU coupled to the wearable neck device and adapted to detect inertial measurement data and a GPS coupled to the device and adapted to detect location data. The wearable neck device further includes a camera adapted to detect image data and a memory adapted to store data. The wearable neck device further includes a processor adapted to recognize an object in the surrounding environment by analyzing the data. The processor can determine a desirable action based on the data and a current time or day. The processor can determine a destination based on the determined desirable action. The processor can determine a navigation path based on the determined destination and the data. The processor is further adapted to determine output based on the navigation path. The wearable neck device further includes a speaker adapted to provide audio information to the user.

Claims

1. A wearable neck device for providing environmental awareness to a user, comprising:

a left portion;

a right portion;

an inertial measurement unit (IMU) sensor coupled to the wearable neck device and configured to detect inertial measurement data corresponding to a positioning, velocity, or acceleration of the wearable neck device;

a global positioning system (GPS) unit coupled to the wearable neck device and configured to detect location data corresponding to a location of the wearable neck device;

at least one camera coupled to the wearable neck device, the at least one camera configured to detect image data corresponding to a surrounding environment of the wearable neck device;

a memory configured to store object data regarding previously determined objects and previously determined user data associated with the user;

a processor connected to the IMU, the GPS unit and the at least one camera and configured to;

recognize an object in the surrounding environment by analyzing the image data based on the stored object data and at least one of the inertial measurement data or the location data,

determine a desirable event or action based on the recognized object, the previously determined user data, and a current time or day,

determine a destination based on the determined desirable event or action,

determine a navigation path for navigating the wearable neck device to the destination based on the determined destination, the image data, and at least one of the inertial measurement data or the location data, and

determine output data based on the determined navigation path; and

a speaker configured to provide audio information to the user based on at least one of the recognized object, determined desirable event or action, or navigation path.

2. The wearable neck device of claim 1 wherein the at least one camera includes a pair of stereo cameras configured to detect depth information.

3. The wearable neck device of claim 1 wherein the at least one camera includes a wide angle camera configured to detect image data within a 120 degree field of view.

4. The wearable neck device of claim 1 further comprising a band connecting the left portion and the right portion.

5. The wearable neck device of claim 4 wherein the memory and the processor are positioned on the band.

6. The wearable neck device of claim 1 wherein the at least one camera includes a pair of stereo cameras positioned on either the left portion or the right portion and a wide angle camera positioned on the other of the left portion or the right portion.

7. The wearable neck device of claim 6 wherein the processor is further configured to:

determine that one of the pair of stereo cameras or the wide angle camera is obstructed, and

alert the user to the obstructed stereo camera or the obstructed wide angle camera.

8. The wearable neck device of claim 7 wherein the processor is further configured to disregard data received from the obstructed stereo camera or the obstructed wide angle camera.

9. The wearable neck device of claim 1 further comprising a first vibratory motor coupled to the left portion and a second vibratory motor coupled to the right portion, the first vibratory motor and the second vibratory motor configured to provide stereo vibration data.

10. The wearable neck device of claim 1 further comprising a wireless communication antenna for establishing an audio or video communication with another portable electronic device or computer used by another person, wherein the processor is further configured to establish the audio or video communication based on the determined desirable event or action.

11. The wearable neck device of claim 1 further comprising a strap adapted to electrically and mechanically attach to the left portion and the right portion for providing electrical and mechanical robustness to the wearable neck device.

12. The wearable neck device of claim 1 further comprising a microphone configured to detect a speech of the user or another person, wherein the processor is further configured to:

parse a conversation of the user or the another person into speech elements;

analyze the speech elements based on the previously determined user data; and

determine the desirable event or action further based on the analyzed speech elements.

13. A method for providing continuous social and environmental awareness to a user by a wearable neck device comprising:

detecting, via a camera, a GPS unit or an IMU, inertial measurement data corresponding to a positioning, velocity, or acceleration of the wearable neck device, location data corresponding to a location of the wearable neck device or image data corresponding to a surrounding environment of the wearable neck device;

storing, in a memory, object data regarding previously determined objects and previously determined user data regarding the user;

recognizing, by a processor, an object in the surrounding environment by analyzing the image data based on the stored object data and at least one of the inertial measurement data or the location data;

determining, by the processor;

a desirable event or action based on the recognized object, the previously determined user data, and a current time or day,

a destination based on the determined desirable event or action,

a navigation path for navigating the wearable neck device to the destination based on the determined destination, the image data, and at least one of the inertial measurement data or the location data, or

output data based on the determined navigation path;

providing, via a speaker or a vibration unit, audio or haptic information to the user based on at least one of the recognized object, the determined desirable event or action, or the navigation path.

14. The method of claim 13 wherein:

detecting data includes detecting, by the camera, image data corresponding to the surrounding environment of the wearable neck device,

determining divergence data includes determining divergence data between the object data and the image data, and

providing audio or haptic information to the user further includes providing audio or haptic information based on the divergence data.

15. The method of claim 13 wherein providing audio or haptic information to the user includes providing stereo haptic information.

16. The method of claim 13 wherein storing object data includes storing object data in a remote database that can be accessed by other intelligent devices.

17. The method of claim 13 further comprising storing, in the memory, map data wherein determining the navigation path includes determining the navigation path based on the image data, the map data, and at least one of the location data or the inertial measurement data.

18. The method of claim 13 further including transmitting, via an antenna, the image data and at least one of the inertial measurement data or the location data to a remote device, wherein the processor is located on the remote device.

19. A wearable neck device for providing environmental awareness to a user, comprising:

an inertial measurement unit (IMU) sensor coupled to the wearable neck device and configured to detect inertial measurement data corresponding to a positioning, velocity, or acceleration of the wearable neck device;

a global positioning system (GPS) unit coupled to the wearable neck device and configured to detect location data corresponding to a location of the wearable neck device;

a wide angle camera coupled to the wearable neck device;

a pair of stereo cameras coupled to the wearable neck device, the pair of stereo cameras configured to detect depth information regarding a surrounding environment, the wide angle camera and the pair of stereo cameras being configured to detect image data corresponding to the surrounding environment of the wearable neck device;

a memory configured to store object data regarding previously determined objects and previously determined user data associated with the user;

a processor connected to the IMU, the GPS unit, the wide angle camera and the pair of stereo cameras and configured to;

recognize an object in the surrounding environment by analyzing the image data based on the stored object data and at least one of the inertial measurement data or the location data,

determine a desirable event or action based on the recognized object, the previously determined user data, and a current time or day,

determine a destination based on the determined desirable event or action,

determine a navigation path for navigating the wearable neck device to the destination based on the determined destination, the image data, and at least one of the inertial measurement data or the location data, and

determine output data based on the determined navigation path;

a speaker configured to provide audio information to the user based on at least one of the recognized object, determined desirable event or action, or navigation path; and

at least one vibratory motor configured to provide haptic information to the user based on at least one of the recognized object, determined desirable event or action, or navigation path.

20. The wearable neck device of claim 19 further comprising a wireless communication antenna for establishing data communication with another portable electronic device or computer.