Vision-assist systems adapted for inter-device communication session

• US 9,996,730 B2
• Filed: 03/18/2016
• Issued: 06/12/2018
• Est. Priority Date: 03/18/2016
• Status: Active Grant

• Current Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Lee, Christopher P.
• Primary Examiner(s): Nakhjavan, Shervin
• Application Number:
  US15/074,554
• Publication Number:
  US 20170270347A1
• Time in Patent Office: 816 Days
• Field of Search: None
• CPC Class Codes: A61H 2201/165 Wearable interfaces
• CPC Class Codes: A61H 2201/5007 computer controlled
• CPC Class Codes: A61H 2201/5043 Displays
• CPC Class Codes: A61H 2201/5048 Audio interfaces, e.g. voice or music controlled
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04W 12/06 Authentication
• CPC Class Codes: H04W 12/33 using wearable devices, e.g. using a smartwatch or smart-glasses

Abstract

Vision-assist systems including user eye tracking cameras are disclosed. A vision-assist system includes a processor, a memory module communicatively coupled to the processor, a user eye tracking camera communicatively coupled to the processor, an environment camera communicatively coupled to the processor, a feedback device communicatively coupled to the processor, and machine readable instructions stored in the memory module that, when executed by the processor, cause the vision-assist system to receive environment image data from the environment camera, determine a location of an individual speaking to a user based on the environment image data, receive user eye tracking image data from the user eye tracking camera, determine a pose of the user'"'"'s eyes based on the user eye tracking image data, and provide feedback to the user with the feedback device based on the location of the individual speaking to the user and the pose of the user'"'"'s eyes.

Claims

1. A vision-assist system comprising:

a processor;

a memory module communicatively coupled to the processor;

communication hardware configured to communicatively couple the processor to at least one remote vision-assist system;

at least one sensor adapted to detect first environment information from a first region of an environment;

a user feedback device communicatively coupled to the processor; and

machine readable instructions stored in the memory module that cause the vision-assist system to perform at least the following when executed by the processor;

receive identification information of a remote vision-assist system;

authenticate the identification information;

initiate a communication session with the remote vision-assist system;

receive second environment information from the remote vision-assist system from a second region of the environment; and

provide feedback to the user with the feedback device derived at least in part based on the first and second environment information received from the remote vision-assist system.

2. The vision-assist system of claim 1, wherein when executed by the processor, the machine readable instructions cause the vision-assist system to authenticate the identification information of the remote vision-assist system by comparing the identification information to information stored on the memory module.

3. The vision-assist system of claim 2, wherein when executed by the processor, the machine readable instructions cause the vision-assist system to automatically generate a communication session request in response to an authentication of the identification information and transmit the communication session request.

4. The vision-assist system of claim 1, wherein when executed by the processor, the machine readable instructions cause the vision-assist system to authenticate the identification information of the remote vision-assist system by providing a prompt via the feedback device and receiving a confirmation via an input device of the vision-assist system.

5. The vision-assist system of claim 1, wherein the identification information comprises at least one of the following group:

a unique device identifier, a unique user identifier, a device-type identifier and a user-type identifier.

6. The vision-assist system of claim 1, wherein the first and second environment information comprises at least one of the following group:

audio information, image information, video information, location information and inertial information.

7. The vision-assist system of claim 1, wherein the communication system includes at least one of the following group:

text message communication, Braille communication, audio communication, voice communication, video communication and image communication.

8. The vision-assist system of claim 1, wherein the at least one sensor of the vision-assist system comprises at least one camera and the received environment information includes image environment information and when executed by the processor, the machine readable instructions cause the vision-assist system to:

analyze the received image environment information and image information obtained via the camera and stored the memory module and determine environment information derived from a combination of the received image environment information and the image information stored on the memory module.

9. The vision-assist system of claim 1, wherein the at least one sensor of the vision-assist system comprises at least one camera and the received second environment information includes video environment information and when executed by the processor, the machine readable instructions cause the vision-assist system to:

analyze the received video environment information and video information obtained via the camera and stored on the memory module and determine environment information derived from a combination of the received video environment information and the video information stored on the memory module.

10. The vision-assist system of claim 1, wherein the received second environment information includes location environment information and when executed by the processor, the machine readable instructions cause the vision-assist system to:

analyze the received location second environment information and location information stored on the memory module and determine environment information derived from a combination of the received location environment information and the location information stored on the memory module.

11. The vision-assist system of claim 1, wherein the received second environment information includes inertial environment information and when executed by the processor, the machine readable instructions cause the vision-assist system to:

analyze the received second inertial environment information and inertial information stored on the memory module and determine environment information derived from a combination of the received inertial environment information and the inertial information stored on the memory module.

12. The vision-assist system of claim 1, wherein the received second environment information includes audio environment information and when executed by the processor, the machine readable instructions cause the vision-assist system to:

analyze the received audio second environment information and image information stored on the memory module and determine environment information derived from a combination of the received audio environment information and the audio information stored on the memory module.

13. The vision-assist system of claim 1 wherein the feedback comprises identification of an obstacle or hazard.

14. The vision-assist system of claim 1 wherein the feedback comprises identification of a safe path.

15. The vision-assist system of claim 1 wherein the feedback comprises providing directions.

16. The vision-assist system of claim 1 wherein the at least one sensor comprises an environment camera and, when executed by the processor, the machine readable instructions cause the vision-assist system to:

receive environment image data from the environment camera;

determine a location of an individual speaking to a user based on the environment image data;

receive user eye tracking image data from the user eye tracking camera;

determine a pose of the user'"'"'s eyes based on the user eye tracking image data; and

provide feedback to the user with the feedback device based on the location of the individual speaking to the user and the pose of the user'"'"'s eyes.

17. A method of obtaining environmental information via a vision-assist system and providing feedback to a user of the vision-assist system, the method comprising:

obtaining first environment information from a first region of an environment via at least one sensor;

identifying at least one remote vision-assist device;

authenticating the remote vision-assist device;

establishing a communications session with the remote vision-assist device;

receiving second environment information from the remote vision-assist device;

analyzing the received second environment information on the vision-assist system; and

providing feedback to the user of the vision-assist system based at least in part on the first and second environment information.

18. The method of claim 17 wherein the received second environment information is compared to the first environment information stored on the on a memory module of the vision-assist system.

19. The method of claim 18 wherein the feedback is determined from a combination of the received environment information and environment information stored on the memory module.

20. A vision-assist system comprising:

a necklace module comprising an environment camera mechanically coupled to the necklace module, the environment camera adapted to detect first environment information;

a processor communicatively coupled to the environment camera;

a memory module communicatively coupled to the processor;

a feedback device communicatively coupled to the necklace module; and

machine readable instructions stored in the memory module that cause the vision-assist system to perform at least the following when executed by the processor;

receive identification information of a remote vision-assist system;

authenticate the identification information;

initiate a communication session with the remote vision-assist system;

receive second environment information from the remote vision-assist system; and

provide feedback to the user with the feedback device derived at least in part based on the first and second environment information.

21. The vision-assist system of claim 20 wherein when executed by the processor, the machine readable instructions cause the vision-assist system to receive the first environment information from the environment camera of the necklace module and the feedback is derived from a combination of the first environment information received from the remote vision-assist system and the second environment information received from the environment camera of the necklace module.