Vision-assist systems with room scanners to detect and notify users of out-of-order room states

• US 10,408,625 B2
• Filed: 04/19/2016
• Issued: 09/10/2019
• Est. Priority Date: 04/19/2016
• Status: Active Grant

• Current Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Lee, Christopher P.
• Primary Examiner(s): Dager, Jonathan M
• Assistant Examiner(s): Evans, Garrett F
• Application Number:
  US15/132,860
• Publication Number:
  US 20170299394A1
• Time in Patent Office: 1,239 Days
• Field of Search: None
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09B 21/006 using audible presentation of the information

Abstract

Vision-assist systems that inform visually impaired users regarding the current state of a room are disclosed. In some embodiments, a system includes an image sensor for obtaining a baseline image of the room including one or more items in the room positioned in an in-order state, and memory to store the baseline image. The system may include a motion sensor to detect when the user is approaching the room to activate the image sensor to obtain a real-time image of the room. A detection module determines whether the room and items in the room are in an in-order state or an out-of-order state when compared with the baseline image, and an output module notifies the user of the current state of room. In this manner, a visually impaired user can be informed of the state of a room which thereby aids the user in navigating or moving within the room.

Claims

1. A wearable vision-assist system for aiding a visually impaired user to navigate a room, comprising:

an image sensor for obtaining a baseline image of the room including one or more items in the room positioned in an in-order state, wherein the baseline image comprises a frame of reference of the one or more items in room with respect to a fixed item within the room;

a location sensor configured to determine location data of the wearable vision-assist system;

a memory operatively coupled with the image sensor and the location sensor, the memory storing the baseline image and location data correlated to the baseline image;

a sensor operatively coupled with the image sensor, the sensor configured to detect when a user is approaching the room, and the sensor configured to activate the image sensor to obtain a real-time image of the room, wherein the real-time image comprises a frame of reference of the one or more items in room with respect to the fixed item within the room;

a processor coupled to the memory and the location sensor and configured to correlate the baseline image with the room based upon the location data determined by the location sensor, determine whether the room including the one or more items in the room are in an in-order state or are in an out-of-order state when compared with the baseline image, wherein the processor is configured to determine a location or distance of the one or more items relative to the fixed item from the baseline image and the real-time image; and

an output device configured to provide an output signal to notify the user of whether the room including the one or more items in the room are in an in-order state or an out-of-order state,

wherein the vision-assist system is configured as a mobile wearable device.

2. The vision-assist system of claim 1, wherein the image sensor includes a wide angle lens.

3. The vision-assist system of claim 1, wherein the image sensor includes a fish-eye lens.

4. The vision-assist system of claim 1, wherein the baseline image is obtained in response to an input from the user while the one or more items in the room are positioned in the in-order state.

5. The vision-assist system of claim 1, wherein the processor compares the real-time image against the baseline image.

6. The vision-assist system of claim 1, wherein the processor determines a location of the one or more items in the room from the baseline image, and the processor determines the location of the one or more items in the room from the real-time image.

7. The vision-assist system of claim 1, wherein the output device provides an audible signal to the user that the room is in an out-of-order state.

8. The vision-assist system of claim 1, wherein the output device provides a signal to the user that the room is in an out-of-order state when the real-time image indicates that the one or more items in the room are in different positions than their respective positions in the baseline image.

9. The vision-assist system of claim 1, wherein the output device provides a signal to the user that the room is in an out-of-order state when the real-time image indicates that the one or more items are missing from the room when compared with the baseline image.

10. The vision-assist system of claim 1, wherein the output device provides a signal to the user that the room is in an out-of-order state when the real-time image indicates that additional items are present in the room when compared with the baseline image.

11. The vision-assist system of claim 1, wherein the location sensor comprises a GPS unit to provide GPS position location data associated with the baseline image.

12. A wearable vision-assist system for aiding a visually impaired user to navigate a room, comprising:

an image sensor for obtaining an image of the room including one or more items in the room positioned in an in-order state, the image sensor activated in part based on received voice commands, wherein the image comprises a frame of reference of the one or more items in the room with respect to a fixed item within the room;

a location sensor configured to determine location data of the wearable vision-assist system;

a memory operatively coupled with the image sensor and the location sensor, the memory storing the image and location data correlated to the image;

a processor operatively coupled with the image sensor for obtaining a real-time image of the room including the one or more items, with the memory for accessing the image and with the location sensor for obtaining location data of the wearable vision-assist system for the real-time image and identifying the room based upon the location data determined by the location sensor for the real-time image, the processor configured to determine whether a current state of the room including the one or more items in the room is in an in-order state or is in an out-of-order state, wherein the processor is configured to determine a location or distance of the one or more items relative to the fixed item from the stored image and the real-time image; and

an output device configured to provide an output signal to notify the user of the current state of the room,

wherein the vision-assist system is configured as a mobile wearable device to be worn by the user.

13. The vision-assist system of claim 12, wherein the processor is activated to obtain the real-time image upon the user approaching the room.

14. The vision-assist system of claim 12, wherein the output device provides a signal to the user that the room is in an out-of-order state when the real-time image indicates that one or more items in the room are in different positions than their respective positions in the image.

15. The vision-assist system of claim 12, wherein the output device provides a signal to the user that the room is in an out-of-order state when the real-time image indicates that the one or more items are missing from the room when compared with the image.

16. The vision-assist system of claim 12, wherein the output device provides a signal to the user that the room is in an out-of-order state when the real-time image indicates that additional items are present in the room when compared with the image.

17. A wearable vision-assist system for aiding a visually impaired user, comprising:

an image sensor for obtaining an image of a room including one or more items in the room positioned in an in-order state, wherein the image comprises a frame of reference of the one or more items in the room with respect to a fixed item within the room;

a location sensor configured to determine location data of the wearable vision-assist system;

a memory operatively coupled with the image sensor and location data correlated to the image, the memory storing the image and location data correlated to the image;

a sensor operatively coupled with the image sensor, the sensor configured to detect when the user is approaching the room and in response activate the image sensor to obtain a real-time image of the room, wherein the real-time image comprises a frame of reference of the one or more items in the room with respect to the fixed item within the room;

a processor operatively coupled with the memory and configured to correlate the stored image with the room based upon the location data determined by the location sensor and to analyze one or more locations of the one or more items in the room as captured in the real-time image and the stored image and to determine whether a current state of the room including the one or more items in the room is in an in-order state or is in an out-of-order state, wherein the processor is configured to determine a location or distance of the one or more items relative to the fixed item from the stored image and the real-time image; and

an output device configured to provide an output signal to notify the user of the current state of the room,

wherein the vision-assist system is configured as a mobile wearable device to be worn by the user.

18. The system of claim 17, wherein the output device provides a signal to the user that the room is in an out-of-order state when the real-time image indicates that one or more additional items are present in the room when compared with the image of the room in an in-order state.

19. The system of claim 17, wherein the output device provides a signal to the user that the room is in an out-of-order state when the real-time image indicates that the one or more items are missing from the room when compared with the image of the room in an in-order state.