Vision-assist systems for orientation and mobility training

• US 10,062,302 B2
• Filed: 03/03/2016
• Issued: 08/28/2018
• Est. Priority Date: 03/03/2016
• Status: Active Grant

• Current Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Chen, Tiffany
• Primary Examiner(s): McClellan, James S
• Application Number:
  US15/059,413
• Publication Number:
  US 20170256181A1
• Time in Patent Office: 908 Days
• CPC Class Codes: G09B 21/003 using tactile presentation of the information, e.g. Braille displays

Abstract

Vision-assist systems and methods include use of processor(s) and a communicatively coupled feedback device and environmental sensing device upon which generated feedback is based. Machine readable instructions stored on a memory module cause the system to perform the following when executed by the processor(s): record a first metric corresponding to a level of reliance by a user on the feedback device at a first time; record a second metric corresponding to the level of reliance by the user on the feedback device at a second time after the first time; and record an awarded amount of reliance points when the second metric is less than the first metric, indicative of a decrease on a reliance by the user on the vision-assist system. The awarded amount of reliance points corresponds to a percentage by which the second metric decreases compared to the first metric.

Claims

1. A vision-assist system comprising:

one or more processors;

a memory module communicatively coupled to the one or more processors;

an environmental sensing device communicatively coupled to the one or more processors and operable to generate information regarding an environment;

a feedback device communicatively coupled to the one or more processors and operable to generate feedback at least partially based on the generated environmental information of the environmental sensing device; and

machine readable instructions stored in the memory module that cause the vision-assist system to perform at least the following when executed by the one or more processors;

record a first metric corresponding to a level of reliance by a user on the feedback device at a first time, wherein the level of reliance is based at least in part on a measurable dependence on the generated feedback, and wherein the level of reliance is at least partially based on at least one of;

a Braille assist feature associated with the environmental sensing device and the memory module,

an object identification assist feature associated with the environmental sensing device and the memory module to assist the user in identifying an object, and

a facial recognition assist feature to assist the user in facial recognition;

record a second metric corresponding to the level of reliance by the user on the feedback device at a second time, wherein the second time occurs after the first time; and

record an awarded amount of reliance points when the second metric is less than the first metric, indicative of a decrease on a reliance by the user on the vision-assist system, wherein the awarded amount of reliance points corresponds to a percentage by which the second metric decreases compared to the first metric.

2. The vision-assist system of claim 1, wherein when executed by the one or more processors, the machine readable instructions cause the vision-assist system to:

receive environmental image data from the environmental sensing device;

provide at least a portion of the generated feedback with the feedback device of at least one of;

one or more definitions of Braille text viewable to the environmental sensing device upon user initiated request, and

indications of user error when the user reads the Braille text at least one of automatically and upon user initiated request.

3. The vision-assist system of claim 1, wherein when executed by the one or more processors, the machine readable instructions cause the vision-assist system to:

receive environmental image data from the environmental sensing device;

determine a location of an obstacle comprising the object within a predetermined range based on the environmental image data; and

provide at least a portion of the generated feedback with the feedback device based on the location of the obstacle.

4. The vision-assist system of claim 3, wherein when executed by the one or more processors, the machine readable instructions cause the vision-assist system to:

detect when a cane utilized by the user and contacts the obstacle;

provide at least a portion of the generated feedback with the feedback device when the cane utilized by the user contacts the obstacle; and

award one or more user training progression points based on at least one of time to contact the obstacle with the cane and an amount of user initiated requests for feedback regarding the location of the obstacle.

5. The vision-assist system of claim 1, wherein when executed by the one or more processors, the machine readable instructions cause the vision-assist system to:

receive environmental image data from the environmental sensing device;

determine an identification of the object within a predetermined range based on the environment image data by recognizing the object in the received environmental image data using an object recognition algorithm; and

provide at least a portion of the generated feedback with the feedback device based on the determined identification of the object.

6. The vision-assist system of claim 5, wherein when executed by the one or more processors, the machine readable instructions cause the vision-assist system to:

prompt the user to provide user input regarding a guessed identification of the object; and

award one or more points when the guessed identification of the object matches the determined identification of the object.

7. The vision-assist system of claim 1, wherein when executed by the one or more processors, the machine readable instructions cause the vision-assist system to:

receive environmental image data from the environmental sensing device;

determine an identity of a person based on the environmental image data by recognizing a face in the received environmental image data using a facial recognition algorithm; and

provide at least a portion of the generated feedback with the feedback device based on the determined identity of the person at least one of automatically or upon user initiated request.

8. The vision-assist system of claim 7, wherein when executed by the one or more processors, the machine readable instructions cause the vision-assist system to:

prompt the user to provide user input regarding a guessed identity of the person prior to providing at least a portion of the generated feedback with the feedback device based on the determined identity of the person; and

award one or more points when the guessed identity of the person matches the determined identity of the person.

9. The vision-assist system of claim 1, wherein when executed by the one or more processors, the machine readable instructions cause the vision-assist system to:

store the awarded amount of reliance points in a cloud database system communicatively coupled to the one or more processors; and

provide one or more reports based on at least one of the awarded amount of reliance points and one or more other user training progression tracking metrics based on user interaction with the vision-assist system.

10. A method for awarding points based on reliance on a vision-assist system comprising a global positioning system sensor and a memory module, each communicatively coupled to one or more processors of the vision-assist system, the method comprising:

recording a first metric corresponding to a level of reliance on a feedback device by a user on the vision-assist system at a first time, wherein;

an environmental sensing device comprising at least one of an environment camera and one or more microphones is communicatively coupled to the one or more processors and is operable to generate information regarding an environment,

the feedback device is communicatively coupled to one or more processors of the vision-assist system and is operable to generate feedback at least partially based on the generated environmental information of the environmental sensing device, and

the level of reliance on the feedback device by the user is at least partially based on at least one of;

a navigation feature utilizing data from the global positioning system sensor and associated with the environmental sensing device,

a Braille assist feature associated with the environmental sensing device and the memory module,

an object identification assist feature associated with the environmental sensing device and the memory module, and

a facial recognition feature associated with the environmental sensing device and the memory module;

recording a second metric corresponding to the level of reliance on the feedback device by the user on the vision-assist system at a second time, wherein the second time occurs after the first time;

awarding an amount of reliance points when the second metric is less than the first metric, indicative of a decrease on a reliance by the user on the vision-assist system, wherein the amount of reliance points corresponds to a percentage by which the second metric decreases compared to the first metric; and

storing in a cloud database system communicatively coupled to the one or more processors at least one of the amount of reliance points and one or more other user training progression tracking metrics based on user interaction with the vision-assist system.

11. A vision-assist system comprising:

one or more processors;

a global positioning system sensor communicatively coupled to the one or more processors;

a memory module communicatively coupled to the one or more processors;

an environmental sensing device communicatively coupled to the one or more processors and operable to generate information regarding an environment;

a feedback device communicatively coupled to the one or more processors and operable to generate feedback at least partially based on the generated environmental information of the environmental sensing device; and

machine readable instructions stored in the memory module that cause the vision-assist system to perform at least the following when executed by the one or more processors;

record a first metric corresponding to a level of reliance by a user on the feedback device at a first time, wherein the level of reliance is based at least in part on a measurable dependence on the generated feedback, the level of reliance is at least partially based on a navigation feature utilizing data from the global positioning system sensor, and the navigation feature determines at least one or more navigational directions providable to the user;

record a second metric corresponding to the level of reliance by the user on the feedback device at a second time, wherein the second time occurs after the first time; and

record an awarded amount of reliance points when the second metric is less than the first metric, indicative of a decrease on a reliance by the user on the vision-assist system, wherein the awarded amount of reliance points corresponds to a percentage by which the second metric decreases compared to the first metric.

12. The vision-assist system of claim 11, wherein when executed by the one or more processors, the machine readable instructions cause the vision-assist system to:

determine a route based on the data from the global positioning system sensor;

receive environmental image data from the environmental sensing device; and

provide at least a portion of the generated feedback with the feedback device of the one or more navigational directions at pre-determined points along the route to the user based on the received environmental image data and the data from the global positioning system sensor.

13. The vision-assist system of claim 12, wherein the provided feedback is based on a user initiated feedback request.

14. The vision-assist system of claim 12, wherein an amount of feedback provided by the feedback device decreases over a period of use such that an amount of the one or more navigation directions provided in an earlier use of the vision-assist system is greater than the amount of the one or more navigation directions provided in a later use of the vision-assist system use after the earlier use.

15. The vision-assist system of claim 14, wherein the decrease in the amount of feedback provided is controlled via user input.

16. The vision-assist system of claim 14, wherein the decrease in the amount of feedback provided is automatically controlled by a feedback reduction algorithm that reduces the amount of feedback based on a pre-determined amount over one or more pre-determined time period intervals.