Augmented reality assistance system for the visually impaired

• US 10,436,593 B2
• Filed: 11/08/2016
• Issued: 10/08/2019
• Est. Priority Date: 11/08/2016
• Status: Active Grant

• Current Assignee: Reem Jafar Alataas
• Original Assignee: Reem Jafar Alataas
• Inventors: Alataas, Reem Jafar
• Primary Examiner(s): Kong, Sze-Hon
• Application Number:
  US15/345,635
• Publication Number:
  US 20180128622A1
• Time in Patent Office: 1,064 Days
• CPC Class Codes: A61H 3/06 Walking aids for blind persons replacing direct visual perception by another kind of perception A61F9/08
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: G01C 21/005 with correlation of navigation data from several sources, e.g. map or contour matching G01C21/30 takes precedence
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/34 Route searching; Route guidance
• CPC Class Codes: G01C 21/3407 specially adapted for specific applications
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback

Abstract

An augmented reality assistance system can include a wearable augmented reality device. The augmented reality device can include an imaging device, a speaker, a microphone, and one or more sensors. The system can also include a server communicably coupled to the wearable augmented reality device. The server can include processing circuitry configured to receive location information via at least one of the wearable augmented reality device, the one or more sensors, and the microphone, determine amplified location information based on the location information, receive image data from the imaging device, perform a web crawl of the image data, determine a navigation recommendation based on the amplified location information and the web crawl of the image data, and output the amplified location information and the navigation recommendation via the speaker.

Claims

1. An augmented reality assistance system, comprising:

a wearable augmented reality device including;

an imaging device configured to capture image data of an environment surrounding the wearable augmented reality device,

a speaker configured to output audio commands including at least one of amplified location information,

a microphone configured to detect audio data including at least one of verbal commands or sounds from the environment surrounding the wearable augmented reality device, and

one or more additional sensors configured to detect one or more types of additional sensor data associated with the environment surrounding the wearable augmented reality device; and

a server communicably coupled to the wearable augmented reality device, the server including processing circuitry configured to;

receive location information from the wearable augmented reality device including at least one of the image data, the audio data or the additional sensor data,

determine amplified location information corresponding to the environment surrounding the wearable augmented reality device based on the received location information indicating a position of the wearable augmented reality device,

detect one or more objects in the image data corresponding to one or more obstacles based on a web crawl of the image data,

determine a navigation recommendation based on the amplified location information and the web crawl of the image data, the navigation information providing avoidance information of the one or more obstacles, and

output the amplified location information and the navigation recommendation via the speaker of the wearable augmented reality device,

wherein the processing circuitry is configured to determine the amplified location information at a predetermined interval of time based on detected, sensed, or identified obstacles within a predetermined distance of the wearable augmented reality device; and

wherein the processing circuitry is configured to determine one or more trends in observed obstacles, the one or more trends being based on a predetermined number of times the obstacle is observed within a predetermined amount of time, and update the predetermined interval of time based on the one or more trends.

2. The system of claim 1, wherein the navigation recommendation includes at least one of directional navigation, a verbal description corresponding to the web crawl of the image data, or a warning corresponding to an obstacle in view of the imaging device.

3. The system of claim 2, wherein the processing circuitry is configured to determine the navigation recommendation at least in part based on user preferences.

4. The system of claim 1, wherein the processing circuitry is configured to automatically output at least one of the amplified location information or the navigation recommendation at a predetermined interval of time or manually output at least one of the amplified location information or the navigation recommendation in response to receiving a command at the microphone.

5. The system of claim 1, wherein the processing circuitry outputs at least one additional navigation recommendation when an obstacle is detected before the predetermined interval of time expires.

6. The system of claim 5, wherein the processing circuitry is configured to classify the obstacle as moving or static and update the navigation recommendation based on the classification of the obstacle.

7. The system of claim 1, wherein the processing circuitry is further configured to determine if an area through which the augmented reality device is navigating is frequently navigated by the augmented reality device based on a comparison of the amplified location information to stored image data.

8. The system of claim 7, wherein the processing circuitry is further configured to determine the navigation recommendation based on the comparison.

9. The system of claim 7, wherein the processing circuitry is configured to update a risk classification of an obstacle when the obstacle is encountered in the frequently navigated area.

10. A method of navigation via the augmented reality system, comprising:

receiving location information from a wearable augmented reality device including at least one of image data, audio data or additional sensor data;

determining, via processing circuitry, amplified location information corresponding to an environment surrounding the wearable augmented reality device based on the received location information indicating a position of the wearable augmented reality device;

detecting one or more objects in the image data corresponding to one or more obstacles based on a web crawl of the image data;

determining, via processing circuitry, a navigation recommendation based on the amplified location information and the web crawl of the image data, the navigation information providing avoidance information of the one or more obstacles;

outputting the amplified location information and the navigation recommendation via a speaker of the wearable augmented reality device;

determining the amplified location information at a predetermined interval of time based on detected, sensed, or identified obstacles within a predetermined distance of the wearable augmented reality device;

determining one or more trends in observed obstacles, the one or more trends being based on a predetermined number of times the obstacle is observed within a predetermined amount of time; and

updating the predetermined interval of time based on the one or more trends.

11. The method of claim 10, wherein the navigation recommendation includes at least one of directional navigation, a verbal description corresponding to the web crawl of the image data, and a warning corresponding to an obstacle in view of an imaging device coupled to the wearable augmented reality device.

12. The method of claim 10, further comprising:

automatically outputting at least one of the amplified location information or the navigation recommendation at a predetermined interval of time or manually outputting at least one of the amplified location information or the navigation recommendation in response to receiving a command at the microphone.

13. The method of claim 10, further comprising:

outputting at least one additional navigation recommendation when an obstacle is detected before the predetermined interval of time expires.

14. The method of claim 13, further comprising:

classifying the obstacle as moving or static and update the navigation recommendation based on the classification of the obstacle.

15. The method of claim 10, further comprising:

determining if an area through which the augmented reality device is navigating is frequently navigated by the augmented reality device based on observations of an area that has been observed more than a predetermined number of times in a predetermined time interval.