Adaptive visual assistive device

• US 10,571,715 B2
• Filed: 11/05/2012
• Issued: 02/25/2020
• Est. Priority Date: 11/04/2011
• Status: Active Grant

• Current Assignee: Massachusetts Eye And Ear Infirmary (Foundation of The Massachusetts Eye and EAR Infirmary Incorporated), US Department of Veterans Affairs (Government of the United States of America)
• Original Assignee: Massachusetts Eye And Ear Infirmary (Foundation of The Massachusetts Eye and EAR Infirmary Incorporated)
• Inventors: Rizzo, III, Joseph F., Priplata, Attila, Xu, Minnan, Wall, III, Conrad
• Primary Examiner(s): Deherrera, Kristina M
• Application Number:
  US14/356,071
• Publication Number:
  US 20150002808A1
• Time in Patent Office: 2,668 Days
• Field of Search: 351158, 607 53, 607 54, 607141
• CPC Class Codes: A61F 9/08 Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
• CPC Class Codes: A61H 2201/0184 by raising an alarm
• CPC Class Codes: A61H 2201/165 Wearable interfaces
• CPC Class Codes: A61H 2201/5007 computer controlled
• CPC Class Codes: A61H 2201/501 connected to external computer devices or networks
• CPC Class Codes: A61H 2201/503 Inertia activation, i.e. activated by movement
• CPC Class Codes: A61H 2201/5048 Audio interfaces, e.g. voice or music controlled
• CPC Class Codes: A61H 2201/5069 Angle sensors
• CPC Class Codes: A61H 2201/5079 Velocity sensors
• CPC Class Codes: A61H 2201/5084 Acceleration sensors
• CPC Class Codes: A61H 2201/5092 Optical sensor
• CPC Class Codes: A61H 2201/5097 wireless
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: A61N 1/36046 of the eye
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G02C 11/10 Electronic devices other than hearing aids
• CPC Class Codes: G02C 5/001 specially adapted for particular purposes, not otherwise provided for or not fully classifiable according to technical characteristics, e.g. therapeutic glasses
• CPC Class Codes: G03B 2205/0007 Movement of one or more optical elements for control of motion blur imaging systems in general using optical elements for stabilisation of the position of the image G02B27/64
• CPC Class Codes: G03B 2217/005 Blur detection
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 40/63 : for local operation
• CPC Class Codes: H04N 23/6812 : based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/6815 : by distinguishing pan or tilt from motion
• CPC Class Codes: H04N 23/683 : performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/687 : by shifting the lens or sensor position

Abstract

The disclosure features visual assistive devices that include a detection apparatus configured to receive information about an environment surrounding a user of the visual assistive device, a communication apparatus configured to transmit information to the user, and an electronic processor coupled to the detection apparatus and the communication apparatus, and configured to: receive information about an activity of the user; filter the information about the environment surrounding the user based on the activity of the user; assign a priority rank to each element of the filtered information based on the activity of the user; and transmit at least some of the elements of the filtered information, according to the assigned priority ranks, to the user using the communication apparatus.

Claims

1. A visual assistive device, comprising:

a detection apparatus configured to receive information about an environment surrounding a user of the visual assistive device;

a communication apparatus configured to transmit information to the user; and

an electronic processor coupled to the detection apparatus and the communication apparatus, and configured to;

obtain information about an activity of the user, wherein the information comprises at least one of a verbal or non-verbal command issued by the user and received by the electronic processor, an audio signal received by the electronic processor, a gesture of the user, and a position of the user;

filter the information about the environment surrounding the user based on the activity of the user to generate multiple elements of filtered information corresponding to a subset of the information about the environment surrounding the user;

assign a priority rank to each element of the filtered information based on the activity of the user; and

order the elements of the filtered information according to the corresponding priority ranks and transmit the ordered, filtered information to the user using the communication apparatus.

2. The device of claim 1, wherein the detection apparatus comprises at least one image detector configured to obtain images of the environment surrounding the user.

3. The device of claim 1, wherein the detection apparatus comprises at least one of a microphone, a global positioning system (GPS) sensor, a range-finding sensor, an acoustic wave sensor, and a motion sensor.

4. The device of claim 1, further comprising a housing for the detection apparatus configured to be worn on a head of the user.

5. The device of claim 1, wherein the communication apparatus comprises at least one speaker configured to transmit information to the user, and wherein the transmitted information comprises audio information derived from information received by the detection apparatus, and transmitted to the speaker by the electronic processor.

6. The device of claim 1, wherein the communication apparatus comprises at least one vibrotactile transmitter configured to contact skin of the user'"'"'s body, and wherein the electronic processor is configured to transmit information to the user based on the information received by the detection apparatus, the transmitted information comprising vibrotactile signals applied to the skin of the user'"'"'s body.

7. The device of claim 6, wherein the communication interface comprises:

a pair of vibrotactile transmitters positioned on opposite sides of the device, wherein the pair of transmitters are configured to transmit navigation information to the user; and

at least one additional vibrotactile transmitter, wherein the at least one additional vibrotactile transmitter is configured to transmit another type of information to the user different from navigation information,

wherein the pair of vibrotactile transmitters are configured to apply at least one of;

vibrotactile signals to the skin of the user'"'"'s body at a first frequency, and the at least one additional vibrotactile transmitter is configured to apply vibrotactile signals to the skin of the user'"'"'s body at a second frequency different from the first frequency; and

vibrotactile signals to the skin of the user'"'"'s body at a first signal amplitude, and the at least one additional vibrotactile transmitter is configured to apply vibrotactile signals to the skin of the user'"'"'s body at a second signal amplitude different from the first signal amplitude.

8. The device of claim 1, further comprising a wireless communication interface, wherein the electronic processor is coupled through the wireless communication interface to an implanted prosthetic device in the user'"'"'s body and configured to transmit information to the implanted prosthetic device using the wireless communication interface.

9. The device of claim 1, wherein the elements comprise at least one of:

one or more objects in visual information received by the detection apparatus;

one or more faces in visual information received by the detection apparatus;

a position of the user in positioning information received by the detection apparatus; and

one or more portions of text in visual information received by the detection apparatus.

10. The device of claim 1, wherein prior to transmitting the ordered filtered information to the user, the electronic processor is configured to further process the filtered information by assigning an identity to elements that correspond to faces in visual information received by the detection apparatus.

11. The device of claim 10, wherein the electronic processor is further configured to refine the elements of the filtered information by transmitting the elements to a remote electronic processor that refines the elements, and receiving information corresponding to the refined elements from the remote electronic processor.

12. The device of claim 1, wherein prior to transmitting the ordered filtered information to the user, the electronic processor is configured to further process the filtered information by assigning an identity to elements that correspond to objects in visual information received by the detection apparatus.

13. The device of claim 1, wherein prior to transmitting the ordered, filtered information to the user, the electronic processor is configured to assign an identity to a human speaker of elements that correspond to speech.

14. The device of claim 1, wherein the electronic processor is configured to assign a priority rank to each element of the filtered information by assigning a first priority rank to elements of the filtered information that correspond to hazards in the environment surrounding the user.

15. The device of claim 14, wherein the electronic processor is configured to assign a priority rank lower than first to elements of the filtered information that correspond to signs, buildings, landmarks, location signals from embedded radiofrequency tags, and navigational instructions.

16. The device of claim 1, wherein the activity of the user is reading, and wherein the electronic processor is configured to assign a highest priority rank to elements of the filtered information that correspond to headlines and titles, and a lower priority rank to elements of the filtered information that correspond to other text.

17. The device of claim 1, wherein the electronic processor is configured to assign the priority rank to each element corresponding to an object in visual information received by the detection apparatus based on at least one of a distance between the object and the user, and an angular relationship between the object and a direction in which the user is oriented.

18. A visual assistive system, comprising:

the device of claim 1; and

a visual prosthetic device, embedded;

in, or in proximity to, an eye of the user;

or in, or in proximity to, an optic nerve of the user;

or within, along, or in proximity to a visual pathway or visual region of a brain of the user,

wherein the electronic processor is configured to transmit at least some of the elements of the filtered information derived from visual information received by the detection apparatus to the visual prosthetic device; and

wherein the visual prosthetic device is configured to transmit a representation of each of the transmitted elements to at least one of the eye of the user, the optic nerve of the user, and the visual pathway or visual region of the brain of the user.

19. The device of claim 1, wherein the electronic processor is configured to adjust a configuration of the detection apparatus prior to receiving at least some of the information about the environment surrounding the user.

20. The device of claim 19, wherein the electronic processor is configured to:

determine components of motion of the user in visual information received by the detection apparatus; and

adjust the configuration of the detection apparatus to compensate for at least some of the components of motion.

27. The device of claim 1, wherein prior to transmitting the ordered filtered information, the electronic processor is configured to:

re-assess the priority ranks of one or more elements of the filtered information; and

transmit the ordered, filtered information according to the re-assessed priority ranks.

28. The device of claim 27, wherein the electronic processor is configured to:

determine whether a triggering event for re-assessment of the priority ranks has occurred prior to transmitting the filtered information; and

re-assess the priority ranks only if a triggering event has occurred.

29. The device of claim 27, wherein the electronic processor is configured to re-assess the priority ranks by retrieving a set of rules from a database based on the activity of the user, and applying the set of rules to the elements of the filtered information.

21. A method for providing environmental information to a visually impaired subject, the method comprising:

receiving information about an environment surrounding the subject;

determining an activity of the subject based on at least one of a received verbal or non-verbal command issued by the subject, a received audio signal, a gesture of the subject, and a position of the subject;

filtering the information about the environment surrounding the subject based on the subject'"'"'s activity to generate multiple elements of filtered information corresponding to a subset of the information about the environment surrounding the subject;

assigning a priority rank to each element of the filtered information based on the subject'"'"'s activity; and

ordering the elements of the filtered information according to the corresponding priority ranks and transmitting the ordered, filtered information to the subject.

22. The method of claim 21, wherein filtering the information about the environment surrounding the subject based on the subject'"'"'s activity comprises filtering the information based on one or more threshold conditions associated with the activity of the subject.

23. The method of claim 21, further comprising, prior to assigning the priority ranks, identifying one or more of the elements of the filtered information, wherein identifying one or more of the elements of the filtered information comprises at least one of locating objects in visual information about the environment surrounding the subject, locating faces in visual information about the environment surrounding the subject, determining a position of the subject relative to the environment surrounding the subject, and locating portions of text in the visual information about the environment surrounding the subject.

24. The method of claim 21, further comprising, prior to transmitting the ordered, filtered information to the subject, further processing the elements of the filtered information to refine the elements, wherein refining the elements comprises at least one of assigning an identity to elements that correspond to faces in visual information about the environment surrounding the subject, assigning an identity to elements that correspond to objects in visual information about the environment surrounding the subject, assigning an identity to a speaker of elements that correspond to speech in audio information about the environment surrounding the subject, and identifying a color associated with elements that correspond to objects in visual information about the environment surrounding the subject.

25. The method of claim 21, wherein assigning a priority rank to each element of the filtered information comprises assigning a first priority rank to elements of the filtered information that correspond to hazards in the environment surrounding the subject, and assigning a priority rank lower than first to elements of the filtered information that correspond to signs, buildings, landmarks, location signals from embedded radiofrequency tags, and navigational instructions.

26. The method of claim 21, further comprising, for each element corresponding to an object in the environment surrounding the subject, assigning the element'"'"'s priority rank based on at least one of a distance between the object and the subject, and an angular relationship between the object and a direction in which the subject is oriented.