ADAPTIVE VISUAL ASSISTIVE DEVICE

• US 20150002808A1
• Filed: 11/05/2012
• Published: 01/01/2015
• Est. Priority Date: 11/04/2011
• Status: Active Grant

• Current Assignee: Massachusetts Eye And Ear Infirmary (Foundation of The Massachusetts Eye and EAR Infirmary Incorporated), US Department of Veterans Affairs (Government of the United States of America)
• Original Assignee: Massachusetts Eye And Ear Infirmary (Foundation of The Massachusetts Eye and EAR Infirmary Incorporated), US Department of Veterans Affairs (Government of the United States of America)
• Inventors: Rizzo, Joseph F. III, Xu, Minnan, Wall, Conrad III, Priplata, Attila
• Granted Patent: US 10,571,715 B2
• Time in Patent Office: Days
• US Class Current: 351/158
• CPC Class Codes: A61F 9/08 Devices or methods enabling eye-patients to replace direct visual perception by another kind of perception
• CPC Class Codes: A61H 2201/0184 by raising an alarm
• CPC Class Codes: A61H 2201/165 Wearable interfaces
• CPC Class Codes: A61H 2201/5007 computer controlled
• CPC Class Codes: A61H 2201/501 connected to external computer devices or networks
• CPC Class Codes: A61H 2201/503 Inertia activation, i.e. activated by movement
• CPC Class Codes: A61H 2201/5048 Audio interfaces, e.g. voice or music controlled
• CPC Class Codes: A61H 2201/5069 Angle sensors
• CPC Class Codes: A61H 2201/5079 Velocity sensors
• CPC Class Codes: A61H 2201/5084 Acceleration sensors
• CPC Class Codes: A61H 2201/5092 Optical sensor
• CPC Class Codes: A61H 2201/5097 wireless
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: A61N 1/36046 of the eye
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G02C 11/10 Electronic devices other than hearing aids
• CPC Class Codes: G02C 5/001 specially adapted for particular purposes, not otherwise provided for or not fully classifiable according to technical characteristics, e.g. therapeutic glasses
• CPC Class Codes: G03B 2205/0007 Movement of one or more optical elements for control of motion blur imaging systems in general using optical elements for stabilisation of the position of the image G02B27/64
• CPC Class Codes: G03B 2217/005 Blur detection
• CPC Class Codes: G16H 20/70 relating to mental therapies, e.g. psychological therapy or autogenous training
• CPC Class Codes: G16H 40/63 : for local operation
• CPC Class Codes: H04N 23/6812 : based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/6815 : by distinguishing pan or tilt from motion
• CPC Class Codes: H04N 23/683 : performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/687 : by shifting the lens or sensor position

Abstract

The disclosure features visual assistive devices that include a detection apparatus configured to receive information about an environment surrounding a user of the visual assistive device, a communication apparatus configured to transmit information to the user, and an electronic processor coupled to the detection apparatus and the communication apparatus, and configured to: receive information about an activity of the user; filter the information about the environment surrounding the user based on the activity of the user; assign a priority rank to each element of the filtered information based on the activity of the user; and transmit at least some of the elements of the filtered information, according to the assigned priority ranks, to the user using the communication apparatus.

Claims

1. A visual assistive device, comprising:

a detection apparatus configured to receive information about an environment surrounding a user of the visual assistive device;

a communication apparatus configured to transmit information to the user; and

an electronic processor coupled to the detection apparatus and the communication apparatus, and configured to;

receive information about an activity of the user;

filter the information about the environment surrounding the user based on the activity of the user;

assign a priority rank to each element of the filtered information based on the activity of the user; and

transmit at least some of the elements of the filtered information, according to the assigned priority ranks, to the user using the communication apparatus.

2. The device of claim 1, wherein the detection apparatus comprises at least one image detector configured to obtain images of the environment surrounding the user.

3. The device of any one of claims 1 and 2, wherein the detection apparatus comprises at least one microphone.

4. The device of any one of claims 1 to 3, wherein the detection apparatus comprises at least one global positioning system (GPS) sensor.

5. The device of any one of claims 1 to 4, wherein the detection apparatus comprises at least one of a range-finding sensor and an acoustic wave sensor.

6. The device of any one of claims 1 to 5, wherein the detection apparatus comprises at least one motion sensor.

7. The device of any one of claims 1 to 6, further comprising a housing for the detection apparatus configured to be worn on a head of the user.

8. The device of claim 7, wherein the housing comprises the electronic processor.

9. The device of any one of claims 1 to 8, further comprising a housing for the electronic processor configured to be worn on a portion of the user'"'"'s body other than the head.

10. The device of any one of claims 1 to 9, wherein the detection apparatus and the electronic processor are coupled through a wireless communication interface.

11. The device of any one of claims 1 to 10, wherein the communication apparatus and the electronic processor are coupled through a wireless communication interface.

12. The device of any one of claims 1 to 11, wherein the communication interface comprises at least one speaker configured to transmit information to the user, and wherein the transmitted information comprises audio information derived from information received by the detection apparatus, and transmitted to the speaker by the electronic processor.

13. The device of any one of claims 1 to 12, wherein the communication interface comprises at least one vibrotactile transmitter configured to contact skin of the user'"'"'s body, and wherein the electronic processor is configured to transmit information to the user based on the information received by the detection apparatus, the transmitted information comprising vibrotactile signals applied to the skin of the user'"'"'s body.

14. The device of any one of claims 1 to 13, further comprising a wireless communication interface, wherein the electronic processor is coupled through the wireless communication interface to an implanted prosthetic device in the user'"'"'s body and configured to transmit information to the implanted prosthetic device using the wireless communication interface.

15. The device of claim 14, wherein the transmitted information comprises visual information derived from the information received by the detection apparatus.

16. The device of any one of claims 1 to 15, wherein the electronic processor is configured to receive information about the activity of the user by processing at least one of an audio command and a non-verbal command issued by the user and received by the detection apparatus.

17. The device of any one of claims 1 to 16, wherein the electronic processor is configured to receive information about the activity of the user by processing at least one of visual, audio, distance, gesture, and textual information received by the detection apparatus.

18. The device of any one of claims 1 to 17, wherein the electronic processor is configured to receive information about the activity of the user by processing position information received by the detection apparatus.

19. The device of any one of claims 1 to 18, wherein prior to assigning the priority ranks, the electronic processor is configured to identify one, a plurality, or all of the elements of the filtered information.

20. The device of any one of claims 1 to 19, wherein the elements comprise one or more objects in visual information received by the detection apparatus.

21. The device of any one of claims 1 to 20, wherein the elements comprise one or more faces in visual information received by the detection apparatus.

22. The device of any one of claims 1 to 21, wherein the elements comprise a position of the user in positioning information received by the detection apparatus.

23. The device of any one of claims 1 to 22, wherein the elements comprise one or more portions of text in visual information received by the detection apparatus.

24. The device of any one of claims 1 to 23, wherein prior to transmitting the at least some elements of the filtered information to the user, the electronic processor is configured to further process the at least some elements of the filtered information to refine the at least some elements.

25. The device of claim 24, wherein refining the at least some elements comprises assigning an identity to elements that correspond to faces in visual information received by the detection apparatus.

26. The device of any one of claims 24 and 25, wherein refining the at least some elements comprises assigning an identity to elements that correspond to objects in visual information received by the detection apparatus.

27. The device of any one of claims 24 to 26, wherein refining the at least some elements comprises assigning an identity to a speaker of elements that correspond to speech in audio information received by the detection apparatus.

28. The device of any one of claims 24 to 27, wherein refining the at least some elements comprises identifying a color associated with elements that correspond to objects in visual information received by the detection apparatus.

29. The device of any one of claims 1 to 28, wherein the electronic processor is configured to filter the information about the environment surrounding the user by determining that the activity of the user comprises at least one task associated with everyday living.

30. The device of any one of claims 1 to 29, wherein the electronic processor is configured to filter the information about the environment surrounding the user by applying one or more threshold conditions associated with the activity of the user.

31. The device of any one of claims 1 to 30, wherein the electronic processor is configured to assign a priority rank to each element of the filtered information by assigning a first priority rank to elements of the filtered information that correspond to hazards in the environment surrounding the user.

32. The device of claim 31, wherein the electronic processor is configured to assign a priority rank lower than first to elements of the filtered information that correspond to signs, buildings, landmarks, location signals from embedded radiofrequency tags, and navigational instructions.

33. The device of any one of claims 1 to 30, wherein the activity of the user is reading, and wherein the electronic processor is configured to assign a highest priority rank to elements of the filtered information that correspond to headlines and titles, and a lower priority rank to elements of the filtered information that correspond to other text.

34. The device of any one of claims 1 to 30, wherein the electronic processor is configured to assign the priority rank to each element corresponding to an object in visual information received by the detection apparatus based on at least one of a distance between the object and the user, and an angular relationship between the object and a direction in which the user is oriented.

35. The device of any one of claims 24 to 28, wherein the electronic processor is further configured to refine the elements of the filtered information by transmitting the elements to a remote electronic processor, and receiving information corresponding to the refined elements from the remote electronic processor.

36. The device of claim 35, wherein the remote electronic processor comprises a remote computer system.

37. The device of claim 7, wherein the housing comprises frames configured to support eyeglass lenses.

38. A visual assistive system, comprising:

the device of any one of claims 1 to 37, 55 to 57, 61 to 63, 67 to 71, and 73; and

a visual prosthetic device, embedded;

in, or in proximity to, an eye of the user;

or in, or in proximity to, an optic nerve of the user;

or within, along, or in proximity to a visual pathway or visual region of a brain of the user,

wherein the electronic processor is configured to transmit at least some of the elements of the filtered radiation derived from visual information received by the detection apparatus to the visual prosthetic device; and

wherein the visual prosthetic device is configured to transmit a representation of each of the transmitted elements to at least one of the eye of the user, the optic nerve of the user, and the visual pathway or visual region of the brain of the user.

55. The device of any one of claims 1 to 37, wherein prior to transmitting the at least some elements of the filtered information, the electronic processor is configured to:

re-assess the priority ranks of one or more elements of the filtered information; and

transmit the one or more elements of the filtered information according to the re-assessed priority ranks.

56. The device of claim 55, wherein the electronic processor is configured to:

determine whether a triggering event for re-assessment of the priority ranks has occurred prior to transmitting the at least some elements of the filtered information; and

re-assess the priority ranks only if a triggering event has occurred.

57. The device of any one of claims 55 and 56, wherein the electronic processor is configured to re-assess the priority ranks by retrieving a set of rules from a database based on the activity of the user, and applying the set of rules to the one or more elements of the filtered information.

61. The device of any one of claims 1 to 37 and 55 to 57, wherein:

the detection apparatus is configured to receive additional information about the environment surrounding the user; and

the electronic processor is configured to adjust a configuration of the detection apparatus prior to the apparatus receiving the additional information.

62. The device of claim 61, wherein the electronic processor is configured to:

determine components of motion of the user in visual information received by the detection apparatus; and

adjust the configuration of the detection apparatus to compensate for at least some of the components of motion.

63. The device of claim 62, wherein the at least some of the components of motion correspond to involuntary motion of the user.

67. The device of claim 13, wherein the communication interface comprises:

a pair of vibrotactile transmitters positioned on opposite sides of the device, wherein the pair of transmitters are configured to transmit navigation information to the user; and

at least one additional vibrotactile transmitter, wherein the at least one additional vibrotactile transmitter is configured to transmit another type of information to the user different from navigation information.

68. The device of claim 67, wherein the pair of vibrotactile transmitters are configured to apply vibrotactile signals to the skin of the user'"'"'s body at a first frequency, and the at least one additional vibrotactile transmitter is configured to apply vibrotactile signals to the skin of the user'"'"'s body at a second frequency different from the first frequency.

69. The device of any one of claims 67 and 68, wherein the pair of vibrotactile transmitters are configured to apply vibrotactile signals to the skin of the user'"'"'s body at a first signal amplitude, and the at least one additional vibrotactile transmitter is configured to apply vibrotactile signals to the skin of the user'"'"'s body at a second signal amplitude different from the first signal amplitude.

70. The device of any one of claims 1 to 37, 55 to 57, 61 to 63, and 67 to 69, wherein the communication interface comprises at least one display configured to transmit information to the user, and wherein the transmitted information comprises visual information derived from information received by the detection apparatus, and transmitted to the display by the electronic processor.

71. The device of any one of claims 1 to 37, 55 to 57, 61 to 63, and 67 to 70, wherein at least one element of the filtered information corresponds to text in the environment surrounding the user, and wherein the electronic processor is configured to assign a priority rank to the at least one element corresponding to text based on a distance between the user and the text.

73. The device of any one of claims 1 to 37, 55 to 57, 61 to 63, and 67 to 71, wherein the electronic processor is configured to determine an identity of the user.

39. A method for providing environmental information to a visually impaired subject, the method comprising:

receiving information about an environment surrounding the subject;

determining an activity of the subject;

filtering the information about the environment surrounding the subject based on the subject'"'"'s activity;

assigning a priority rank to each element of the filtered information based on the subject'"'"'s activity; and

transmitting at least some of the elements of the filtered information, according to the assigned priority ranks, to the subject.

40. The method of claim 39, wherein filtering the information about the environment surrounding the subject based on the subject'"'"'s activity comprises filtering the information based on one or more threshold conditions associated with the activity of the subject.

41. The method of any one of claims 39 and 40, wherein the information about the environment surrounding the subject comprises one or more of visual information about the environment, audio information about the environment, motion information about the subject, motion information about the environment, position information about the subject, position information about one or more objects in the environment, an ambient light intensity of the environment, information indicating whether the subject is indoors or outdoors, and information about distances between objects and the subject in the environment.

42. The method of any one of claims 39 to 41, wherein transmitting the at least some elements of the filtered information to the subject comprises transmitting the at least some elements to a communications interface, and using the communications interface to transmit the at least some elements to the subject.

43. The method of claim 42, wherein the at least some elements of the filtered information are transmitted wirelessly to the communications interface.

44. The method of any one of claims 39 to 43, wherein determining an activity of the subject comprises processing at least one of an audio command issued by the subject and a non-verbal command issued by the subject.

45. The method of any one of claims 39 to 44, wherein determining an activity of the subject comprises processing at least one of visual, audio, position, distance, gesture, and textual information about the environment surrounding the subject.

46. The method of any one of claims 39 to 45, further comprising, prior to assigning the priority ranks, identifying one or more of the elements of the filtered radiation.

47. The method of claim 46, wherein identifying one or more of the elements of the filtered radiation comprises at least one of locating objects in visual information about the environment surrounding the subject, locating faces in visual information about the environment surrounding the subject, determining a position of the subject relative to the environment surrounding the subject, and locating portions of text in the visual information about the environment surrounding the subject.

48. The method of any one of claims 39 to 47, further comprising, prior to transmitting the at least some elements of the filtered information to the subject, further processing the at least some elements of the filtered information to refine the at least some elements.

49. The method of claim 48, wherein refining the at least some elements comprises at least one of assigning an identity to elements that correspond to faces in visual information about the environment surrounding the subject, assigning an identity to elements that correspond to objects in visual information about the environment surrounding the subject, assigning an identity to a speaker of elements that correspond to speech in audio information about the environment surrounding the subject, and identifying a color associated with elements that correspond to objects in visual information about the environment surrounding the subject.

50. The method of any one of claims 39 to 49, wherein the activity of the subject comprises at least one task of everyday living.

51. The method of any one of claims 39 to 50, wherein assigning a priority rank to each element of the filtered information comprises assigning a first priority rank to elements of the filtered information that correspond to hazards in the environment surrounding the subject.

52. The method of claim 51, further comprising assigning a priority rank lower than first to elements of the filtered information that correspond to signs, buildings, landmarks, location signals from embedded radiofrequency tags, and navigational instructions.

53. The method of any one of claims 39 to 50, further comprising, for each element corresponding to an object in the environment surrounding the subject, assigning the element'"'"'s priority rank based on at least one of a distance between the object and the subject, and an angular relationship between the object and a direction in which the subject is oriented.

54. The method of any one of claims 48 and 49, wherein refining the elements of the filtered radiation comprises transmitting the elements to a remote electronic processor, and receiving information corresponding to the refined elements from the remote electronic processor.

58. The method of any one of claims 39 to 54, further comprising, prior to transmitting the at least some elements of the filtered information:

re-assessing the priority ranks of one or more elements of the filtered information; and

transmitting the one or more elements of the filtered radiation according to the re-assessed priority ranks.

59. The method of claim 58, further comprising determining whether a triggering event for re-assessment of the priority ranks has occurred prior to transmitting the at least some elements of the filtered information, and re-assessing the priority ranks only if a triggering event has occurred.

60. The method of any one of claims 58 and 59, further comprising, re-assessing the priority ranks by retrieving a set of rules from a database based on the activity of the subject, and applying the set of rules to the one or more elements of the filtered information.

64. The method of any one of claims 39 to 54 and 58 to 60, comprising:

receiving additional information about the environment surrounding the subject; and

prior to receiving the additional information, adjusting a configuration of a detection apparatus used to receive the additional information.

65. The method of claim 64, comprising:

determining components of motion of the user in the information about the environment surrounding the subject; and

adjusting the configuration of the detection apparatus to compensate for at least some of the components of motion.

66. The method of claim 65, wherein the at least some of the components of motion correspond to involuntary motion of the user.

72. The method of any one of claims 39 to 54, 58 to 60, and 64 to 66, wherein at least one element of the filtered information corresponds to text in the environment surrounding the subject, and wherein the method comprises assigning a priority rank to the at least one element corresponding to text based on a distance between the subject and the text.

74. The method of any one of claims 39 to 54, 58 to 60, 64 to 66, and 72, further comprising determining an identity of the subject.