Reading assistance system for visually impaired

• US 10,667,981 B2
• Filed: 02/29/2016
• Issued: 06/02/2020
• Est. Priority Date: 02/29/2016
• Status: Active Grant

• Current Assignee: Mentor Acquisition One, LLC (Magic Leap, Inc.)
• Original Assignee: Mentor Acquisition One, LLC (Magic Leap, Inc.)
• Inventors: Osterhout, Ralph F., Border, John N., Haddick, John D., Riehle, Ed
• Primary Examiner(s): Sosanya, Obafemi O
• Application Number:
  US15/056,573
• Publication Number:
  US 20170248788A1
• Time in Patent Office: 1,555 Days
• CPC Class Codes: A61H 2003/063 with tactile perception
• CPC Class Codes: A61H 2201/0188 Illumination related features
• CPC Class Codes: A61H 2201/1604 Head
• CPC Class Codes: A61H 2201/165 Wearable interfaces
• CPC Class Codes: A61H 2201/501 connected to external computer devices or networks
• CPC Class Codes: A61H 2201/5043 Displays
• CPC Class Codes: A61H 2201/5048 Audio interfaces, e.g. voice or music controlled
• CPC Class Codes: A61H 2201/5058 Sensors or detectors
• CPC Class Codes: A61H 2201/5061 Force sensors
• CPC Class Codes: A61H 2201/5064 Position sensors
• CPC Class Codes: A61H 2201/5071 Pressure sensors
• CPC Class Codes: A61H 2201/5082 Temperature sensors
• CPC Class Codes: A61H 2201/5084 Acceleration sensors
• CPC Class Codes: A61H 2201/5092 Optical sensor
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0156 with optionally usable elements
• CPC Class Codes: G02B 2027/0178 : Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 : characterised by optical features
• CPC Class Codes: G02B 27/0176 : characterised by mechanical features
• CPC Class Codes: G02B 7/023 : permitting adjustment
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 : Head tracking input arrangements
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/20208 : High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/20221 : Image fusion; Image merging
• CPC Class Codes: G06T 3/40 : Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/73 : Deblurring; Sharpening
• CPC Class Codes: G06T 5/92 : based on global image properties
• CPC Class Codes: G06V 10/235 : based on user input or interaction
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/63 : Scene text, e.g. street names
• CPC Class Codes: G06V 40/19 : Sensors therefor
• CPC Class Codes: G09B 21/008 : using visual presentation of the information for the partially sighted
• CPC Class Codes: H04N 23/63 : by using electronic viewfinders
• CPC Class Codes: H04N 23/682 : Vibration or motion blur correction
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Aspects of the present disclosure relate aids for the visually impaired.

Claims

1. A head-worn computer including a processor, comprising:

a sensor system comprising a camera, the sensor system adapted to detect a gesture comprising a user touching a document at a first region of the document; and

an augmented reality display, wherein;

the camera is adapted to capture an image of the document in response to the sensor system detecting the gesture; and

the processor is adapted to enhance a portion of the captured image corresponding to the first region of the document and

to present the enhanced portion of the captured image via the augmented reality display concurrent with presenting, via the augmented reality display, a second region of the document different from the first region.

2. The head-worn computer of claim 1, wherein enhancing a portion of the captured image comprises applying a digital zoom to the portion of the captured image.

3. The head-worn computer of claim 1, wherein the camera includes a telephoto lens and the captured image is captured via the telephone lens, and wherein enhancing a portion of the captured image comprises cropping the captured image.

4. The head-worn computer of claim 1, wherein the camera is further adapted to capture the image of the document via a lens using optical image stabilization.

5. A method comprising:

presenting, to a user of a wearable device via a transmissive display of the wearable device, a first view including a view of a first region of a document and a view of a second region of the document, the second region different from the first region;

detecting, via one or more sensors of the wearable device, an identification of the first region by the user;

capturing an image of the view of the first region;

enhancing a visibility of the image to produce an enhanced image of the first region;

presenting, to the user via the transmissive display, a second view including the enhanced image of the first region and the view of the second region.

6. The method of claim 5, wherein the identification of the first region comprises the user touching the document at the first region.

7. The method of claim 5, wherein enhancing the visibility of the image comprises magnifying the first view of the first region.

8. The method of claim 5, wherein the wearable device comprises a camera, and capturing the image comprises capturing the image via the camera.

9. The method of claim 5, wherein the wearable device further comprises a telephoto lens and enhancing the visibility of the image comprises magnifying the image via the telephoto lens.

10. The method of claim 5, wherein the enhanced image is associated with a first focal length and the view of the second region is associated with a second focal length different from the first focal length.

11. The method of claim 5, wherein presenting the second view comprises presenting the enhanced image overlapping the view of the second region.

12. The method of claim 5, wherein the image is captured using a lens and enhancing the visibility of the image comprises applying optical stabilization to the lens.

13. The method of claim 5, wherein presenting the second view comprises presenting the enhanced image in the center of the second view and maintaining the presentation of the enhanced image in the center of the second view.

14. The method of claim 5, wherein the enhanced image is associated with a first field of view and the view of the second region is associated with a second field of view different from the first field of view.

15. The method of claim 5, further comprising detecting text in the image, wherein enhancing the visibility of the image comprises enhancing the readability of the text.

16. The method of claim 15, further comprising presenting, to the user via a speaker of the wearable device, speech corresponding to the text.

17. The method of claim 5, wherein enhancing the visibility of the image comprises adjusting at least one of a contrast, a brightness, a sharpness, or a saturation of the image.

18. A wearable device comprising:

a transmissive display;

one or more sensors, the one or more sensors comprising a camera; and

one or more processors configured to perform;

presenting, to a user of the wearable device via the transmissive display, a first view including a view of a first region of a document and a view of a second region of the document, the second region different from the first region;

detecting, via the one or more sensors, an identification of the first region by the user;

capturing, via the camera, an image of the view of the first region;

enhancing a visibility of the image to produce an enhanced image of the first region; and

presenting, to the user via the transmissive display, a second view including the enhanced image of the first region and the view of the second region.

19. The wearable device of claim 18, wherein the identification of the first region comprises the user touching the document at the first region.

20. The wearable device of claim 18, wherein:

the wearable device further comprises a speaker;

the one or more processors are further configured to perform;

detecting text in the image; and

presenting, to the user via the speaker, speech corresponding to the text.

21. A non-transitory computer-readable storage medium storing instructions that, when executed, perform the following operations:

presenting, to a user of a wearable device via a transmissive display of the wearable device, a first view including a view of a first region of a document and a view of a second region of the document, the second region different from the first region;

detecting, via one or more sensors of the wearable device, an identification of the first region by the user;

capturing an image of the view of the first region;

enhancing a visibility of the image to produce an enhanced image of the first region;

presenting, to the user via the transmissive display, a second view including the enhanced image of the first region and the view of the second region.

22. The non-transitory computer-readable storage medium of claim 21, wherein the identification of the first region comprises the user touching the document at the first region.