Providing enhanced images for navigation

• US 10,849,817 B2
• Filed: 03/08/2016
• Issued: 12/01/2020
• Est. Priority Date: 02/29/2016
• Status: Active Grant

• Current Assignee: Mentor Acquisition One, LLC (Magic Leap, Inc.)
• Original Assignee: Mentor Acquisition One, LLC (Magic Leap, Inc.)
• Inventors: Border, John N.
• Primary Examiner(s): Sosanya, Obafemi O
• Application Number:
  US15/063,682
• Publication Number:
  US 20170249861A1
• Time in Patent Office: 1,729 Days
• CPC Class Codes: A61H 2003/063 with tactile perception
• CPC Class Codes: A61H 2201/0188 Illumination related features
• CPC Class Codes: A61H 2201/1604 Head
• CPC Class Codes: A61H 2201/165 Wearable interfaces
• CPC Class Codes: A61H 2201/501 connected to external computer devices or networks
• CPC Class Codes: A61H 2201/5043 Displays
• CPC Class Codes: A61H 2201/5048 Audio interfaces, e.g. voice or music controlled
• CPC Class Codes: A61H 2201/5058 Sensors or detectors
• CPC Class Codes: A61H 2201/5061 Force sensors
• CPC Class Codes: A61H 2201/5064 Position sensors
• CPC Class Codes: A61H 2201/5071 Pressure sensors
• CPC Class Codes: A61H 2201/5082 Temperature sensors
• CPC Class Codes: A61H 2201/5084 Acceleration sensors
• CPC Class Codes: A61H 2201/5092 Optical sensor
• CPC Class Codes: A61H 3/061 with electronic detecting or guiding means
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0156 with optionally usable elements
• CPC Class Codes: G02B 2027/0178 : Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 : characterised by optical features
• CPC Class Codes: G02B 27/0176 : characterised by mechanical features
• CPC Class Codes: G02B 7/023 : permitting adjustment
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 : Head tracking input arrangements
• CPC Class Codes: G06F 3/013 : Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 11/60 : Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/20208 : High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/20221 : Image fusion; Image merging
• CPC Class Codes: G06T 3/40 : Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/73 : Deblurring; Sharpening
• CPC Class Codes: G06T 5/92 : based on global image properties
• CPC Class Codes: G06V 10/235 : based on user input or interaction
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/63 : Scene text, e.g. street names
• CPC Class Codes: G06V 40/19 : Sensors therefor
• CPC Class Codes: G09B 21/008 : using visual presentation of the information for the partially sighted
• CPC Class Codes: H04N 23/63 : by using electronic viewfinders
• CPC Class Codes: H04N 23/682 : Vibration or motion blur correction
• CPC Class Codes: H04N 23/69 : Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control

Abstract

Aspects of the present disclosure relate aids for the visually impaired.

Claims

1. A method comprising:

capturing, via a camera of a wearable head device comprising a see-through display, an image comprising a surrounding environment of a user of the wearable head device;

enhancing a visibility of a first portion of the captured image corresponding to a first portion of the surrounding environment;

forgoing enhancing a visibility of a second portion of the captured image corresponding to a second portion of the surrounding environment; and

presenting, concurrently via the see-through display of the wearable head device, the enhanced first portion of the captured image and a see-through view of the second portion of the surrounding environment,

wherein presenting the enhanced first portion comprises presenting the enhanced first portion at a region of the see-through display determined based on a position of an eye of the user relative to a field of view of the see-through display.

2. The method of claim 1, wherein enhancing a visibility of a first portion of the captured image comprises increasing a contrast of the first portion of the captured image.

3. The method of claim 1, wherein enhancing a visibility of a first portion of the captured image comprises increasing a brightness of the first portion of the captured image.

4. The method of claim 1, wherein enhancing a visibility of a first portion of the captured image comprises increasing a sharpness of the first portion of the captured image.

5. The method of claim 1, wherein enhancing a visibility of a first portion of the captured image comprises increasing a color saturation of the first portion of the captured image.

6. The method of claim 1, wherein the method is performed repeatedly as the user navigates in the surrounding environment.

7. The method of claim 1, wherein enhancing a visibility of a first portion of the captured image comprises changing a focus distance of the first portion of the captured image.

8. The method of claim 7, wherein the focus distance is determined based on an ophthalmic characteristic of the eye.

9. The method of claim 1, wherein enhancing a visibility of a first portion of the enhanced image comprises adjusting one or more digital code values corresponding to one or more pixels of the first portion of the captured image.

10. The method of claim 9, wherein enhancing the visibility of the first portion of the enhanced image further comprises increasing code values that exceed a predetermined threshold and decreasing code values that do not exceed the predetermined threshold.

11. The method of claim 9, wherein enhancing the visibility of the first portion of the enhanced image further comprises increasing luma for pixels that have luma values above a predetermined threshold and decreasing luma for pixels that have luma values below the predetermined threshold.

12. The method of claim 11, wherein increasing or decreasing luma comprises applying a scale factor to the one or more digital code values associated with the pixels.

13. The method of claim 10, wherein the predetermined threshold comprises a value between 35% and 60% of a maximum code value.

14. A wearable head device comprising:

a see-through display;

a camera; and

one or more processors configured to perform a method comprising;

capturing, via the camera, an image comprising a surrounding environment of a user of the wearable head device;

enhancing a visibility of a first portion of the captured image corresponding to a first portion of the surrounding environment;

forgoing enhancing a visibility of a second portion of the captured image corresponding to a second portion of the surrounding environment; and

presenting, concurrently via the see-through display, the enhanced first portion of the captured image and a see-through view of the second portion of the surrounding environment,

wherein presenting the enhanced first portion comprises presenting the enhanced first portion at a region of the see-through display determined based on a position of an eye of the user relative to a field of view of the see-through display.

15. The wearable head device of claim 14, wherein enhancing a visibility of a first portion of the captured image comprises increasing a contrast of the first portion of the captured image.

16. The wearable head device of claim 14, wherein enhancing a visibility of a first portion of the captured image comprises increasing a brightness of the first portion of the captured image.

17. The wearable head device of claim 14, wherein enhancing a visibility of a first portion of the captured image comprises increasing a color saturation of the first portion of the captured image.

18. The wearable head device of claim 14, wherein enhancing a visibility of a first portion of the captured image comprises increasing a sharpness of the first portion of the captured image.

19. The wearable head device of claim 14, wherein enhancing a visibility of a first portion of the captured image comprises changing a focus distance of the first portion of the captured image.

20. The wearable head device of claim 19, wherein the focus distance is determined based on an ophthalmic characteristic of the eye.