Apparatus and method for improving, augmenting or enhancing vision

• US 10,459,254 B2
• Filed: 02/19/2015
• Issued: 10/29/2019
• Est. Priority Date: 02/19/2014
• Status: Active Grant

• Current Assignee: Evergaze, Inc.
• Original Assignee: Evergaze, Inc.
• Inventors: Antaki, Patrick R., Dunn, Ronnie, Lemburg, Russell
• Primary Examiner(s): Hunter, Mishawn N.
• Application Number:
  US15/119,399
• Publication Number:
  US 20170068119A1
• Time in Patent Office: 1,713 Days
• CPC Class Codes: A61H 5/00 Exercisers for the eyes
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02C 11/10 Electronic devices other than hearing aids
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G09B 21/008 using visual presentation of the information for the partially sighted
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/50 Constructional details
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/651 for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera
• CPC Class Codes: H04N 23/671 in combination with active ranging signals, e.g. using light or sound signals emitted toward objects
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

The present invention provides various apparatus and methods for improving, augmenting or enhancing a vision of a person that include a first camera, one or more sensors, a microdisplay, and one or more processors communicably coupled to the first camera, the one or more sensors and the microdisplay. The first camera is configured to acquire a first image of a scene facing away from an eye of the person. The microdisplay is configured to display a modified first image to the eye. In one operational mode, the one or more processors are configured to acquire the first image of the scene using the first camera, modify the first image based on one or more vision improvement parameters, and display the modified first image on the microdisplay to improve, augment or enhance the vision of the person.

Claims

1. A computerized method for improving, augmenting or enhancing a vision of a person, comprising the steps of:

providing a wearable apparatus proximate to an eye of the person such that a second eye of the person is unobstructed, the wearable apparatus comprising a first camera configured to acquire a first image of a scene facing away from the eye, a second camera configured to acquire a second image of the eye, one or more sensors, a microdisplay configured to display a modified first image to the eye, and one or more processors communicably coupled to the first camera, the second camera, the one or more sensors and the microdisplay, and wherein the wearable apparatus is sized to maintain a peripheral vision of the first eye, and the one or more sensors comprise one or more of a motion sensor, a temperature sensor, an ambient light detector, a rangefinder, a proximity sensor and an infrared sensor;

acquiring the first image of the scene using the first camera;

acquiring the second image of the eye using the second camera;

modifying the second image using the one or more processors;

determining an eye gaze angle based on the second image or the modified second image using the one or more processors;

modifying the first image based on one or more vision improvement parameters and a data from the one or more sensors, and offsetting the first image by an image offset based the eye gaze angle using the one or more processors; and

displaying the modified first image on the microdisplay to improve, augment or enhance the vision of the person.

2. The computerized method as recited in claim 1, further comprising the step of occasionally flashing a predetermined image on the microdisplay.

3. The computerized method as recited in claim 1, wherein the step of displaying the modified first image on the microdisplay further comprises the step of scanning the modified first image onto the microdisplay in a predetermined pattern to emulate the person'"'"'s eye or head movements.

4. The computerized method as recited in claim 1, wherein the step of modifying the first image based on the one or more vision improvement parameters comprises the step of:

offsetting the first image based on a preferred retina locus or an eccentric viewing of the person;

enhancing a contrast of the first image;

or

adjusting a brightness of the first image based on a medical diagnosis, an eye sensitivity, or a background illumination.

5. The computerized method as recited in claim 1, further comprising the step of magnifying the first image of the scene based on a current magnification selected from two or more magnification settings using the first camera or the one or more processors, wherein the current magnification magnifies a whole field of view or only a region centered on a point of gaze.

6. The computerized method as recited in claim 1, further comprising the step of automatically focusing the first image using:

an image analysis based on the eye gaze angle, the one or more sensors or a combination thereof, wherein the one or more sensors comprise a range finder;

a third camera configured to acquire a third image of the scene facing away from the eye that is set to a different focus range than the first camera;

or

a complex lens optically connected to the third camera that sets a focus in different parts of the third image at separate unique focus distance.

7. The computerized method as recited in claim 1, wherein the image offset is further based on an image stabilization comprising the steps of:

measuring a motion data comprising an acceleration data using the one or more sensors, wherein the one or more sensors comprise a motion measurement sensor;

determining an estimated motion data by comparing the first image of the scene to one or more previous first images of the scene; and

determining the image stabilization by merging the acceleration data with the estimated motion data.

8. The computerized method as recited in claim 1, wherein the step of modifying the first image based on the one or more vision improvement parameters comprises the step of enhancing the first image of the scene using one or more image processing algorithms selected from at least one of a contrast enhancement algorithm, an edge sharpening algorithm, a virtual real-time aided vision algorithm, an automatic scene detection and mode setting algorithm, a magnification or image offset algorithm, an artificial edge highlighting/substitution algorithm or a gaze determination algorithm.

9. The computerized method as recited in claim 8, wherein the automatic scene detection and mode setting algorithm comprises the steps of:

determining a scene type and a current task by analyzing the first image, the data from the one or more sensors, or a combination thereof;

changing the one or more vision improvement parameters to match the scene type and the current task or one or more stored settings or preferences, or a combination thereof; and

wherein the current task comprises a close-in reading, a far distance reading, gazing at an external electronic display, looking at another person, walking or driving.

10. The computerized method as recited in claim 1, wherein the step of modifying the first image based on the one or more vision improvement parameters further comprises the step of enhancing the first image of the scene using an eye gesture recognition and mode setting algorithm comprising the steps of:

determining an eye gaze rate of change;

determining a direction of an eye gaze motion;

determining an eye gesture command based on the eye gaze angle, the eye gaze rate of change and the direction of the eye gaze motion; and

changing the one or more vision improvement parameters or magnification in response to the eye gesture command based on one or more stored settings or preferences.

11. The computerized method as recited in claim 1, further comprising the step of activating or deactivating a visible or infrared illuminator based on a light level or a distance determination, wherein the visible or infrared illuminator is configured to face towards the scene and is communicably coupled to the one or more processors.

12. The computerized method as recited in claim 1, wherein the step of modifying the first image based on the one or more vision improvement parameters further comprises the step of maintaining a size of a text within the modified first image at a specified size irrespective of an actual size of the text within the first image.

13. The computerized method as recited in claim 1, further comprising the steps of:

entering a low power mode whenever the one or more sensors detects the person removing the wearable apparatus or going to sleep or the second image indicates that the eye is closed for a specified period of time; and

entering a normal power mode whenever the one or more sensors detects the person putting the wearable apparatus on or awakening from sleep, the second image indicates that the eye is open after being closed for the specified period of time.

14. The computerized method as recited in claim 1, further comprising the step of configuring one or more stored settings or preferences by:

receiving a first message to enter a device setting/calibration mode from a remote device;

transmitting the first image or the modified first image or both the first image and the modified first image to the remote device;

receiving a second message containing a change to the one or more stored settings or preferences;

implementing the change during one or more of the steps of acquiring the first image, modifying the first image and displaying the modified first image;

transmitting the first image or the modified first image or both the first image and the modified first image to the remote device;

storing the change to the one or more stored settings or preferences whenever a third message is received indicating that the first image or the modified first image or both the first image and the modified first image are acceptable;

removing the change to the one or more stored settings or preferences whenever a fourth message is received indicating that the first image or the modified first image or both the first image and the modified first image are not acceptable; and

receiving a fifth message to exit the device setting/calibration mode from the remote device.

15. The computerized method as recited in claim 1, further comprising the step of configuring one or more stored settings or preferences by:

receiving a first message to enter a device setting/calibration mode from a remote device;

transmitting the first image or the modified first image or the second image or the modified second image or a combination thereof to the remote device;

storing the second image or the modified second image as the stored image of the eye whenever a sixth message is received indicating that the second image is acceptable;

repeating the steps of acquiring and transmitting the first image or the modified first image or the second image or the modified second image or a combination thereof whenever a seventh message is received indicating that the second image is not acceptable; and

receiving a fifth message to exit the device setting/calibration mode from the remote device.

16. The computerized method as recited in claim 1, further comprising the steps of:

measuring one or more eye movements based on the second image or the modified second image;

detecting an indication of a potential medical problem by analyzing the one or more eye movements; and

notifying the user of the indication of the potential medical problem or transmitting the indication of the potential medical problem to a remote device, or storing the indication of the potential medical problem.

17. The computerized method as recited in claim 1, further comprising the step of:

performing an eye test by inserting an eye test chart into the modified first image;

performing an eye exercise by inserting a pre-programmed sequence of images into the modified first image;

or

inserting a pre-programmed sequence of images into the modified first image to reduce a strain of the eye or to relax the person.

18. The computerized method as recited in claim 1, wherein:

the first image and the modified first image are not buffered;

or

there is substantially no propagation delay between acquiring the first image and displaying the modified first image.

19. The computerized method as recited in claim 1, further comprising the step of transmitting a time sequence of the first images or the second images or the modified first images or the modified second images or any combination thereof to an external device.

20. The computerized method as recited in claim 1, further comprising the steps of:

receiving a fourth image or a sequence of images or an information from an external device;

creating a fifth image by processing the received fourth image, or the sequence of images or the information using the one or more processors; and

displaying the sixth image on the microdisplay.

21. A wearable apparatus for improving, augmenting or enhancing a vision of a person, comprising:

a first camera configured to acquire a first image of a scene facing away from an eye of the person;

a second camera configured to acquire a second image of the eye;

one or more sensors comprising one or more of a motion sensor, a temperature sensor, an ambient light detector, a rangefinder, a proximity sensor and an infrared sensor;

a microdisplay configured to display a modified first image to the eye such that a second eye of the person is unobstructed;

one or more processors communicably coupled to the first camera, the second camera, the one or more sensors and the microdisplay, wherein the one or more processors are configured to acquire the first image of the scene using the first camera, acquire the second image of the eye using the second camera, modify the second image, determining an eye gaze angle based on the second image or the modified second image, modify the first image based on one or more vision improvement parameters and a data from the one or more sensors, and by offsetting the first image using an the image offset based on the eye gaze angle, and display the modified first image on the microdisplay to improve, augment or enhance the vision of the person; and

the wearable apparatus is sized to maintain a peripheral vision of the first eye.

22. The wearable apparatus as recited in claim 21, further comprising a visible or infrared illuminator communicably coupled to the one or more processors and configured to face towards the scene, wherein the visible or infrared illuminator is activated or deactivated based on a light level or a distance determination.

23. The wearable apparatus as recited in claim 21, wherein the wearable apparatus is attached to or integrated into a monocle, a pair of glasses, a pair of sunglasses or a frame to support the apparatus.

24. The wearable apparatus as recited in claim 23, wherein the frame for the pair of glasses or sunglasses provides a heat sink for the apparatus.

25. The wearable apparatus as recited in claim 23, further comprising a second apparatus attached to or integrated into the pair of glasses, the pair of sunglasses or the frame, wherein the wearable apparatus and the second apparatus communicate with one another.

26. The wearable apparatus as recited in claim 21, wherein the wearable apparatus is mounted onto a frame in front of the eye or a pupil of the eye.

27. The wearable apparatus as recited in claim 21, wherein the wearable apparatus further comprises one or more controls or one or more batteries.

28. The wearable apparatus as recited in claim 21, wherein the first camera and the microdisplay are substantially coaxially aligned with the eye electrically, optically, mechanically or a combination thereof.

29. The wearable apparatus as recited in claim 21, further comprising one or more controls communicably coupled to the one or more processors wherein the one or more controls comprise a knob, a button, a capsense, a slider, a voice activated control, an eye motion activated control, a focus control, a gesture recognition control, an automatic sleep/wake-up control, or a combination thereof.

30. The wearable apparatus as recited in claim 21, further comprising an automatic focusing device communicably coupled to the first camera.

31. The wearable apparatus as recited in claim 21, wherein the one or more processors and the microdisplay are integrated into a single semiconductor die.

32. The wearable apparatus as recited in claim 21, wherein:

the wearable apparatus further comprises a control unit that is communicably coupled to the wearable apparatus wirelessly or via one or more conductors;

the control unit comprises one or more status indicators, one or more controls, one or more batteries and a battery charger electrically connected to the one or more batteries, or a combination thereof.

33. The wearable apparatus as recited in claim 21, wherein the wearable apparatus is configured to complement, coordinate or communicate with an implant within the eye or the eye comprises an artificial eye.

34. The wearable apparatus as recited in claim 21, wherein:

the microdisplay is defined by a first zone and a second zone;

the first zone comprises a whole region of the microdisplay magnified by a background magnification amount; and

the second zone comprises a contiguous zone within the first zone magnified by a different magnification amount.

35. The wearable apparatus as recited in claim 21, further comprising an external device communicably coupled to the one or more processors, wherein the one or more processors are further configured to transmit a time sequence of the first images or the second images or the modified first images or the modified second images or any combination thereof to an external device.

36. The wearable apparatus as recited in claim 21, further comprising an external device communicably coupled to the one or more processors, wherein the one or more processors are further configured to:

receive a fourth image or a sequence of images or an information from the external device;

create a fifth image by processing the received fourth image, or the sequence of images or the information; and

displaying the sixth image on the microdisplay.